Sep 09 16:13:54.065565 ip-10-0-141-202 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 16:13:54.419353 ip-10-0-141-202 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 16:13:54.419353 ip-10-0-141-202 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 16:13:54.419353 ip-10-0-141-202 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 16:13:54.419353 ip-10-0-141-202 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 16:13:54.419353 ip-10-0-141-202 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 16:13:54.420201 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.420085    2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 16:13:54.425533 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425506    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 16:13:54.425533 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425529    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 16:13:54.425533 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425535    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 16:13:54.425533 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425540    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 16:13:54.425723 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425546    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 16:13:54.425723 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425551    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 16:13:54.425723 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425556    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 16:13:54.425723 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425560    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 16:13:54.425723 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425565    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 16:13:54.425723 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425574    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 16:13:54.425723 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425579    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 16:13:54.425723 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425583    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 16:13:54.425723 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425588    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 16:13:54.425723 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425593    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 16:13:54.425723 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425597    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 16:13:54.425723 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425601    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 16:13:54.425723 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425676    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 16:13:54.426032 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425782    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 16:13:54.426032 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425791    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 16:13:54.426032 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425796    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 16:13:54.426032 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425799    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 16:13:54.426032 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425803    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 16:13:54.426032 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425806    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 16:13:54.426032 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425809    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 16:13:54.426032 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425812    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 16:13:54.426032 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425816    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 16:13:54.426032 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425818    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 16:13:54.426032 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425821    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 16:13:54.426032 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425824    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 16:13:54.426032 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425828    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 16:13:54.426032 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425831    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 16:13:54.426032 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425834    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 16:13:54.426032 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425837    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 16:13:54.426032 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425841    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 16:13:54.426032 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425844    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 16:13:54.426032 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425847    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 16:13:54.426032 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425850    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 16:13:54.426536 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425853    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 16:13:54.426536 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425856    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 16:13:54.426536 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425859    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 16:13:54.426536 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425862    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 16:13:54.426536 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425865    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 16:13:54.426536 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425867    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 16:13:54.426536 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425870    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 16:13:54.426536 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425873    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 16:13:54.426536 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425875    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 16:13:54.426536 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425878    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 16:13:54.426536 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425881    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 16:13:54.426536 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425884    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 16:13:54.426536 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425887    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 16:13:54.426536 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425890    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 16:13:54.426536 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425893    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 16:13:54.426536 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425895    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 16:13:54.426536 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425900    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 16:13:54.426536 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425906    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 16:13:54.426536 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425909    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 16:13:54.426536 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425913    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 16:13:54.427017 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425916    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 16:13:54.427017 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425919    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 16:13:54.427017 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425922    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 16:13:54.427017 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425925    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 16:13:54.427017 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425931    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 16:13:54.427017 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425935    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 16:13:54.427017 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425938    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 16:13:54.427017 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425941    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 16:13:54.427017 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425944    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 16:13:54.427017 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425948    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 16:13:54.427017 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425952    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 16:13:54.427017 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425955    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 16:13:54.427017 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425957    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 16:13:54.427017 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425960    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 16:13:54.427017 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425963    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 16:13:54.427017 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425966    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 16:13:54.427017 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425969    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 16:13:54.427017 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425971    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 16:13:54.427017 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425974    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 16:13:54.427524 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425976    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 16:13:54.427524 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425979    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 16:13:54.427524 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425982    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 16:13:54.427524 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425984    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 16:13:54.427524 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425987    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 16:13:54.427524 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425990    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 16:13:54.427524 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425994    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 16:13:54.427524 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.425999    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 16:13:54.427524 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426001    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 16:13:54.427524 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426006    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 16:13:54.427524 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426010    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 16:13:54.427524 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426013    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 16:13:54.427524 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426015    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 16:13:54.427524 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426018    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 16:13:54.427524 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426021    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 16:13:54.427524 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426024    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 16:13:54.427524 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426027    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 16:13:54.427524 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426030    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 16:13:54.427524 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426032    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 16:13:54.427972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426035    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 16:13:54.427972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426038    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 16:13:54.427972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426041    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 16:13:54.427972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426044    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 16:13:54.427972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426772    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 16:13:54.427972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426779    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 16:13:54.427972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426781    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 16:13:54.427972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426784    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 16:13:54.427972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426788    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 16:13:54.427972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426790    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 16:13:54.427972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426793    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 16:13:54.427972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426795    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 16:13:54.427972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426798    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 16:13:54.427972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426801    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 16:13:54.427972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426803    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 16:13:54.427972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426806    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 16:13:54.427972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426808    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 16:13:54.427972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426811    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 16:13:54.427972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426815    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 16:13:54.427972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426818    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 16:13:54.428484 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426821    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 16:13:54.428484 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426824    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 16:13:54.428484 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426827    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 16:13:54.428484 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426829    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 16:13:54.428484 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426832    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 16:13:54.428484 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426834    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 16:13:54.428484 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426837    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 16:13:54.428484 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426840    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 16:13:54.428484 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426842    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 16:13:54.428484 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426845    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 16:13:54.428484 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426847    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 16:13:54.428484 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426850    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 16:13:54.428484 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426852    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 16:13:54.428484 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426855    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 16:13:54.428484 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426857    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 16:13:54.428484 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426859    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 16:13:54.428484 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426862    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 16:13:54.428484 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426864    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 16:13:54.428484 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426867    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 16:13:54.428484 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426869    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 16:13:54.428972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426872    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 16:13:54.428972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426875    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 16:13:54.428972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426877    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 16:13:54.428972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426880    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 16:13:54.428972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426883    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 16:13:54.428972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426885    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 16:13:54.428972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426888    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 16:13:54.428972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426891    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 16:13:54.428972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426894    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 16:13:54.428972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426898    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 16:13:54.428972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426903    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 16:13:54.428972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426906    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 16:13:54.428972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426909    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 16:13:54.428972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426912    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 16:13:54.428972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426915    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 16:13:54.428972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426918    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 16:13:54.428972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426921    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 16:13:54.428972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426924    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 16:13:54.428972 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426926    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 16:13:54.429483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426929    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 16:13:54.429483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426931    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 16:13:54.429483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426935    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 16:13:54.429483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426937    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 16:13:54.429483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426940    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 16:13:54.429483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426942    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 16:13:54.429483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426945    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 16:13:54.429483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426948    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 16:13:54.429483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426950    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 16:13:54.429483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426953    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 16:13:54.429483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426955    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 16:13:54.429483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426958    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 16:13:54.429483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426962    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 16:13:54.429483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426966    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 16:13:54.429483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426970    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 16:13:54.429483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426973    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 16:13:54.429483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426977    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 16:13:54.429483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426980    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 16:13:54.429483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426983    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 16:13:54.429930 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426986    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 16:13:54.429930 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426988    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 16:13:54.429930 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426991    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 16:13:54.429930 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426993    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 16:13:54.429930 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426996    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 16:13:54.429930 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.426999    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 16:13:54.429930 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427002    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 16:13:54.429930 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427005    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 16:13:54.429930 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427007    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 16:13:54.429930 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427011    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 16:13:54.429930 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427013    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 16:13:54.429930 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427016    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 16:13:54.429930 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427018    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 16:13:54.429930 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427021    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 16:13:54.429930 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427024    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 16:13:54.429930 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427026    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 16:13:54.429930 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427029    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 16:13:54.429930 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427032    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 16:13:54.429930 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427034    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 16:13:54.430421 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427037    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 16:13:54.430421 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427039    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 16:13:54.430421 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427042    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 16:13:54.430421 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427044    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 16:13:54.430421 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427047    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 16:13:54.430421 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427049    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 16:13:54.430421 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427479    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 16:13:54.430421 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427485    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 16:13:54.430421 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427488    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 16:13:54.430421 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427491    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 16:13:54.430421 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427495    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 16:13:54.430421 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427498    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 16:13:54.430421 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427500    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 16:13:54.430421 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427503    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 16:13:54.430421 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427506    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 16:13:54.430421 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427508    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 16:13:54.430421 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427511    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 16:13:54.430421 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427514    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 16:13:54.430421 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427516    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 16:13:54.430875 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427519    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 16:13:54.430875 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427522    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 16:13:54.430875 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427525    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 16:13:54.430875 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427528    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 16:13:54.430875 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427531    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 16:13:54.430875 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427533    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 16:13:54.430875 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427536    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 16:13:54.430875 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427539    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 16:13:54.430875 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427542    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 16:13:54.430875 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427544    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 16:13:54.430875 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427547    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 16:13:54.430875 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427550    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 16:13:54.430875 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427552    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 16:13:54.430875 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427555    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 16:13:54.430875 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427557    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 16:13:54.430875 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427560    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 16:13:54.430875 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427562    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 16:13:54.430875 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427565    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 16:13:54.430875 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427568    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 16:13:54.430875 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427570    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 16:13:54.431450 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427574    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 16:13:54.431450 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427577    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 16:13:54.431450 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427580    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 16:13:54.431450 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427583    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 16:13:54.431450 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427585    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 16:13:54.431450 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427588    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 16:13:54.431450 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427590    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 16:13:54.431450 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427593    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 16:13:54.431450 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427595    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 16:13:54.431450 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427598    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 16:13:54.431450 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427602    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 16:13:54.431450 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427606    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 16:13:54.431450 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427609    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 16:13:54.431450 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427613    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 16:13:54.431450 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427615    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 16:13:54.431450 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427618    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 16:13:54.431450 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427621    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 16:13:54.431450 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427624    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 16:13:54.431450 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427626    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 16:13:54.431912 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427629    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 16:13:54.431912 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427632    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 16:13:54.431912 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427635    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 16:13:54.431912 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427637    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 16:13:54.431912 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427639    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 16:13:54.431912 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427642    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 16:13:54.431912 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427644    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 16:13:54.431912 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427647    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 16:13:54.431912 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427649    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 16:13:54.431912 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427652    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 16:13:54.431912 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427654    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 16:13:54.431912 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427657    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 16:13:54.431912 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427662    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 16:13:54.431912 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427664    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 16:13:54.431912 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427667    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 16:13:54.431912 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427669    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 16:13:54.431912 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427672    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 16:13:54.431912 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427675    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 16:13:54.431912 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427677    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 16:13:54.431912 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427681    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 16:13:54.432445 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427683    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 16:13:54.432445 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427686    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 16:13:54.432445 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427688    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 16:13:54.432445 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427691    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 16:13:54.432445 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427694    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 16:13:54.432445 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427697    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 16:13:54.432445 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427699    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 16:13:54.432445 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427702    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 16:13:54.432445 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427704    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 16:13:54.432445 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427708    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 16:13:54.432445 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427712    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 16:13:54.432445 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427715    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 16:13:54.432445 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427718    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 16:13:54.432445 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427720    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 16:13:54.432445 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427723    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 16:13:54.432445 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427725    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 16:13:54.432445 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427728    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 16:13:54.432445 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427730    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 16:13:54.432445 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427733    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 16:13:54.432900 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427735    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 16:13:54.432900 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427738    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 16:13:54.432900 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427740    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 16:13:54.432900 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427743    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 16:13:54.432900 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427745    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 16:13:54.432900 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427749    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 16:13:54.432900 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427752    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 16:13:54.432900 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.427754    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 16:13:54.432900 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427824    2555 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 16:13:54.432900 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427857    2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 16:13:54.432900 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427864    2555 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 16:13:54.432900 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427869    2555 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 16:13:54.432900 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427872    2555 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 16:13:54.432900 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427876    2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 16:13:54.432900 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427880    2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 16:13:54.432900 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427884    2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 16:13:54.432900 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427888    2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 16:13:54.432900 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427891    2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 16:13:54.432900 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427895    2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 16:13:54.432900 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427898    2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 16:13:54.432900 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427901    2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 16:13:54.432900 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427904    2555 flags.go:64] FLAG: --cgroup-root=""
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427907    2555 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427910    2555 flags.go:64] FLAG: --client-ca-file=""
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427914    2555 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427917    2555 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427922    2555 flags.go:64] FLAG: --cluster-domain=""
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427925    2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427928    2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427931    2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427934    2555 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427938    2555 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427941    2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427944    2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427947    2555 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427950    2555 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427953    2555 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427957    2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427960    2555 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427963    2555 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427967    2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427970    2555 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427973    2555 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427976    2555 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427979    2555 flags.go:64] FLAG: --enable-server="true"
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427982    2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 16:13:54.433460 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427987    2555 flags.go:64] FLAG: --event-burst="100"
Sep 09 16:13:54.434056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427990    2555 flags.go:64] FLAG: --event-qps="50"
Sep 09 16:13:54.434056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427994    2555 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 16:13:54.434056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.427997    2555 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 16:13:54.434056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428000    2555 flags.go:64] FLAG: --eviction-hard=""
Sep 09 16:13:54.434056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428003    2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 16:13:54.434056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428006    2555 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 16:13:54.434056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428010    2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 16:13:54.434056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428012    2555 flags.go:64] FLAG: --eviction-soft=""
Sep 09 16:13:54.434056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428015    2555 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 16:13:54.434056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428019    2555 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 16:13:54.434056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428022    2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 16:13:54.434056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428025    2555 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 16:13:54.434056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428028    2555 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 16:13:54.434056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428031    2555 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 16:13:54.434056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428034    2555 flags.go:64] FLAG: --feature-gates=""
Sep 09 16:13:54.434056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428037    2555 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 16:13:54.434056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428040    2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 16:13:54.434056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428043    2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 16:13:54.434056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428047    2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 16:13:54.434056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428050    2555 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 16:13:54.434056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428053    2555 flags.go:64] FLAG: --help="false"
Sep 09 16:13:54.434056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428055    2555 flags.go:64] FLAG: --hostname-override="ip-10-0-141-202.ec2.internal"
Sep 09 16:13:54.434056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428060    2555 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 16:13:54.434056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428063    2555 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 16:13:54.434676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428065    2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 16:13:54.434676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428069    2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 16:13:54.434676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428072    2555 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 16:13:54.434676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428075    2555 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 16:13:54.434676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428077    2555 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 16:13:54.434676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428080    2555 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 16:13:54.434676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428083    2555 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 16:13:54.434676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428086    2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 16:13:54.434676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428090    2555 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 16:13:54.434676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428094    2555 flags.go:64] FLAG: --kube-reserved=""
Sep 09 16:13:54.434676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428097    2555 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 16:13:54.434676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428100    2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 16:13:54.434676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428103    2555 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 16:13:54.434676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428105    2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 16:13:54.434676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428108    2555 flags.go:64] FLAG: --lock-file=""
Sep 09 16:13:54.434676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428111    2555 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 16:13:54.434676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428114    2555 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 16:13:54.434676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428117    2555 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 16:13:54.434676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428122    2555 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 16:13:54.434676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428130    2555 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 16:13:54.434676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428133    2555 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 16:13:54.434676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428136    2555 flags.go:64] FLAG: --logging-format="text"
Sep 09 16:13:54.434676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428138    2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 16:13:54.435246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428142    2555 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 16:13:54.435246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428144    2555 flags.go:64] FLAG: --manifest-url=""
Sep 09 16:13:54.435246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428147    2555 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 16:13:54.435246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428151    2555 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 16:13:54.435246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428154    2555 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 16:13:54.435246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428158    2555 flags.go:64] FLAG: --max-pods="110"
Sep 09 16:13:54.435246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428161    2555 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 16:13:54.435246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428165    2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 16:13:54.435246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428168    2555 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 16:13:54.435246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428171    2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 16:13:54.435246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428174    2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 16:13:54.435246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428177    2555 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 16:13:54.435246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428180    2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 16:13:54.435246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428186    2555 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 16:13:54.435246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428189    2555 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 16:13:54.435246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428192    2555 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 16:13:54.435246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428195    2555 flags.go:64] FLAG: --pod-cidr=""
Sep 09 16:13:54.435246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428198    2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 16:13:54.435246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428205    2555 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 16:13:54.435246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428207    2555 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 16:13:54.435246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428210    2555 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 16:13:54.435246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428213    2555 flags.go:64] FLAG: --port="10250"
Sep 09 16:13:54.435246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428216    2555 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 16:13:54.435246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428219    2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03cd9d039b2359996"
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428222    2555 flags.go:64] FLAG: --qos-reserved=""
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428225    2555 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428228    2555 flags.go:64] FLAG: --register-node="true"
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428231    2555 flags.go:64] FLAG: --register-with-taints=""
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428234    2555 flags.go:64] FLAG: --registry-burst="10"
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428238    2555 flags.go:64] FLAG: --registry-qps="5"
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428241    2555 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428244    2555 flags.go:64] FLAG: --reserved-memory=""
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428247    2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428250    2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428253    2555 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428256    2555 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428259    2555 flags.go:64] FLAG: --runonce="false"
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428262    2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428264    2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428268    2555 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428271    2555 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428274    2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428277    2555 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428280    2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428283    2555 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428285    2555 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428301    2555 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428305    2555 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428308    2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 16:13:54.435888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428311    2555 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 16:13:54.436541 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428315    2555 flags.go:64] FLAG: --system-cgroups=""
Sep 09 16:13:54.436541 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428318    2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 16:13:54.436541 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428323    2555 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 16:13:54.436541 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428326    2555 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 16:13:54.436541 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428329    2555 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 16:13:54.436541 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428334    2555 flags.go:64] FLAG: --tls-min-version=""
Sep 09 16:13:54.436541 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428337    2555 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 16:13:54.436541 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428339    2555 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 16:13:54.436541 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428342    2555 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 16:13:54.436541 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428345    2555 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 16:13:54.436541 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428348    2555 flags.go:64] FLAG: --v="2"
Sep 09 16:13:54.436541 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428352    2555 flags.go:64] FLAG: --version="false"
Sep 09 16:13:54.436541 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428364    2555 flags.go:64] FLAG: --vmodule=""
Sep 09 16:13:54.436541 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428367    2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 16:13:54.436541 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.428370    2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 16:13:54.436541 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428489    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 16:13:54.436541 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428493    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 16:13:54.436541 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428496    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 16:13:54.436541 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428499    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 16:13:54.436541 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428503    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 16:13:54.436541 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428507    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 16:13:54.436541 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428510    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 16:13:54.436541 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428515    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 16:13:54.437088 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428518    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 16:13:54.437088 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428521    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 16:13:54.437088 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428524    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 16:13:54.437088 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428528    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 16:13:54.437088 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428531    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 16:13:54.437088 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428534    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 16:13:54.437088 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428537    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 16:13:54.437088 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428540    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 16:13:54.437088 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428543    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 16:13:54.437088 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428547    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 16:13:54.437088 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428549    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 16:13:54.437088 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428552    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 16:13:54.437088 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428554    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 16:13:54.437088 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428557    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 16:13:54.437088 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428559    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 16:13:54.437088 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428562    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 16:13:54.437088 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428565    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 16:13:54.437088 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428567    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 16:13:54.437088 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428570    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 16:13:54.437088 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428572    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 16:13:54.437606 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428575    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 16:13:54.437606 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428577    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 16:13:54.437606 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428585    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 16:13:54.437606 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428588    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 16:13:54.437606 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428591    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 16:13:54.437606 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428594    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 16:13:54.437606 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428596    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 16:13:54.437606 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428599    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 16:13:54.437606 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428601    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 16:13:54.437606 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428604    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 16:13:54.437606 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428606    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 16:13:54.437606 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428610    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 16:13:54.437606 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428613    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 16:13:54.437606 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428615    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 16:13:54.437606 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428618    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 16:13:54.437606 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428620    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 16:13:54.437606 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428623    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 16:13:54.437606 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428626    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 16:13:54.437606 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428628    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 16:13:54.437606 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428631    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 16:13:54.438092 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428633    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 16:13:54.438092 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428637    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 16:13:54.438092 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428640    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 16:13:54.438092 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428643    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 16:13:54.438092 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428645    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 16:13:54.438092 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428648    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 16:13:54.438092 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428650    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 16:13:54.438092 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428653    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 16:13:54.438092 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428655    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 16:13:54.438092 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428658    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 16:13:54.438092 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428660    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 16:13:54.438092 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428663    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 16:13:54.438092 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428666    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 16:13:54.438092 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428668    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 16:13:54.438092 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428671    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 16:13:54.438092 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428679    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 16:13:54.438092 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428681    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 16:13:54.438092 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428684    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 16:13:54.438092 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428687    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 16:13:54.438577 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428689    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 16:13:54.438577 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428692    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 16:13:54.438577 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428694    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 16:13:54.438577 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428697    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 16:13:54.438577 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428700    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 16:13:54.438577 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428703    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 16:13:54.438577 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428706    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 16:13:54.438577 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428708    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 16:13:54.438577 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428711    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 16:13:54.438577 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428714    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 16:13:54.438577 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428716    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 16:13:54.438577 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428719    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 16:13:54.438577 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428721    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 16:13:54.438577 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428725    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 16:13:54.438577 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428730    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 16:13:54.438577 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428733    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 16:13:54.438577 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428736    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 16:13:54.438577 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428739    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 16:13:54.438577 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428742    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 16:13:54.439050 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428744    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 16:13:54.439050 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428747    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 16:13:54.439050 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428749    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 16:13:54.439050 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428752    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 16:13:54.439050 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428755    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 16:13:54.439050 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428757    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 16:13:54.439050 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428760    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 16:13:54.439050 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428762    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 16:13:54.439050 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428765    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 16:13:54.439050 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428767    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 16:13:54.439050 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428771    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 16:13:54.439050 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428773    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 16:13:54.439050 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.428776    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 16:13:54.439050 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.429337    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 16:13:54.439050 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.436227    2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 16:13:54.439482 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.436240    2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 16:13:54.439482 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.436979    2555 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 16:13:54.439482 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.436990    2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 16:13:54.439482 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437052    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 16:13:54.439482 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437057    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 16:13:54.439482 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437061    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 16:13:54.439482 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437064    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 16:13:54.439482 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437067    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 16:13:54.439482 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437070    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 16:13:54.439482 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437072    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 16:13:54.439482 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437075    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 16:13:54.439482 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437078    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 16:13:54.439482 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437081    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 16:13:54.439482 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437084    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 16:13:54.439482 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437086    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 16:13:54.439482 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437089    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 16:13:54.439482 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437092    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 16:13:54.439482 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437094    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 16:13:54.439482 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437097    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 16:13:54.439949 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437100    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 16:13:54.439949 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437103    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 16:13:54.439949 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437106    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 16:13:54.439949 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437108    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 16:13:54.439949 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437111    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 16:13:54.439949 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437114    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 16:13:54.439949 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437116    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 16:13:54.439949 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437119    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 16:13:54.439949 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437122    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 16:13:54.439949 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437124    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 16:13:54.439949 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437129    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 16:13:54.439949 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437132    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 16:13:54.439949 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437135    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 16:13:54.439949 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437140    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 16:13:54.439949 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437143    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 16:13:54.439949 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437145    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 16:13:54.439949 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437148    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 16:13:54.439949 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437151    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 16:13:54.439949 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437153    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 16:13:54.440483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437156    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 16:13:54.440483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437159    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 16:13:54.440483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437161    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 16:13:54.440483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437164    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 16:13:54.440483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437166    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 16:13:54.440483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437169    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 16:13:54.440483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437172    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 16:13:54.440483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437174    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 16:13:54.440483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437177    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 16:13:54.440483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437180    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 16:13:54.440483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437182    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 16:13:54.440483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437185    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 16:13:54.440483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437188    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 16:13:54.440483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437190    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 16:13:54.440483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437193    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 16:13:54.440483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437196    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 16:13:54.440483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437198    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 16:13:54.440483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437201    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 16:13:54.440483 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437205    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 16:13:54.440938 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437209    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 16:13:54.440938 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437213    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 16:13:54.440938 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437216    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 16:13:54.440938 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437226    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 16:13:54.440938 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437231    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 16:13:54.440938 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437234    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 16:13:54.440938 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437236    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 16:13:54.440938 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437240    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 16:13:54.440938 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437244    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 16:13:54.440938 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437246    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 16:13:54.440938 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437249    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 16:13:54.440938 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437252    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 16:13:54.440938 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437254    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 16:13:54.440938 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437257    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 16:13:54.440938 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437260    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 16:13:54.440938 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437263    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 16:13:54.440938 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437265    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 16:13:54.440938 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437268    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 16:13:54.440938 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437271    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 16:13:54.440938 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437273    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 16:13:54.441528 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437276    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 16:13:54.441528 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437278    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 16:13:54.441528 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437281    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 16:13:54.441528 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437284    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 16:13:54.441528 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437286    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 16:13:54.441528 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437301    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 16:13:54.441528 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437306    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 16:13:54.441528 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437308    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 16:13:54.441528 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437311    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 16:13:54.441528 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437314    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 16:13:54.441528 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437317    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 16:13:54.441528 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437319    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 16:13:54.441528 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437322    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 16:13:54.441528 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437325    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 16:13:54.441528 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437328    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 16:13:54.441528 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437330    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 16:13:54.441528 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437333    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 16:13:54.441528 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437335    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 16:13:54.441528 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437338    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 16:13:54.441528 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437342    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 16:13:54.442007 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437345    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 16:13:54.442007 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437348    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 16:13:54.442007 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437351    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 16:13:54.442007 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437353    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 16:13:54.442007 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437356    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 16:13:54.442007 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.437361    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 16:13:54.442007 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437491    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 16:13:54.442007 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437497    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 16:13:54.442007 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437500    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 16:13:54.442007 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437504    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 16:13:54.442007 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437507    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 16:13:54.442007 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437510    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 16:13:54.442007 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437513    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 16:13:54.442007 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437516    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 16:13:54.442007 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437519    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 16:13:54.442388 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437522    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 16:13:54.442388 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437525    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 16:13:54.442388 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437528    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 16:13:54.442388 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437530    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 16:13:54.442388 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437533    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 16:13:54.442388 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437536    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 16:13:54.442388 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437538    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 16:13:54.442388 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437541    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 16:13:54.442388 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437544    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 16:13:54.442388 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437546    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 16:13:54.442388 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437549    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 16:13:54.442388 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437551    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 16:13:54.442388 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437554    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 16:13:54.442388 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437557    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 16:13:54.442388 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437559    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 16:13:54.442388 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437562    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 16:13:54.442388 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437565    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 16:13:54.442388 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437568    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 16:13:54.442388 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437571    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 16:13:54.442388 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437576    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 16:13:54.442872 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437579    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 16:13:54.442872 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437583    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 16:13:54.442872 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437587    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 16:13:54.442872 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437590    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 16:13:54.442872 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437593    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 16:13:54.442872 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437596    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 16:13:54.442872 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437598    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 16:13:54.442872 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437601    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 16:13:54.442872 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437604    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 16:13:54.442872 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437607    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 16:13:54.442872 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437610    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 16:13:54.442872 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437621    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 16:13:54.442872 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437625    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 16:13:54.442872 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437628    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 16:13:54.442872 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437631    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 16:13:54.442872 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437635    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 16:13:54.442872 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437637    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 16:13:54.442872 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437640    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 16:13:54.442872 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437643    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 16:13:54.443342 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437646    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 16:13:54.443342 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437648    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 16:13:54.443342 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437651    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 16:13:54.443342 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437654    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 16:13:54.443342 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437657    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 16:13:54.443342 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437659    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 16:13:54.443342 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437662    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 16:13:54.443342 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437665    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 16:13:54.443342 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437667    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 16:13:54.443342 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437670    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 16:13:54.443342 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437673    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 16:13:54.443342 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437676    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 16:13:54.443342 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437679    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 16:13:54.443342 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437687    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 16:13:54.443342 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437690    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 16:13:54.443342 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437692    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 16:13:54.443342 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437695    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 16:13:54.443342 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437698    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 16:13:54.443342 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437700    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 16:13:54.443809 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437703    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 16:13:54.443809 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437705    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 16:13:54.443809 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437708    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 16:13:54.443809 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437710    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 16:13:54.443809 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437713    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 16:13:54.443809 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437716    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 16:13:54.443809 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437718    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 16:13:54.443809 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437721    2555 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 16:13:54.443809 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437723    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 16:13:54.443809 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437726    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 16:13:54.443809 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437729    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 16:13:54.443809 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437731    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 16:13:54.443809 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437734    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 16:13:54.443809 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437736    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 16:13:54.443809 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437739    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 16:13:54.443809 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437742    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 16:13:54.443809 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437744    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 16:13:54.443809 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437747    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 16:13:54.443809 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437750    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 16:13:54.443809 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437752    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 16:13:54.444339 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437755    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 16:13:54.444339 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437758    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 16:13:54.444339 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437761    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 16:13:54.444339 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437764    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 16:13:54.444339 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437767    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 16:13:54.444339 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437769    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 16:13:54.444339 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437772    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 16:13:54.444339 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437780    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 16:13:54.444339 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437783    2555 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 16:13:54.444339 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437786    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 16:13:54.444339 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437788    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 16:13:54.444339 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:54.437790    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 16:13:54.444339 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.437795    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 16:13:54.444339 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.437911    2555 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 16:13:54.444723 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.440207    2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 16:13:54.444723 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.441106    2555 server.go:1019] "Starting client certificate rotation"
Sep 09 16:13:54.444723 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.441212    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 16:13:54.444723 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.441968    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 16:13:54.444723 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.443967    2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 16:13:54.447570 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.447544    2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 16:13:54.458533 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.458505    2555 log.go:25] "Validated CRI v1 runtime API"
Sep 09 16:13:54.462456 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.462441    2555 log.go:25] "Validated CRI v1 image API"
Sep 09 16:13:54.463338 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.463323    2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 16:13:54.467398 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.467377    2555 fs.go:135] Filesystem UUIDs: map[1b72d169-ff49-4b6d-82de-99ee2e80ae12:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7cc432c1-45d1-4278-9c09-0f3a070e5610:/dev/nvme0n1p3]
Sep 09 16:13:54.467481 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.467397    2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 16:13:54.470817 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.470796    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 16:13:54.473078 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.472864    2555 manager.go:216] Machine: {Timestamp:2026-09-09 16:13:54.471716607 +0000 UTC m=+0.301924671 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101297 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c895bb1ece6e9b23340fe4b20ad1c SystemUUID:ec2c895b-b1ec-e6e9-b233-40fe4b20ad1c BootID:5c751c64-3c77-40f3-be8c-179b63eee320 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b2:be:ab:15:0d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b2:be:ab:15:0d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:c4:36:ad:e8:50 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 16:13:54.473078 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.473073    2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 16:13:54.473235 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.473177    2555 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 16:13:54.476080 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.476058    2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 16:13:54.476206 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.476084    2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-202.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 16:13:54.476898 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.476887    2555 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 16:13:54.476929 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.476900    2555 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 16:13:54.476929 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.476908    2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 16:13:54.477515 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.477505    2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 16:13:54.479253 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.479243    2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 16:13:54.483188 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.483177    2555 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 16:13:54.483309 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.483285    2555 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 16:13:54.483357 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.483326    2555 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 16:13:54.483357 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.483335    2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 16:13:54.483357 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.483350    2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 16:13:54.483480 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.483362    2555 kubelet.go:398] "Adding apiserver pod source"
Sep 09 16:13:54.483480 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.483371    2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 16:13:54.483761 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.483750    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 16:13:54.483805 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.483769    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 16:13:54.485705 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.485687    2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 16:13:54.488203 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.488190    2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 16:13:54.488253 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.488220    2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 16:13:54.489044 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.489033    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 16:13:54.489079 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.489049    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 16:13:54.489079 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.489055    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 16:13:54.489079 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.489060    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 16:13:54.489079 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.489068    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 16:13:54.489079 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.489074    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 16:13:54.489196 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.489094    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 16:13:54.489196 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.489100    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 16:13:54.489196 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.489106    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 16:13:54.489196 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.489111    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 16:13:54.489196 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.489125    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 16:13:54.489196 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.489130    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 16:13:54.489827 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.489818    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 16:13:54.489827 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.489827    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 16:13:54.493327 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.493313    2555 server.go:1268] "Started kubelet"
Sep 09 16:13:54.493430 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.493380    2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 16:13:54.493481 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.493427    2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 16:13:54.493527 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.493497    2555 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 16:13:54.493922 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.493906    2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-202.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 16:13:54.494234 ip-10-0-141-202 systemd[1]: Started Kubernetes Kubelet.
Sep 09 16:13:54.494645 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:54.494617    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 16:13:54.494645 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:54.494616    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-202.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 16:13:54.494790 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.494741    2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 16:13:54.495580 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.495559    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c6sd7"
Sep 09 16:13:54.499376 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.499283    2555 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 16:13:54.500820 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.500801    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c6sd7"
Sep 09 16:13:54.501332 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.501317    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 16:13:54.501585 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:54.500448    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-202.ec2.internal.18d3b321bda5c568  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-202.ec2.internal,UID:ip-10-0-141-202.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-202.ec2.internal,},FirstTimestamp:2026-09-09 16:13:54.493269352 +0000 UTC m=+0.323477404,LastTimestamp:2026-09-09 16:13:54.493269352 +0000 UTC m=+0.323477404,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-202.ec2.internal,}"
Sep 09 16:13:54.501890 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.501771    2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 16:13:54.502031 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:54.502015    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-202.ec2.internal\" not found"
Sep 09 16:13:54.502322 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.502309    2555 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 16:13:54.502420 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.502411    2555 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 16:13:54.502548 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.502537    2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 16:13:54.502699 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.502688    2555 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 16:13:54.502773 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.502764    2555 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 16:13:54.503717 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.503700    2555 factory.go:153] Registering CRI-O factory
Sep 09 16:13:54.503717 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.503721    2555 factory.go:223] Registration of the crio container factory successfully
Sep 09 16:13:54.503862 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.503769    2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 16:13:54.503862 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.503780    2555 factory.go:55] Registering systemd factory
Sep 09 16:13:54.503862 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.503788    2555 factory.go:223] Registration of the systemd container factory successfully
Sep 09 16:13:54.503862 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.503802    2555 factory.go:103] Registering Raw factory
Sep 09 16:13:54.503862 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.503816    2555 manager.go:1196] Started watching for new ooms in manager
Sep 09 16:13:54.504062 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:54.504029    2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 16:13:54.504237 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.504222    2555 manager.go:349] Starting recovery of all containers
Sep 09 16:13:54.511958 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.511934    2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 16:13:54.514095 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.514079    2555 manager.go:354] Recovery completed
Sep 09 16:13:54.514620 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:54.514602    2555 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-202.ec2.internal\" not found" node="ip-10-0-141-202.ec2.internal"
Sep 09 16:13:54.518887 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.518185    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 16:13:54.519850 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.519835    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 16:13:54.519942 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.519860    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 16:13:54.519942 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.519871    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-202.ec2.internal" event="NodeHasSufficientPID"
Sep 09 16:13:54.520586 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.520566    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 16:13:54.520656 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.520565    2555 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 16:13:54.520656 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.520625    2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 16:13:54.520656 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.520654    2555 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 16:13:54.521335 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.521316    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 16:13:54.521410 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.521353    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 16:13:54.521410 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.521369    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-202.ec2.internal" event="NodeHasSufficientPID"
Sep 09 16:13:54.522650 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.522638    2555 policy_none.go:49] "None policy: Start"
Sep 09 16:13:54.522733 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.522654    2555 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 16:13:54.522733 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.522665    2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 16:13:54.524918 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.524423    2555 policy_none.go:47] "Start"
Sep 09 16:13:54.576506 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.557681    2555 manager.go:337] "Starting Device Plugin manager"
Sep 09 16:13:54.576506 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:54.557722    2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 16:13:54.576506 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.557735    2555 server.go:87] "Starting device plugin registration server"
Sep 09 16:13:54.576506 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.558096    2555 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 16:13:54.576506 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.558108    2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 16:13:54.576506 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.558209    2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 16:13:54.576506 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.558310    2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 16:13:54.576506 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.558320    2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 16:13:54.576506 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:54.558789    2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 16:13:54.576506 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:54.558839    2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-202.ec2.internal\" not found"
Sep 09 16:13:54.599962 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.599941    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 16:13:54.601108 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.601095    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 16:13:54.601168 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.601141    2555 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 16:13:54.601168 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.601161    2555 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 16:13:54.601263 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:54.601196    2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 16:13:54.602864 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.602847    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 16:13:54.659110 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.659089    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 16:13:54.659957 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.659941    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 16:13:54.660028 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.659969    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 16:13:54.660028 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.659979    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-202.ec2.internal" event="NodeHasSufficientPID"
Sep 09 16:13:54.660028 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.660003    2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-202.ec2.internal"
Sep 09 16:13:54.667692 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.667678    2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-202.ec2.internal"
Sep 09 16:13:54.667759 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:54.667698    2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-202.ec2.internal\": node \"ip-10-0-141-202.ec2.internal\" not found"
Sep 09 16:13:54.682654 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:54.682587    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-202.ec2.internal\" not found"
Sep 09 16:13:54.702165 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.702146    2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-202.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-202.ec2.internal"]
Sep 09 16:13:54.702249 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.702211    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 16:13:54.703581 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.703567    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 16:13:54.703661 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.703591    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 16:13:54.703661 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.703605    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-202.ec2.internal" event="NodeHasSufficientPID"
Sep 09 16:13:54.705825 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.705812    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 16:13:54.705971 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.705959    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-202.ec2.internal"
Sep 09 16:13:54.706043 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.705984    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 16:13:54.706534 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.706515    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 16:13:54.706620 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.706540    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 16:13:54.706620 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.706549    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 16:13:54.706620 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.706559    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-202.ec2.internal" event="NodeHasSufficientPID"
Sep 09 16:13:54.706620 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.706565    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 16:13:54.706620 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.706575    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-202.ec2.internal" event="NodeHasSufficientPID"
Sep 09 16:13:54.708794 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.708779    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-202.ec2.internal"
Sep 09 16:13:54.708878 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.708812    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 16:13:54.709533 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.709517    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 16:13:54.709588 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.709548    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 16:13:54.709588 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.709564    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-202.ec2.internal" event="NodeHasSufficientPID"
Sep 09 16:13:54.722825 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:54.722807    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-202.ec2.internal\" not found" node="ip-10-0-141-202.ec2.internal"
Sep 09 16:13:54.727167 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:54.727150    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-202.ec2.internal\" not found" node="ip-10-0-141-202.ec2.internal"
Sep 09 16:13:54.782703 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:54.782680    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-202.ec2.internal\" not found"
Sep 09 16:13:54.804374 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.804357    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1489a50f476055d1d230538a489fe1f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-202.ec2.internal\" (UID: \"1489a50f476055d1d230538a489fe1f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-202.ec2.internal"
Sep 09 16:13:54.804457 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.804419    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1489a50f476055d1d230538a489fe1f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-202.ec2.internal\" (UID: \"1489a50f476055d1d230538a489fe1f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-202.ec2.internal"
Sep 09 16:13:54.804457 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.804439    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a40d2b498bc21892b32d687ae581fc71-config\") pod \"kube-apiserver-proxy-ip-10-0-141-202.ec2.internal\" (UID: \"a40d2b498bc21892b32d687ae581fc71\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-202.ec2.internal"
Sep 09 16:13:54.883108 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:54.883073    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-202.ec2.internal\" not found"
Sep 09 16:13:54.905409 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.905388    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1489a50f476055d1d230538a489fe1f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-202.ec2.internal\" (UID: \"1489a50f476055d1d230538a489fe1f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-202.ec2.internal"
Sep 09 16:13:54.905468 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.905414    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1489a50f476055d1d230538a489fe1f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-202.ec2.internal\" (UID: \"1489a50f476055d1d230538a489fe1f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-202.ec2.internal"
Sep 09 16:13:54.905468 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.905430    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a40d2b498bc21892b32d687ae581fc71-config\") pod \"kube-apiserver-proxy-ip-10-0-141-202.ec2.internal\" (UID: \"a40d2b498bc21892b32d687ae581fc71\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-202.ec2.internal"
Sep 09 16:13:54.905568 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.905495    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a40d2b498bc21892b32d687ae581fc71-config\") pod \"kube-apiserver-proxy-ip-10-0-141-202.ec2.internal\" (UID: \"a40d2b498bc21892b32d687ae581fc71\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-202.ec2.internal"
Sep 09 16:13:54.905568 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.905499    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1489a50f476055d1d230538a489fe1f4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-202.ec2.internal\" (UID: \"1489a50f476055d1d230538a489fe1f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-202.ec2.internal"
Sep 09 16:13:54.905568 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:54.905518    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1489a50f476055d1d230538a489fe1f4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-202.ec2.internal\" (UID: \"1489a50f476055d1d230538a489fe1f4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-202.ec2.internal"
Sep 09 16:13:54.983700 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:54.983640    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-202.ec2.internal\" not found"
Sep 09 16:13:55.026174 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.026143    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-202.ec2.internal"
Sep 09 16:13:55.029990 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.029968    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-202.ec2.internal"
Sep 09 16:13:55.084356 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:55.084327    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-202.ec2.internal\" not found"
Sep 09 16:13:55.184870 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:55.184845    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-202.ec2.internal\" not found"
Sep 09 16:13:55.285386 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:55.285308    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-202.ec2.internal\" not found"
Sep 09 16:13:55.360867 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.360842    2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 16:13:55.385387 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:55.385358    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-202.ec2.internal\" not found"
Sep 09 16:13:55.411369 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.411351    2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 16:13:55.440962 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.440942    2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 16:13:55.441382 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.441056    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 16:13:55.441382 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.441058    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 16:13:55.441382 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.441073    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 16:13:55.441382 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.441057    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 16:13:55.484346 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.484323    2555 apiserver.go:52] "Watching apiserver"
Sep 09 16:13:55.502535 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.502507    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-202.ec2.internal"
Sep 09 16:13:55.502630 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.502545    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 16:13:55.503476 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.503445    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 16:08:55 +0000 UTC" deadline="2028-05-10 09:42:08.969093236 +0000 UTC"
Sep 09 16:13:55.503476 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.503475    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14609h28m13.465621225s"
Sep 09 16:13:55.503944 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.503919    2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 16:13:55.505336 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.505313    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-6hgj9","openshift-multus/network-metrics-daemon-gcb4p","openshift-network-diagnostics/network-check-target-slbqw","openshift-network-operator/iptables-alerter-rhdkg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv","openshift-cluster-node-tuning-operator/tuned-trqnv","openshift-image-registry/node-ca-hkc7k","openshift-ovn-kubernetes/ovnkube-node-phs9w","kube-system/global-pull-secret-syncer-hjz7d","kube-system/konnectivity-agent-9shcg","openshift-dns/node-resolver-92lzv","openshift-multus/multus-2d5w7"]
Sep 09 16:13:55.508019 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.507995    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6hgj9"
Sep 09 16:13:55.511883 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.511383    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 16:13:55.511883 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.511481    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-202.ec2.internal"
Sep 09 16:13:55.512597 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.512467    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:13:55.512690 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:55.512578    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gcb4p" podUID="7334d4d3-9b79-4b0d-ab28-e0aec88c18cb"
Sep 09 16:13:55.512690 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.512613    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 16:13:55.512690 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.512665    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 16:13:55.512853 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.512815    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 16:13:55.512853 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.512820    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 16:13:55.512853 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.512838    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2v25j\""
Sep 09 16:13:55.513022 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.512816    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 16:13:55.514894 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.514877    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:13:55.514962 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:55.514941    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slbqw" podUID="f1bd8aa2-35fd-458b-8a84-69ccf7a6276f"
Sep 09 16:13:55.517167 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.517148    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rhdkg"
Sep 09 16:13:55.517673 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.517655    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 16:13:55.519559 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.519539    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 16:13:55.519647 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.519555    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 16:13:55.519647 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.519604    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 16:13:55.519647 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.519629    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv"
Sep 09 16:13:55.519938 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.519915    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hxb6n\""
Sep 09 16:13:55.521049 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.521029    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 16:13:55.522232 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.522212    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 16:13:55.522328 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.522275    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.522516 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.522484    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 16:13:55.522594 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.522487    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j29t9\""
Sep 09 16:13:55.522708 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.522689    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 16:13:55.524607 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.524570    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hkc7k"
Sep 09 16:13:55.524767 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.524747    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 16:13:55.524941 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.524916    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 16:13:55.525164 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.525144    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b62br\""
Sep 09 16:13:55.526979 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.526957    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 16:13:55.527183 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.527167    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.527244 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.527211    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 16:13:55.527478 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.527450    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-v855q\""
Sep 09 16:13:55.527666 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.527652    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 16:13:55.529595 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.529578    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 16:13:55.529693 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.529614    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bvgrk\""
Sep 09 16:13:55.529693 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.529688    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 16:13:55.529778 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.529702    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:13:55.529778 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.529733    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 16:13:55.529778 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:55.529764    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjz7d" podUID="98c72774-4b40-43b2-b4dd-7590320acc60"
Sep 09 16:13:55.531077 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.531060    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 16:13:55.531179 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.531065    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 16:13:55.531179 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.531128    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 16:13:55.532877 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.532858    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9shcg"
Sep 09 16:13:55.535272 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.535255    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-92lzv"
Sep 09 16:13:55.535539 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.535488    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 16:13:55.535624 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.535581    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7wbz9\""
Sep 09 16:13:55.535796 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.535777    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 16:13:55.537841 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.537819    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 16:13:55.537982 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.537956    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 16:13:55.538076 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.538013    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f9fcc\""
Sep 09 16:13:55.538076 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.538048    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nj9sq"
Sep 09 16:13:55.538347 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.538319    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-202.ec2.internal"]
Sep 09 16:13:55.538426 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.538355    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-202.ec2.internal"]
Sep 09 16:13:55.540161 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.538624    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.541908 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.541753    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 16:13:55.542002 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.541845    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4kfhd\""
Sep 09 16:13:55.546057 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.546040    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nj9sq"
Sep 09 16:13:55.603540 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.603513    2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 16:13:55.608041 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608023    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bd350005-0808-410e-bef5-9ecfd227869d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6hgj9\" (UID: \"bd350005-0808-410e-bef5-9ecfd227869d\") " pod="openshift-multus/multus-additional-cni-plugins-6hgj9"
Sep 09 16:13:55.608114 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608050    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-sys\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.608114 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608072    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/98c72774-4b40-43b2-b4dd-7590320acc60-kubelet-config\") pod \"global-pull-secret-syncer-hjz7d\" (UID: \"98c72774-4b40-43b2-b4dd-7590320acc60\") " pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:13:55.608114 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608104    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3aa9556d-7d31-44de-8861-6ef97bc87b66-sys-fs\") pod \"aws-ebs-csi-driver-node-qsnkv\" (UID: \"3aa9556d-7d31-44de-8861-6ef97bc87b66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv"
Sep 09 16:13:55.608233 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608173    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-host-cni-bin\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.608233 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608211    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsdz6\" (UniqueName: \"kubernetes.io/projected/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-kube-api-access-bsdz6\") pod \"network-metrics-daemon-gcb4p\" (UID: \"7334d4d3-9b79-4b0d-ab28-e0aec88c18cb\") " pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:13:55.608323 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608288    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd350005-0808-410e-bef5-9ecfd227869d-cnibin\") pod \"multus-additional-cni-plugins-6hgj9\" (UID: \"bd350005-0808-410e-bef5-9ecfd227869d\") " pod="openshift-multus/multus-additional-cni-plugins-6hgj9"
Sep 09 16:13:55.608403 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608356    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-etc-sysconfig\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.608403 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608390    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/68024951-17f4-4df9-b48a-7674411fd492-agent-certs\") pod \"konnectivity-agent-9shcg\" (UID: \"68024951-17f4-4df9-b48a-7674411fd492\") " pod="kube-system/konnectivity-agent-9shcg"
Sep 09 16:13:55.608496 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608421    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-node-log\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.608496 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608469    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-multus-socket-dir-parent\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.608571 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608503    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-host-run-netns\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.608571 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608529    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dqgw\" (UniqueName: \"kubernetes.io/projected/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-kube-api-access-2dqgw\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.608571 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608555    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bd350005-0808-410e-bef5-9ecfd227869d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6hgj9\" (UID: \"bd350005-0808-410e-bef5-9ecfd227869d\") " pod="openshift-multus/multus-additional-cni-plugins-6hgj9"
Sep 09 16:13:55.608684 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608580    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f46d54d3-e17a-4ef8-9e1e-41cb4820d5fd-iptables-alerter-script\") pod \"iptables-alerter-rhdkg\" (UID: \"f46d54d3-e17a-4ef8-9e1e-41cb4820d5fd\") " pod="openshift-network-operator/iptables-alerter-rhdkg"
Sep 09 16:13:55.608684 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608603    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-etc-kubernetes\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.608684 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608629    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3aa9556d-7d31-44de-8861-6ef97bc87b66-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qsnkv\" (UID: \"3aa9556d-7d31-44de-8861-6ef97bc87b66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv"
Sep 09 16:13:55.608684 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608654    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-host-var-lib-cni-bin\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.608684 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608672    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-multus-conf-dir\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.608914 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608687    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtk84\" (UniqueName: \"kubernetes.io/projected/f46d54d3-e17a-4ef8-9e1e-41cb4820d5fd-kube-api-access-vtk84\") pod \"iptables-alerter-rhdkg\" (UID: \"f46d54d3-e17a-4ef8-9e1e-41cb4820d5fd\") " pod="openshift-network-operator/iptables-alerter-rhdkg"
Sep 09 16:13:55.608914 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608702    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-lib-modules\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.608914 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608718    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/68024951-17f4-4df9-b48a-7674411fd492-konnectivity-ca\") pod \"konnectivity-agent-9shcg\" (UID: \"68024951-17f4-4df9-b48a-7674411fd492\") " pod="kube-system/konnectivity-agent-9shcg"
Sep 09 16:13:55.608914 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608754    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-host-run-ovn-kubernetes\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.608914 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608797    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc6eae27-5d1d-4957-99b5-e87c1d177376-hosts-file\") pod \"node-resolver-92lzv\" (UID: \"fc6eae27-5d1d-4957-99b5-e87c1d177376\") " pod="openshift-dns/node-resolver-92lzv"
Sep 09 16:13:55.608914 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608826    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-cnibin\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.608914 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608857    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd350005-0808-410e-bef5-9ecfd227869d-cni-binary-copy\") pod \"multus-additional-cni-plugins-6hgj9\" (UID: \"bd350005-0808-410e-bef5-9ecfd227869d\") " pod="openshift-multus/multus-additional-cni-plugins-6hgj9"
Sep 09 16:13:55.608914 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608889    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cq97\" (UniqueName: \"kubernetes.io/projected/bd350005-0808-410e-bef5-9ecfd227869d-kube-api-access-8cq97\") pod \"multus-additional-cni-plugins-6hgj9\" (UID: \"bd350005-0808-410e-bef5-9ecfd227869d\") " pod="openshift-multus/multus-additional-cni-plugins-6hgj9"
Sep 09 16:13:55.608914 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608912    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3f37385e-b449-4281-b738-7edce088387c-serviceca\") pod \"node-ca-hkc7k\" (UID: \"3f37385e-b449-4281-b738-7edce088387c\") " pod="openshift-image-registry/node-ca-hkc7k"
Sep 09 16:13:55.609224 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608931    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-host-slash\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.609224 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608957    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd350005-0808-410e-bef5-9ecfd227869d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6hgj9\" (UID: \"bd350005-0808-410e-bef5-9ecfd227869d\") " pod="openshift-multus/multus-additional-cni-plugins-6hgj9"
Sep 09 16:13:55.609224 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608973    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-etc-modprobe-d\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.609224 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.608987    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-run\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.609224 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609003    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3aa9556d-7d31-44de-8861-6ef97bc87b66-device-dir\") pod \"aws-ebs-csi-driver-node-qsnkv\" (UID: \"3aa9556d-7d31-44de-8861-6ef97bc87b66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv"
Sep 09 16:13:55.609224 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609017    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-log-socket\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.609224 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609033    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-host-var-lib-kubelet\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.609224 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609052    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f46d54d3-e17a-4ef8-9e1e-41cb4820d5fd-host-slash\") pod \"iptables-alerter-rhdkg\" (UID: \"f46d54d3-e17a-4ef8-9e1e-41cb4820d5fd\") " pod="openshift-network-operator/iptables-alerter-rhdkg"
Sep 09 16:13:55.609224 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609080    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-etc-sysctl-conf\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.609224 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609099    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afea1994-1cf2-4678-b897-aeaa41412fc1-tmp\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.609224 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609120    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-host-run-netns\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.609224 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609153    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-var-lib-openvswitch\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.609224 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609183    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4fce212-3111-43ec-8fee-a58bd7ffdff9-ovnkube-script-lib\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.609224 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609203    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b78gh\" (UniqueName: \"kubernetes.io/projected/e4fce212-3111-43ec-8fee-a58bd7ffdff9-kube-api-access-b78gh\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.609224 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609217    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-host-var-lib-cni-multus\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.609822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609236    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-host\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.609822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609259    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98c72774-4b40-43b2-b4dd-7590320acc60-original-pull-secret\") pod \"global-pull-secret-syncer-hjz7d\" (UID: \"98c72774-4b40-43b2-b4dd-7590320acc60\") " pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:13:55.609822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609275    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4fce212-3111-43ec-8fee-a58bd7ffdff9-ovn-node-metrics-cert\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.609822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609298    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-host-run-multus-certs\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.609822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609316    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-etc-kubernetes\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.609822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609355    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd350005-0808-410e-bef5-9ecfd227869d-system-cni-dir\") pod \"multus-additional-cni-plugins-6hgj9\" (UID: \"bd350005-0808-410e-bef5-9ecfd227869d\") " pod="openshift-multus/multus-additional-cni-plugins-6hgj9"
Sep 09 16:13:55.609822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609385    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/afea1994-1cf2-4678-b897-aeaa41412fc1-etc-tuned\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.609822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609418    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3aa9556d-7d31-44de-8861-6ef97bc87b66-registration-dir\") pod \"aws-ebs-csi-driver-node-qsnkv\" (UID: \"3aa9556d-7d31-44de-8861-6ef97bc87b66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv"
Sep 09 16:13:55.609822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609439    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2pr9\" (UniqueName: \"kubernetes.io/projected/3aa9556d-7d31-44de-8861-6ef97bc87b66-kube-api-access-c2pr9\") pod \"aws-ebs-csi-driver-node-qsnkv\" (UID: \"3aa9556d-7d31-44de-8861-6ef97bc87b66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv"
Sep 09 16:13:55.609822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609460    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-host-kubelet\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.609822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609485    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3aa9556d-7d31-44de-8861-6ef97bc87b66-socket-dir\") pod \"aws-ebs-csi-driver-node-qsnkv\" (UID: \"3aa9556d-7d31-44de-8861-6ef97bc87b66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv"
Sep 09 16:13:55.609822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609533    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-run-systemd\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.609822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609559    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-host-cni-netd\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.609822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609580    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-system-cni-dir\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.609822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609603    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dphp\" (UniqueName: \"kubernetes.io/projected/3f37385e-b449-4281-b738-7edce088387c-kube-api-access-4dphp\") pod \"node-ca-hkc7k\" (UID: \"3f37385e-b449-4281-b738-7edce088387c\") " pod="openshift-image-registry/node-ca-hkc7k"
Sep 09 16:13:55.609822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609621    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52hd9\" (UniqueName: \"kubernetes.io/projected/fc6eae27-5d1d-4957-99b5-e87c1d177376-kube-api-access-52hd9\") pod \"node-resolver-92lzv\" (UID: \"fc6eae27-5d1d-4957-99b5-e87c1d177376\") " pod="openshift-dns/node-resolver-92lzv"
Sep 09 16:13:55.610489 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609636    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv7fn\" (UniqueName: \"kubernetes.io/projected/f1bd8aa2-35fd-458b-8a84-69ccf7a6276f-kube-api-access-cv7fn\") pod \"network-check-target-slbqw\" (UID: \"f1bd8aa2-35fd-458b-8a84-69ccf7a6276f\") " pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:13:55.610489 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609655    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.610489 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609687    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-multus-cni-dir\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.610489 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609720    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc6eae27-5d1d-4957-99b5-e87c1d177376-tmp-dir\") pod \"node-resolver-92lzv\" (UID: \"fc6eae27-5d1d-4957-99b5-e87c1d177376\") " pod="openshift-dns/node-resolver-92lzv"
Sep 09 16:13:55.610489 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609776    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-cni-binary-copy\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.610489 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609805    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-hostroot\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.610489 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609828    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-multus-daemon-config\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.610489 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609852    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/98c72774-4b40-43b2-b4dd-7590320acc60-dbus\") pod \"global-pull-secret-syncer-hjz7d\" (UID: \"98c72774-4b40-43b2-b4dd-7590320acc60\") " pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:13:55.610489 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609874    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-run-ovn\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.610489 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609905    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs\") pod \"network-metrics-daemon-gcb4p\" (UID: \"7334d4d3-9b79-4b0d-ab28-e0aec88c18cb\") " pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:13:55.610489 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609930    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-etc-systemd\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.610489 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.609953    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-var-lib-kubelet\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.610489 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.610001    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwngc\" (UniqueName: \"kubernetes.io/projected/afea1994-1cf2-4678-b897-aeaa41412fc1-kube-api-access-wwngc\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.610489 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.610041    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-run-openvswitch\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.610489 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.610057    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4fce212-3111-43ec-8fee-a58bd7ffdff9-ovnkube-config\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.610489 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.610098    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-host-run-k8s-cni-cncf-io\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.611283 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.610130    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-etc-sysctl-d\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.611283 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.610154    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3aa9556d-7d31-44de-8861-6ef97bc87b66-etc-selinux\") pod \"aws-ebs-csi-driver-node-qsnkv\" (UID: \"3aa9556d-7d31-44de-8861-6ef97bc87b66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv"
Sep 09 16:13:55.611283 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.610179    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f37385e-b449-4281-b738-7edce088387c-host\") pod \"node-ca-hkc7k\" (UID: \"3f37385e-b449-4281-b738-7edce088387c\") " pod="openshift-image-registry/node-ca-hkc7k"
Sep 09 16:13:55.611283 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.610204    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-systemd-units\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.611283 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.610236    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-etc-openvswitch\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.611283 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.610265    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4fce212-3111-43ec-8fee-a58bd7ffdff9-env-overrides\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.617558 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:55.617524    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1489a50f476055d1d230538a489fe1f4.slice/crio-3a3cbc51c277b5a93ffe0ce546aa15223137a7a8a72ea5b198a86751b3a0301f WatchSource:0}: Error finding container 3a3cbc51c277b5a93ffe0ce546aa15223137a7a8a72ea5b198a86751b3a0301f: Status 404 returned error can't find the container with id 3a3cbc51c277b5a93ffe0ce546aa15223137a7a8a72ea5b198a86751b3a0301f
Sep 09 16:13:55.618505 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:55.618487    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40d2b498bc21892b32d687ae581fc71.slice/crio-a37e3fcb6fbbea30944c9a34d57e46ad9e2789123351ff2b9faf167ef9e0648f WatchSource:0}: Error finding container a37e3fcb6fbbea30944c9a34d57e46ad9e2789123351ff2b9faf167ef9e0648f: Status 404 returned error can't find the container with id a37e3fcb6fbbea30944c9a34d57e46ad9e2789123351ff2b9faf167ef9e0648f
Sep 09 16:13:55.624834 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.624815    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 16:13:55.710827 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.710795    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3f37385e-b449-4281-b738-7edce088387c-serviceca\") pod \"node-ca-hkc7k\" (UID: \"3f37385e-b449-4281-b738-7edce088387c\") " pod="openshift-image-registry/node-ca-hkc7k"
Sep 09 16:13:55.710827 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.710828    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-host-slash\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.711018 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.710845    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd350005-0808-410e-bef5-9ecfd227869d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6hgj9\" (UID: \"bd350005-0808-410e-bef5-9ecfd227869d\") " pod="openshift-multus/multus-additional-cni-plugins-6hgj9"
Sep 09 16:13:55.711018 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.710862    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-etc-modprobe-d\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.711018 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.710877    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-run\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.711018 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.710894    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3aa9556d-7d31-44de-8861-6ef97bc87b66-device-dir\") pod \"aws-ebs-csi-driver-node-qsnkv\" (UID: \"3aa9556d-7d31-44de-8861-6ef97bc87b66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv"
Sep 09 16:13:55.711018 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.710922    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-host-slash\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.711018 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.710969    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-log-socket\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.711018 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.710997    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-host-var-lib-kubelet\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.711298 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711024    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f46d54d3-e17a-4ef8-9e1e-41cb4820d5fd-host-slash\") pod \"iptables-alerter-rhdkg\" (UID: \"f46d54d3-e17a-4ef8-9e1e-41cb4820d5fd\") " pod="openshift-network-operator/iptables-alerter-rhdkg"
Sep 09 16:13:55.711298 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711026    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-run\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.711298 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.710953    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3aa9556d-7d31-44de-8861-6ef97bc87b66-device-dir\") pod \"aws-ebs-csi-driver-node-qsnkv\" (UID: \"3aa9556d-7d31-44de-8861-6ef97bc87b66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv"
Sep 09 16:13:55.711298 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711051    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-etc-sysctl-conf\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.711298 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711059    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-log-socket\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.711298 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711059    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd350005-0808-410e-bef5-9ecfd227869d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6hgj9\" (UID: \"bd350005-0808-410e-bef5-9ecfd227869d\") " pod="openshift-multus/multus-additional-cni-plugins-6hgj9"
Sep 09 16:13:55.711298 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711023    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-etc-modprobe-d\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.711298 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711084    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afea1994-1cf2-4678-b897-aeaa41412fc1-tmp\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.711298 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711085    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f46d54d3-e17a-4ef8-9e1e-41cb4820d5fd-host-slash\") pod \"iptables-alerter-rhdkg\" (UID: \"f46d54d3-e17a-4ef8-9e1e-41cb4820d5fd\") " pod="openshift-network-operator/iptables-alerter-rhdkg"
Sep 09 16:13:55.711298 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711091    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-host-var-lib-kubelet\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.711298 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711111    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-host-run-netns\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.711298 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711133    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-var-lib-openvswitch\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.711298 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711157    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4fce212-3111-43ec-8fee-a58bd7ffdff9-ovnkube-script-lib\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.711298 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711172    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-host-run-netns\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.711298 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711195    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-var-lib-openvswitch\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.711298 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711217    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b78gh\" (UniqueName: \"kubernetes.io/projected/e4fce212-3111-43ec-8fee-a58bd7ffdff9-kube-api-access-b78gh\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.711298 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711246    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-host-var-lib-cni-multus\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.712087 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711250    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-etc-sysctl-conf\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.712087 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711262    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-host\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.712087 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711279    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98c72774-4b40-43b2-b4dd-7590320acc60-original-pull-secret\") pod \"global-pull-secret-syncer-hjz7d\" (UID: \"98c72774-4b40-43b2-b4dd-7590320acc60\") " pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:13:55.712087 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711330    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-host-var-lib-cni-multus\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.712087 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711360    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4fce212-3111-43ec-8fee-a58bd7ffdff9-ovn-node-metrics-cert\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.712087 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711382    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-host-run-multus-certs\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.712087 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711402    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-etc-kubernetes\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.712087 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711424    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd350005-0808-410e-bef5-9ecfd227869d-system-cni-dir\") pod \"multus-additional-cni-plugins-6hgj9\" (UID: \"bd350005-0808-410e-bef5-9ecfd227869d\") " pod="openshift-multus/multus-additional-cni-plugins-6hgj9"
Sep 09 16:13:55.712087 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711450    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-host-run-multus-certs\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.712087 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711461    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd350005-0808-410e-bef5-9ecfd227869d-system-cni-dir\") pod \"multus-additional-cni-plugins-6hgj9\" (UID: \"bd350005-0808-410e-bef5-9ecfd227869d\") " pod="openshift-multus/multus-additional-cni-plugins-6hgj9"
Sep 09 16:13:55.712087 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711462    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-etc-kubernetes\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.712087 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711517    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/afea1994-1cf2-4678-b897-aeaa41412fc1-etc-tuned\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.712087 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:55.711531    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 16:13:55.712087 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711584    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-host\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.712087 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:55.711591    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98c72774-4b40-43b2-b4dd-7590320acc60-original-pull-secret podName:98c72774-4b40-43b2-b4dd-7590320acc60 nodeName:}" failed. No retries permitted until 2026-09-09 16:13:56.211569979 +0000 UTC m=+2.041778048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98c72774-4b40-43b2-b4dd-7590320acc60-original-pull-secret") pod "global-pull-secret-syncer-hjz7d" (UID: "98c72774-4b40-43b2-b4dd-7590320acc60") : object "kube-system"/"original-pull-secret" not registered
Sep 09 16:13:55.712087 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711547    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3aa9556d-7d31-44de-8861-6ef97bc87b66-registration-dir\") pod \"aws-ebs-csi-driver-node-qsnkv\" (UID: \"3aa9556d-7d31-44de-8861-6ef97bc87b66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv"
Sep 09 16:13:55.712087 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711703    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2pr9\" (UniqueName: \"kubernetes.io/projected/3aa9556d-7d31-44de-8861-6ef97bc87b66-kube-api-access-c2pr9\") pod \"aws-ebs-csi-driver-node-qsnkv\" (UID: \"3aa9556d-7d31-44de-8861-6ef97bc87b66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv"
Sep 09 16:13:55.712858 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711731    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-host-kubelet\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.712858 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711757    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3aa9556d-7d31-44de-8861-6ef97bc87b66-socket-dir\") pod \"aws-ebs-csi-driver-node-qsnkv\" (UID: \"3aa9556d-7d31-44de-8861-6ef97bc87b66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv"
Sep 09 16:13:55.712858 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711760    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3aa9556d-7d31-44de-8861-6ef97bc87b66-registration-dir\") pod \"aws-ebs-csi-driver-node-qsnkv\" (UID: \"3aa9556d-7d31-44de-8861-6ef97bc87b66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv"
Sep 09 16:13:55.712858 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711763    2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 16:13:55.712858 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711783    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-run-systemd\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.712858 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711801    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-host-kubelet\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.712858 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711807    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-host-cni-netd\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.712858 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711848    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-system-cni-dir\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.712858 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711858    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-run-systemd\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.712858 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711874    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dphp\" (UniqueName: \"kubernetes.io/projected/3f37385e-b449-4281-b738-7edce088387c-kube-api-access-4dphp\") pod \"node-ca-hkc7k\" (UID: \"3f37385e-b449-4281-b738-7edce088387c\") " pod="openshift-image-registry/node-ca-hkc7k"
Sep 09 16:13:55.712858 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711898    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52hd9\" (UniqueName: \"kubernetes.io/projected/fc6eae27-5d1d-4957-99b5-e87c1d177376-kube-api-access-52hd9\") pod \"node-resolver-92lzv\" (UID: \"fc6eae27-5d1d-4957-99b5-e87c1d177376\") " pod="openshift-dns/node-resolver-92lzv"
Sep 09 16:13:55.712858 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711954    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3aa9556d-7d31-44de-8861-6ef97bc87b66-socket-dir\") pod \"aws-ebs-csi-driver-node-qsnkv\" (UID: \"3aa9556d-7d31-44de-8861-6ef97bc87b66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv"
Sep 09 16:13:55.712858 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711996    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-host-cni-netd\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.712858 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712004    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-system-cni-dir\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.712858 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.711788    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4fce212-3111-43ec-8fee-a58bd7ffdff9-ovnkube-script-lib\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.712858 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712043    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3f37385e-b449-4281-b738-7edce088387c-serviceca\") pod \"node-ca-hkc7k\" (UID: \"3f37385e-b449-4281-b738-7edce088387c\") " pod="openshift-image-registry/node-ca-hkc7k"
Sep 09 16:13:55.712858 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712066    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv7fn\" (UniqueName: \"kubernetes.io/projected/f1bd8aa2-35fd-458b-8a84-69ccf7a6276f-kube-api-access-cv7fn\") pod \"network-check-target-slbqw\" (UID: \"f1bd8aa2-35fd-458b-8a84-69ccf7a6276f\") " pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:13:55.712858 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712095    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.713603 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712117    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-multus-cni-dir\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.713603 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712141    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc6eae27-5d1d-4957-99b5-e87c1d177376-tmp-dir\") pod \"node-resolver-92lzv\" (UID: \"fc6eae27-5d1d-4957-99b5-e87c1d177376\") " pod="openshift-dns/node-resolver-92lzv"
Sep 09 16:13:55.713603 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712169    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-cni-binary-copy\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.713603 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712168    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.713603 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712219    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-hostroot\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.713603 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712248    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-multus-daemon-config\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.713603 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712258    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-multus-cni-dir\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.713603 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712288    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/98c72774-4b40-43b2-b4dd-7590320acc60-dbus\") pod \"global-pull-secret-syncer-hjz7d\" (UID: \"98c72774-4b40-43b2-b4dd-7590320acc60\") " pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:13:55.713603 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712332    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-run-ovn\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.713603 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712341    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-hostroot\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.713603 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712420    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs\") pod \"network-metrics-daemon-gcb4p\" (UID: \"7334d4d3-9b79-4b0d-ab28-e0aec88c18cb\") " pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:13:55.713603 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712465    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-etc-systemd\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.713603 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712485    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/98c72774-4b40-43b2-b4dd-7590320acc60-dbus\") pod \"global-pull-secret-syncer-hjz7d\" (UID: \"98c72774-4b40-43b2-b4dd-7590320acc60\") " pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:13:55.713603 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712492    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-var-lib-kubelet\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.713603 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712519    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwngc\" (UniqueName: \"kubernetes.io/projected/afea1994-1cf2-4678-b897-aeaa41412fc1-kube-api-access-wwngc\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.713603 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712543    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc6eae27-5d1d-4957-99b5-e87c1d177376-tmp-dir\") pod \"node-resolver-92lzv\" (UID: \"fc6eae27-5d1d-4957-99b5-e87c1d177376\") " pod="openshift-dns/node-resolver-92lzv"
Sep 09 16:13:55.713603 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712567    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-run-openvswitch\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.713603 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712593    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4fce212-3111-43ec-8fee-a58bd7ffdff9-ovnkube-config\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.714560 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712637    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-host-run-k8s-cni-cncf-io\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.714560 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712662    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-etc-sysctl-d\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.714560 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712689    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-var-lib-kubelet\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.714560 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712689    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3aa9556d-7d31-44de-8861-6ef97bc87b66-etc-selinux\") pod \"aws-ebs-csi-driver-node-qsnkv\" (UID: \"3aa9556d-7d31-44de-8861-6ef97bc87b66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv"
Sep 09 16:13:55.714560 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712729    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-run-ovn\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.714560 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712731    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f37385e-b449-4281-b738-7edce088387c-host\") pod \"node-ca-hkc7k\" (UID: \"3f37385e-b449-4281-b738-7edce088387c\") " pod="openshift-image-registry/node-ca-hkc7k"
Sep 09 16:13:55.714560 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712745    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-cni-binary-copy\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.714560 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:55.712797    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:13:55.714560 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712799    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f37385e-b449-4281-b738-7edce088387c-host\") pod \"node-ca-hkc7k\" (UID: \"3f37385e-b449-4281-b738-7edce088387c\") " pod="openshift-image-registry/node-ca-hkc7k"
Sep 09 16:13:55.714560 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:55.712854    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs podName:7334d4d3-9b79-4b0d-ab28-e0aec88c18cb nodeName:}" failed. No retries permitted until 2026-09-09 16:13:56.212837386 +0000 UTC m=+2.043045425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs") pod "network-metrics-daemon-gcb4p" (UID: "7334d4d3-9b79-4b0d-ab28-e0aec88c18cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:13:55.714560 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712919    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-etc-systemd\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.714560 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712937    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-multus-daemon-config\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.714560 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712944    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-systemd-units\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.714560 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712980    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-etc-openvswitch\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.714560 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.712999    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-systemd-units\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.714560 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713010    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4fce212-3111-43ec-8fee-a58bd7ffdff9-env-overrides\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.714560 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713032    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bd350005-0808-410e-bef5-9ecfd227869d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6hgj9\" (UID: \"bd350005-0808-410e-bef5-9ecfd227869d\") " pod="openshift-multus/multus-additional-cni-plugins-6hgj9"
Sep 09 16:13:55.715453 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713045    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-etc-openvswitch\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.715453 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713051    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-sys\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.715453 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713073    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/98c72774-4b40-43b2-b4dd-7590320acc60-kubelet-config\") pod \"global-pull-secret-syncer-hjz7d\" (UID: \"98c72774-4b40-43b2-b4dd-7590320acc60\") " pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:13:55.715453 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713093    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-host-run-k8s-cni-cncf-io\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.715453 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713165    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3aa9556d-7d31-44de-8861-6ef97bc87b66-etc-selinux\") pod \"aws-ebs-csi-driver-node-qsnkv\" (UID: \"3aa9556d-7d31-44de-8861-6ef97bc87b66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv"
Sep 09 16:13:55.715453 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713212    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-run-openvswitch\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.715453 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713242    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/98c72774-4b40-43b2-b4dd-7590320acc60-kubelet-config\") pod \"global-pull-secret-syncer-hjz7d\" (UID: \"98c72774-4b40-43b2-b4dd-7590320acc60\") " pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:13:55.715453 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713265    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-sys\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.715453 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713364    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-etc-sysctl-d\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.715453 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713095    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3aa9556d-7d31-44de-8861-6ef97bc87b66-sys-fs\") pod \"aws-ebs-csi-driver-node-qsnkv\" (UID: \"3aa9556d-7d31-44de-8861-6ef97bc87b66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv"
Sep 09 16:13:55.715453 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713426    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-host-cni-bin\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.715453 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713455    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsdz6\" (UniqueName: \"kubernetes.io/projected/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-kube-api-access-bsdz6\") pod \"network-metrics-daemon-gcb4p\" (UID: \"7334d4d3-9b79-4b0d-ab28-e0aec88c18cb\") " pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:13:55.715453 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713486    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd350005-0808-410e-bef5-9ecfd227869d-cnibin\") pod \"multus-additional-cni-plugins-6hgj9\" (UID: \"bd350005-0808-410e-bef5-9ecfd227869d\") " pod="openshift-multus/multus-additional-cni-plugins-6hgj9"
Sep 09 16:13:55.715453 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713507    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4fce212-3111-43ec-8fee-a58bd7ffdff9-env-overrides\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.715453 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713545    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-etc-sysconfig\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.715453 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713567    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-host-cni-bin\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.715453 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713574    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/68024951-17f4-4df9-b48a-7674411fd492-agent-certs\") pod \"konnectivity-agent-9shcg\" (UID: \"68024951-17f4-4df9-b48a-7674411fd492\") " pod="kube-system/konnectivity-agent-9shcg"
Sep 09 16:13:55.716304 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713611    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd350005-0808-410e-bef5-9ecfd227869d-cnibin\") pod \"multus-additional-cni-plugins-6hgj9\" (UID: \"bd350005-0808-410e-bef5-9ecfd227869d\") " pod="openshift-multus/multus-additional-cni-plugins-6hgj9"
Sep 09 16:13:55.716304 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713617    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-node-log\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.716304 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713645    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bd350005-0808-410e-bef5-9ecfd227869d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6hgj9\" (UID: \"bd350005-0808-410e-bef5-9ecfd227869d\") " pod="openshift-multus/multus-additional-cni-plugins-6hgj9"
Sep 09 16:13:55.716304 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713681    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-node-log\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.716304 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713711    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3aa9556d-7d31-44de-8861-6ef97bc87b66-sys-fs\") pod \"aws-ebs-csi-driver-node-qsnkv\" (UID: \"3aa9556d-7d31-44de-8861-6ef97bc87b66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv"
Sep 09 16:13:55.716304 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713714    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-multus-socket-dir-parent\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.716304 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713747    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-host-run-netns\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.716304 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713770    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dqgw\" (UniqueName: \"kubernetes.io/projected/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-kube-api-access-2dqgw\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.716304 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713786    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4fce212-3111-43ec-8fee-a58bd7ffdff9-ovnkube-config\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.716304 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713796    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bd350005-0808-410e-bef5-9ecfd227869d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6hgj9\" (UID: \"bd350005-0808-410e-bef5-9ecfd227869d\") " pod="openshift-multus/multus-additional-cni-plugins-6hgj9"
Sep 09 16:13:55.716304 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713822    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f46d54d3-e17a-4ef8-9e1e-41cb4820d5fd-iptables-alerter-script\") pod \"iptables-alerter-rhdkg\" (UID: \"f46d54d3-e17a-4ef8-9e1e-41cb4820d5fd\") " pod="openshift-network-operator/iptables-alerter-rhdkg"
Sep 09 16:13:55.716304 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713849    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-etc-kubernetes\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.716304 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713856    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-etc-sysconfig\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.716304 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713874    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3aa9556d-7d31-44de-8861-6ef97bc87b66-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qsnkv\" (UID: \"3aa9556d-7d31-44de-8861-6ef97bc87b66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv"
Sep 09 16:13:55.716304 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713878    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-host-run-netns\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.716304 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713932    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3aa9556d-7d31-44de-8861-6ef97bc87b66-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qsnkv\" (UID: \"3aa9556d-7d31-44de-8861-6ef97bc87b66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv"
Sep 09 16:13:55.716304 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.713980    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-etc-kubernetes\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.716876 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.714026    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-host-var-lib-cni-bin\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.716876 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.714060    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-multus-conf-dir\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.716876 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.714085    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtk84\" (UniqueName: \"kubernetes.io/projected/f46d54d3-e17a-4ef8-9e1e-41cb4820d5fd-kube-api-access-vtk84\") pod \"iptables-alerter-rhdkg\" (UID: \"f46d54d3-e17a-4ef8-9e1e-41cb4820d5fd\") " pod="openshift-network-operator/iptables-alerter-rhdkg"
Sep 09 16:13:55.716876 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.714108    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-lib-modules\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.716876 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.714116    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-host-var-lib-cni-bin\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.716876 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.714137    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/68024951-17f4-4df9-b48a-7674411fd492-konnectivity-ca\") pod \"konnectivity-agent-9shcg\" (UID: \"68024951-17f4-4df9-b48a-7674411fd492\") " pod="kube-system/konnectivity-agent-9shcg"
Sep 09 16:13:55.716876 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.714166    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-host-run-ovn-kubernetes\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.716876 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.714180    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-multus-conf-dir\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.716876 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.714192    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc6eae27-5d1d-4957-99b5-e87c1d177376-hosts-file\") pod \"node-resolver-92lzv\" (UID: \"fc6eae27-5d1d-4957-99b5-e87c1d177376\") " pod="openshift-dns/node-resolver-92lzv"
Sep 09 16:13:55.716876 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.714217    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-cnibin\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.716876 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.714242    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd350005-0808-410e-bef5-9ecfd227869d-cni-binary-copy\") pod \"multus-additional-cni-plugins-6hgj9\" (UID: \"bd350005-0808-410e-bef5-9ecfd227869d\") " pod="openshift-multus/multus-additional-cni-plugins-6hgj9"
Sep 09 16:13:55.716876 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.714250    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-multus-socket-dir-parent\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.716876 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.714268    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cq97\" (UniqueName: \"kubernetes.io/projected/bd350005-0808-410e-bef5-9ecfd227869d-kube-api-access-8cq97\") pod \"multus-additional-cni-plugins-6hgj9\" (UID: \"bd350005-0808-410e-bef5-9ecfd227869d\") " pod="openshift-multus/multus-additional-cni-plugins-6hgj9"
Sep 09 16:13:55.716876 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.714449    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc6eae27-5d1d-4957-99b5-e87c1d177376-hosts-file\") pod \"node-resolver-92lzv\" (UID: \"fc6eae27-5d1d-4957-99b5-e87c1d177376\") " pod="openshift-dns/node-resolver-92lzv"
Sep 09 16:13:55.716876 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.714475    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bd350005-0808-410e-bef5-9ecfd227869d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6hgj9\" (UID: \"bd350005-0808-410e-bef5-9ecfd227869d\") " pod="openshift-multus/multus-additional-cni-plugins-6hgj9"
Sep 09 16:13:55.716876 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.714504    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4fce212-3111-43ec-8fee-a58bd7ffdff9-host-run-ovn-kubernetes\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.716876 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.714565    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-cnibin\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.717404 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.714804    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/68024951-17f4-4df9-b48a-7674411fd492-konnectivity-ca\") pod \"konnectivity-agent-9shcg\" (UID: \"68024951-17f4-4df9-b48a-7674411fd492\") " pod="kube-system/konnectivity-agent-9shcg"
Sep 09 16:13:55.717404 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.714863    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/afea1994-1cf2-4678-b897-aeaa41412fc1-lib-modules\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.717404 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.715022    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd350005-0808-410e-bef5-9ecfd227869d-cni-binary-copy\") pod \"multus-additional-cni-plugins-6hgj9\" (UID: \"bd350005-0808-410e-bef5-9ecfd227869d\") " pod="openshift-multus/multus-additional-cni-plugins-6hgj9"
Sep 09 16:13:55.717404 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.715071    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f46d54d3-e17a-4ef8-9e1e-41cb4820d5fd-iptables-alerter-script\") pod \"iptables-alerter-rhdkg\" (UID: \"f46d54d3-e17a-4ef8-9e1e-41cb4820d5fd\") " pod="openshift-network-operator/iptables-alerter-rhdkg"
Sep 09 16:13:55.717404 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.716256    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/afea1994-1cf2-4678-b897-aeaa41412fc1-etc-tuned\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.717404 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.716537    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4fce212-3111-43ec-8fee-a58bd7ffdff9-ovn-node-metrics-cert\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.717404 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.716543    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afea1994-1cf2-4678-b897-aeaa41412fc1-tmp\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.717404 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.716752    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/68024951-17f4-4df9-b48a-7674411fd492-agent-certs\") pod \"konnectivity-agent-9shcg\" (UID: \"68024951-17f4-4df9-b48a-7674411fd492\") " pod="kube-system/konnectivity-agent-9shcg"
Sep 09 16:13:55.719853 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:55.719832    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 16:13:55.719952 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:55.719861    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 16:13:55.719952 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:55.719876    2555 projected.go:196] Error preparing data for projected volume kube-api-access-cv7fn for pod openshift-network-diagnostics/network-check-target-slbqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:13:55.719952 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:55.719950    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1bd8aa2-35fd-458b-8a84-69ccf7a6276f-kube-api-access-cv7fn podName:f1bd8aa2-35fd-458b-8a84-69ccf7a6276f nodeName:}" failed. No retries permitted until 2026-09-09 16:13:56.219933047 +0000 UTC m=+2.050141113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cv7fn" (UniqueName: "kubernetes.io/projected/f1bd8aa2-35fd-458b-8a84-69ccf7a6276f-kube-api-access-cv7fn") pod "network-check-target-slbqw" (UID: "f1bd8aa2-35fd-458b-8a84-69ccf7a6276f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:13:55.720984 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.720910    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2pr9\" (UniqueName: \"kubernetes.io/projected/3aa9556d-7d31-44de-8861-6ef97bc87b66-kube-api-access-c2pr9\") pod \"aws-ebs-csi-driver-node-qsnkv\" (UID: \"3aa9556d-7d31-44de-8861-6ef97bc87b66\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv"
Sep 09 16:13:55.721644 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.721621    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b78gh\" (UniqueName: \"kubernetes.io/projected/e4fce212-3111-43ec-8fee-a58bd7ffdff9-kube-api-access-b78gh\") pod \"ovnkube-node-phs9w\" (UID: \"e4fce212-3111-43ec-8fee-a58bd7ffdff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.722857 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.722828    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52hd9\" (UniqueName: \"kubernetes.io/projected/fc6eae27-5d1d-4957-99b5-e87c1d177376-kube-api-access-52hd9\") pod \"node-resolver-92lzv\" (UID: \"fc6eae27-5d1d-4957-99b5-e87c1d177376\") " pod="openshift-dns/node-resolver-92lzv"
Sep 09 16:13:55.723685 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.723644    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dphp\" (UniqueName: \"kubernetes.io/projected/3f37385e-b449-4281-b738-7edce088387c-kube-api-access-4dphp\") pod \"node-ca-hkc7k\" (UID: \"3f37385e-b449-4281-b738-7edce088387c\") " pod="openshift-image-registry/node-ca-hkc7k"
Sep 09 16:13:55.723860 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.723834    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwngc\" (UniqueName: \"kubernetes.io/projected/afea1994-1cf2-4678-b897-aeaa41412fc1-kube-api-access-wwngc\") pod \"tuned-trqnv\" (UID: \"afea1994-1cf2-4678-b897-aeaa41412fc1\") " pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.723913 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.723868    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsdz6\" (UniqueName: \"kubernetes.io/projected/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-kube-api-access-bsdz6\") pod \"network-metrics-daemon-gcb4p\" (UID: \"7334d4d3-9b79-4b0d-ab28-e0aec88c18cb\") " pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:13:55.724066 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.724044    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtk84\" (UniqueName: \"kubernetes.io/projected/f46d54d3-e17a-4ef8-9e1e-41cb4820d5fd-kube-api-access-vtk84\") pod \"iptables-alerter-rhdkg\" (UID: \"f46d54d3-e17a-4ef8-9e1e-41cb4820d5fd\") " pod="openshift-network-operator/iptables-alerter-rhdkg"
Sep 09 16:13:55.724159 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.724139    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dqgw\" (UniqueName: \"kubernetes.io/projected/bf8b3ddd-efd5-4277-a2dc-c5e93467f290-kube-api-access-2dqgw\") pod \"multus-2d5w7\" (UID: \"bf8b3ddd-efd5-4277-a2dc-c5e93467f290\") " pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.724201 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.724183    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cq97\" (UniqueName: \"kubernetes.io/projected/bd350005-0808-410e-bef5-9ecfd227869d-kube-api-access-8cq97\") pod \"multus-additional-cni-plugins-6hgj9\" (UID: \"bd350005-0808-410e-bef5-9ecfd227869d\") " pod="openshift-multus/multus-additional-cni-plugins-6hgj9"
Sep 09 16:13:55.834414 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.834355    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6hgj9"
Sep 09 16:13:55.841079 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.841062    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rhdkg"
Sep 09 16:13:55.845713 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:55.845696    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd350005_0808_410e_bef5_9ecfd227869d.slice/crio-7223f7bba51c839c36da89b1938f9804736f5e7e086e79cda1145472298add79 WatchSource:0}: Error finding container 7223f7bba51c839c36da89b1938f9804736f5e7e086e79cda1145472298add79: Status 404 returned error can't find the container with id 7223f7bba51c839c36da89b1938f9804736f5e7e086e79cda1145472298add79
Sep 09 16:13:55.850299 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:55.850279    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46d54d3_e17a_4ef8_9e1e_41cb4820d5fd.slice/crio-075ceb7b14b72c49c6ce4de0d189a9e330336cf6260faa6a04cc4e9fc0ad6fd5 WatchSource:0}: Error finding container 075ceb7b14b72c49c6ce4de0d189a9e330336cf6260faa6a04cc4e9fc0ad6fd5: Status 404 returned error can't find the container with id 075ceb7b14b72c49c6ce4de0d189a9e330336cf6260faa6a04cc4e9fc0ad6fd5
Sep 09 16:13:55.865238 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.865219    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv"
Sep 09 16:13:55.874416 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:55.874394    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa9556d_7d31_44de_8861_6ef97bc87b66.slice/crio-a06c2c20b16fd6901827b103dc4b13e35a38191df62f907c9981cfe69e5e8680 WatchSource:0}: Error finding container a06c2c20b16fd6901827b103dc4b13e35a38191df62f907c9981cfe69e5e8680: Status 404 returned error can't find the container with id a06c2c20b16fd6901827b103dc4b13e35a38191df62f907c9981cfe69e5e8680
Sep 09 16:13:55.885906 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.885888    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-trqnv"
Sep 09 16:13:55.891055 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.891038    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hkc7k"
Sep 09 16:13:55.893855 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:55.893748    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafea1994_1cf2_4678_b897_aeaa41412fc1.slice/crio-16d167de62a7d2621807ca8d59c0b296d08217fd99cdcfc1b4794081555cdd6c WatchSource:0}: Error finding container 16d167de62a7d2621807ca8d59c0b296d08217fd99cdcfc1b4794081555cdd6c: Status 404 returned error can't find the container with id 16d167de62a7d2621807ca8d59c0b296d08217fd99cdcfc1b4794081555cdd6c
Sep 09 16:13:55.900111 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:55.900092    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f37385e_b449_4281_b738_7edce088387c.slice/crio-4641b286cf064311c89e348d4ee295a55757797041a13a14dcf4af820a046b8f WatchSource:0}: Error finding container 4641b286cf064311c89e348d4ee295a55757797041a13a14dcf4af820a046b8f: Status 404 returned error can't find the container with id 4641b286cf064311c89e348d4ee295a55757797041a13a14dcf4af820a046b8f
Sep 09 16:13:55.913515 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.913494    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:13:55.922363 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:55.922341    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4fce212_3111_43ec_8fee_a58bd7ffdff9.slice/crio-15d07a6da32aedb1bfb025376140ac01f886b1840d2855d6fae513123a2e32df WatchSource:0}: Error finding container 15d07a6da32aedb1bfb025376140ac01f886b1840d2855d6fae513123a2e32df: Status 404 returned error can't find the container with id 15d07a6da32aedb1bfb025376140ac01f886b1840d2855d6fae513123a2e32df
Sep 09 16:13:55.927589 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.927573    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9shcg"
Sep 09 16:13:55.933649 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.933626    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-92lzv"
Sep 09 16:13:55.935488 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:55.935467    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68024951_17f4_4df9_b48a_7674411fd492.slice/crio-f646f9de0a6990fd565952052e31e9fc2444fba3d07df2819f1fff43a3b9e242 WatchSource:0}: Error finding container f646f9de0a6990fd565952052e31e9fc2444fba3d07df2819f1fff43a3b9e242: Status 404 returned error can't find the container with id f646f9de0a6990fd565952052e31e9fc2444fba3d07df2819f1fff43a3b9e242
Sep 09 16:13:55.938563 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:55.938542    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2d5w7"
Sep 09 16:13:55.947383 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:13:55.947355    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8b3ddd_efd5_4277_a2dc_c5e93467f290.slice/crio-25aa4fd87117228651865cfa30896d34412a97cb83db256601af8a79843e8129 WatchSource:0}: Error finding container 25aa4fd87117228651865cfa30896d34412a97cb83db256601af8a79843e8129: Status 404 returned error can't find the container with id 25aa4fd87117228651865cfa30896d34412a97cb83db256601af8a79843e8129
Sep 09 16:13:56.219290 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:56.219181    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs\") pod \"network-metrics-daemon-gcb4p\" (UID: \"7334d4d3-9b79-4b0d-ab28-e0aec88c18cb\") " pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:13:56.219290 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:56.219259    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98c72774-4b40-43b2-b4dd-7590320acc60-original-pull-secret\") pod \"global-pull-secret-syncer-hjz7d\" (UID: \"98c72774-4b40-43b2-b4dd-7590320acc60\") " pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:13:56.219514 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:56.219379    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:13:56.219514 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:56.219448    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 16:13:56.219514 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:56.219458    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs podName:7334d4d3-9b79-4b0d-ab28-e0aec88c18cb nodeName:}" failed. No retries permitted until 2026-09-09 16:13:57.219437826 +0000 UTC m=+3.049645868 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs") pod "network-metrics-daemon-gcb4p" (UID: "7334d4d3-9b79-4b0d-ab28-e0aec88c18cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:13:56.219666 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:56.219541    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98c72774-4b40-43b2-b4dd-7590320acc60-original-pull-secret podName:98c72774-4b40-43b2-b4dd-7590320acc60 nodeName:}" failed. No retries permitted until 2026-09-09 16:13:57.219491895 +0000 UTC m=+3.049699957 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98c72774-4b40-43b2-b4dd-7590320acc60-original-pull-secret") pod "global-pull-secret-syncer-hjz7d" (UID: "98c72774-4b40-43b2-b4dd-7590320acc60") : object "kube-system"/"original-pull-secret" not registered
Sep 09 16:13:56.320567 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:56.320534    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv7fn\" (UniqueName: \"kubernetes.io/projected/f1bd8aa2-35fd-458b-8a84-69ccf7a6276f-kube-api-access-cv7fn\") pod \"network-check-target-slbqw\" (UID: \"f1bd8aa2-35fd-458b-8a84-69ccf7a6276f\") " pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:13:56.320756 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:56.320707    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 16:13:56.320756 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:56.320731    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 16:13:56.320756 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:56.320744    2555 projected.go:196] Error preparing data for projected volume kube-api-access-cv7fn for pod openshift-network-diagnostics/network-check-target-slbqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:13:56.320916 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:56.320815    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1bd8aa2-35fd-458b-8a84-69ccf7a6276f-kube-api-access-cv7fn podName:f1bd8aa2-35fd-458b-8a84-69ccf7a6276f nodeName:}" failed. No retries permitted until 2026-09-09 16:13:57.320793651 +0000 UTC m=+3.151001709 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cv7fn" (UniqueName: "kubernetes.io/projected/f1bd8aa2-35fd-458b-8a84-69ccf7a6276f-kube-api-access-cv7fn") pod "network-check-target-slbqw" (UID: "f1bd8aa2-35fd-458b-8a84-69ccf7a6276f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:13:56.554433 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:56.549517    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 16:08:56 +0000 UTC" deadline="2028-05-13 15:07:20.912750797 +0000 UTC"
Sep 09 16:13:56.554433 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:56.549559    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14686h53m24.363197334s"
Sep 09 16:13:56.556676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:56.556649    2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 16:13:56.641837 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:56.641427    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hkc7k" event={"ID":"3f37385e-b449-4281-b738-7edce088387c","Type":"ContainerStarted","Data":"4641b286cf064311c89e348d4ee295a55757797041a13a14dcf4af820a046b8f"}
Sep 09 16:13:56.660000 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:56.659961    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-trqnv" event={"ID":"afea1994-1cf2-4678-b897-aeaa41412fc1","Type":"ContainerStarted","Data":"16d167de62a7d2621807ca8d59c0b296d08217fd99cdcfc1b4794081555cdd6c"}
Sep 09 16:13:56.673648 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:56.673565    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rhdkg" event={"ID":"f46d54d3-e17a-4ef8-9e1e-41cb4820d5fd","Type":"ContainerStarted","Data":"075ceb7b14b72c49c6ce4de0d189a9e330336cf6260faa6a04cc4e9fc0ad6fd5"}
Sep 09 16:13:56.679227 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:56.679154    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hgj9" event={"ID":"bd350005-0808-410e-bef5-9ecfd227869d","Type":"ContainerStarted","Data":"7223f7bba51c839c36da89b1938f9804736f5e7e086e79cda1145472298add79"}
Sep 09 16:13:56.689316 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:56.689249    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-202.ec2.internal" event={"ID":"a40d2b498bc21892b32d687ae581fc71","Type":"ContainerStarted","Data":"a37e3fcb6fbbea30944c9a34d57e46ad9e2789123351ff2b9faf167ef9e0648f"}
Sep 09 16:13:56.709582 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:56.709532    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2d5w7" event={"ID":"bf8b3ddd-efd5-4277-a2dc-c5e93467f290","Type":"ContainerStarted","Data":"25aa4fd87117228651865cfa30896d34412a97cb83db256601af8a79843e8129"}
Sep 09 16:13:56.719988 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:56.719962    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv" event={"ID":"3aa9556d-7d31-44de-8861-6ef97bc87b66","Type":"ContainerStarted","Data":"a06c2c20b16fd6901827b103dc4b13e35a38191df62f907c9981cfe69e5e8680"}
Sep 09 16:13:56.725366 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:56.725339    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-202.ec2.internal" event={"ID":"1489a50f476055d1d230538a489fe1f4","Type":"ContainerStarted","Data":"3a3cbc51c277b5a93ffe0ce546aa15223137a7a8a72ea5b198a86751b3a0301f"}
Sep 09 16:13:56.735503 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:56.735473    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-92lzv" event={"ID":"fc6eae27-5d1d-4957-99b5-e87c1d177376","Type":"ContainerStarted","Data":"d4de146fcec9d096dce08b659b4ebf7a70db4984a66848c1ec830ffe42e8a4d9"}
Sep 09 16:13:56.742911 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:56.742876    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9shcg" event={"ID":"68024951-17f4-4df9-b48a-7674411fd492","Type":"ContainerStarted","Data":"f646f9de0a6990fd565952052e31e9fc2444fba3d07df2819f1fff43a3b9e242"}
Sep 09 16:13:56.755942 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:56.755845    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phs9w" event={"ID":"e4fce212-3111-43ec-8fee-a58bd7ffdff9","Type":"ContainerStarted","Data":"15d07a6da32aedb1bfb025376140ac01f886b1840d2855d6fae513123a2e32df"}
Sep 09 16:13:56.910220 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:56.909834    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 16:13:57.227156 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:57.227068    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs\") pod \"network-metrics-daemon-gcb4p\" (UID: \"7334d4d3-9b79-4b0d-ab28-e0aec88c18cb\") " pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:13:57.227339 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:57.227154    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98c72774-4b40-43b2-b4dd-7590320acc60-original-pull-secret\") pod \"global-pull-secret-syncer-hjz7d\" (UID: \"98c72774-4b40-43b2-b4dd-7590320acc60\") " pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:13:57.227444 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:57.227320    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 16:13:57.227444 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:57.227403    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98c72774-4b40-43b2-b4dd-7590320acc60-original-pull-secret podName:98c72774-4b40-43b2-b4dd-7590320acc60 nodeName:}" failed. No retries permitted until 2026-09-09 16:13:59.227382564 +0000 UTC m=+5.057590606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98c72774-4b40-43b2-b4dd-7590320acc60-original-pull-secret") pod "global-pull-secret-syncer-hjz7d" (UID: "98c72774-4b40-43b2-b4dd-7590320acc60") : object "kube-system"/"original-pull-secret" not registered
Sep 09 16:13:57.227579 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:57.227477    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:13:57.227579 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:57.227517    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs podName:7334d4d3-9b79-4b0d-ab28-e0aec88c18cb nodeName:}" failed. No retries permitted until 2026-09-09 16:13:59.227504472 +0000 UTC m=+5.057712515 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs") pod "network-metrics-daemon-gcb4p" (UID: "7334d4d3-9b79-4b0d-ab28-e0aec88c18cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:13:57.327577 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:57.327537    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv7fn\" (UniqueName: \"kubernetes.io/projected/f1bd8aa2-35fd-458b-8a84-69ccf7a6276f-kube-api-access-cv7fn\") pod \"network-check-target-slbqw\" (UID: \"f1bd8aa2-35fd-458b-8a84-69ccf7a6276f\") " pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:13:57.327792 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:57.327773    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 16:13:57.327861 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:57.327801    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 16:13:57.327861 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:57.327816    2555 projected.go:196] Error preparing data for projected volume kube-api-access-cv7fn for pod openshift-network-diagnostics/network-check-target-slbqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:13:57.327972 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:57.327879    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1bd8aa2-35fd-458b-8a84-69ccf7a6276f-kube-api-access-cv7fn podName:f1bd8aa2-35fd-458b-8a84-69ccf7a6276f nodeName:}" failed. No retries permitted until 2026-09-09 16:13:59.327858885 +0000 UTC m=+5.158066949 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cv7fn" (UniqueName: "kubernetes.io/projected/f1bd8aa2-35fd-458b-8a84-69ccf7a6276f-kube-api-access-cv7fn") pod "network-check-target-slbqw" (UID: "f1bd8aa2-35fd-458b-8a84-69ccf7a6276f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:13:57.550642 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:57.550525    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 16:08:56 +0000 UTC" deadline="2028-05-29 22:07:44.307064922 +0000 UTC"
Sep 09 16:13:57.550642 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:57.550590    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15077h53m46.756478689s"
Sep 09 16:13:57.601664 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:57.601633    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:13:57.602135 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:57.601780    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slbqw" podUID="f1bd8aa2-35fd-458b-8a84-69ccf7a6276f"
Sep 09 16:13:57.602303 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:57.602272    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:13:57.602470 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:57.602446    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gcb4p" podUID="7334d4d3-9b79-4b0d-ab28-e0aec88c18cb"
Sep 09 16:13:57.602551 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:57.602526    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:13:57.602619 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:57.602602    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjz7d" podUID="98c72774-4b40-43b2-b4dd-7590320acc60"
Sep 09 16:13:57.670886 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:57.670856    2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 16:13:58.063963 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:58.063934    2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 16:13:59.246626 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:59.245730    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98c72774-4b40-43b2-b4dd-7590320acc60-original-pull-secret\") pod \"global-pull-secret-syncer-hjz7d\" (UID: \"98c72774-4b40-43b2-b4dd-7590320acc60\") " pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:13:59.246626 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:59.245811    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs\") pod \"network-metrics-daemon-gcb4p\" (UID: \"7334d4d3-9b79-4b0d-ab28-e0aec88c18cb\") " pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:13:59.246626 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:59.245960    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:13:59.246626 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:59.246030    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs podName:7334d4d3-9b79-4b0d-ab28-e0aec88c18cb nodeName:}" failed. No retries permitted until 2026-09-09 16:14:03.246008456 +0000 UTC m=+9.076216518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs") pod "network-metrics-daemon-gcb4p" (UID: "7334d4d3-9b79-4b0d-ab28-e0aec88c18cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:13:59.246626 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:59.246486    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 16:13:59.246626 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:59.246541    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98c72774-4b40-43b2-b4dd-7590320acc60-original-pull-secret podName:98c72774-4b40-43b2-b4dd-7590320acc60 nodeName:}" failed. No retries permitted until 2026-09-09 16:14:03.246524052 +0000 UTC m=+9.076732093 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98c72774-4b40-43b2-b4dd-7590320acc60-original-pull-secret") pod "global-pull-secret-syncer-hjz7d" (UID: "98c72774-4b40-43b2-b4dd-7590320acc60") : object "kube-system"/"original-pull-secret" not registered
Sep 09 16:13:59.347349 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:59.346675    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv7fn\" (UniqueName: \"kubernetes.io/projected/f1bd8aa2-35fd-458b-8a84-69ccf7a6276f-kube-api-access-cv7fn\") pod \"network-check-target-slbqw\" (UID: \"f1bd8aa2-35fd-458b-8a84-69ccf7a6276f\") " pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:13:59.347349 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:59.346871    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 16:13:59.347349 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:59.346895    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 16:13:59.347349 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:59.346910    2555 projected.go:196] Error preparing data for projected volume kube-api-access-cv7fn for pod openshift-network-diagnostics/network-check-target-slbqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:13:59.347349 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:59.346973    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1bd8aa2-35fd-458b-8a84-69ccf7a6276f-kube-api-access-cv7fn podName:f1bd8aa2-35fd-458b-8a84-69ccf7a6276f nodeName:}" failed. No retries permitted until 2026-09-09 16:14:03.3469518 +0000 UTC m=+9.177159853 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cv7fn" (UniqueName: "kubernetes.io/projected/f1bd8aa2-35fd-458b-8a84-69ccf7a6276f-kube-api-access-cv7fn") pod "network-check-target-slbqw" (UID: "f1bd8aa2-35fd-458b-8a84-69ccf7a6276f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:13:59.602065 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:59.601977    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:13:59.602221 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:59.602130    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gcb4p" podUID="7334d4d3-9b79-4b0d-ab28-e0aec88c18cb"
Sep 09 16:13:59.602863 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:59.602544    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:13:59.602863 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:59.602656    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjz7d" podUID="98c72774-4b40-43b2-b4dd-7590320acc60"
Sep 09 16:13:59.602863 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:13:59.602709    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:13:59.602863 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:13:59.602779    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slbqw" podUID="f1bd8aa2-35fd-458b-8a84-69ccf7a6276f"
Sep 09 16:14:01.602185 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:01.601805    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:14:01.602185 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:01.601952    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slbqw" podUID="f1bd8aa2-35fd-458b-8a84-69ccf7a6276f"
Sep 09 16:14:01.602185 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:01.602043    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:14:01.602185 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:01.602139    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gcb4p" podUID="7334d4d3-9b79-4b0d-ab28-e0aec88c18cb"
Sep 09 16:14:01.602888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:01.602228    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:14:01.602888 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:01.602349    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjz7d" podUID="98c72774-4b40-43b2-b4dd-7590320acc60"
Sep 09 16:14:03.281443 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:03.281401    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98c72774-4b40-43b2-b4dd-7590320acc60-original-pull-secret\") pod \"global-pull-secret-syncer-hjz7d\" (UID: \"98c72774-4b40-43b2-b4dd-7590320acc60\") " pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:14:03.282032 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:03.281478    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs\") pod \"network-metrics-daemon-gcb4p\" (UID: \"7334d4d3-9b79-4b0d-ab28-e0aec88c18cb\") " pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:14:03.282032 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:03.281591    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 16:14:03.282032 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:03.281599    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:14:03.282032 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:03.281662    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs podName:7334d4d3-9b79-4b0d-ab28-e0aec88c18cb nodeName:}" failed. No retries permitted until 2026-09-09 16:14:11.281643371 +0000 UTC m=+17.111851408 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs") pod "network-metrics-daemon-gcb4p" (UID: "7334d4d3-9b79-4b0d-ab28-e0aec88c18cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:14:03.282032 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:03.281683    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98c72774-4b40-43b2-b4dd-7590320acc60-original-pull-secret podName:98c72774-4b40-43b2-b4dd-7590320acc60 nodeName:}" failed. No retries permitted until 2026-09-09 16:14:11.281673339 +0000 UTC m=+17.111881378 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98c72774-4b40-43b2-b4dd-7590320acc60-original-pull-secret") pod "global-pull-secret-syncer-hjz7d" (UID: "98c72774-4b40-43b2-b4dd-7590320acc60") : object "kube-system"/"original-pull-secret" not registered
Sep 09 16:14:03.383579 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:03.382799    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv7fn\" (UniqueName: \"kubernetes.io/projected/f1bd8aa2-35fd-458b-8a84-69ccf7a6276f-kube-api-access-cv7fn\") pod \"network-check-target-slbqw\" (UID: \"f1bd8aa2-35fd-458b-8a84-69ccf7a6276f\") " pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:14:03.383579 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:03.382963    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 16:14:03.383579 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:03.382983    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 16:14:03.383579 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:03.382997    2555 projected.go:196] Error preparing data for projected volume kube-api-access-cv7fn for pod openshift-network-diagnostics/network-check-target-slbqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:14:03.383579 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:03.383060    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1bd8aa2-35fd-458b-8a84-69ccf7a6276f-kube-api-access-cv7fn podName:f1bd8aa2-35fd-458b-8a84-69ccf7a6276f nodeName:}" failed. No retries permitted until 2026-09-09 16:14:11.383040402 +0000 UTC m=+17.213248443 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cv7fn" (UniqueName: "kubernetes.io/projected/f1bd8aa2-35fd-458b-8a84-69ccf7a6276f-kube-api-access-cv7fn") pod "network-check-target-slbqw" (UID: "f1bd8aa2-35fd-458b-8a84-69ccf7a6276f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:14:03.601492 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:03.601405    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:14:03.601492 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:03.601439    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:14:03.601492 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:03.601439    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:14:03.601776 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:03.601535    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjz7d" podUID="98c72774-4b40-43b2-b4dd-7590320acc60"
Sep 09 16:14:03.601776 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:03.601647    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gcb4p" podUID="7334d4d3-9b79-4b0d-ab28-e0aec88c18cb"
Sep 09 16:14:03.601776 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:03.601731    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slbqw" podUID="f1bd8aa2-35fd-458b-8a84-69ccf7a6276f"
Sep 09 16:14:05.601737 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:05.601694    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:14:05.602096 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:05.601742    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:14:05.602096 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:05.601694    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:14:05.602096 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:05.601848    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gcb4p" podUID="7334d4d3-9b79-4b0d-ab28-e0aec88c18cb"
Sep 09 16:14:05.602096 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:05.601959    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slbqw" podUID="f1bd8aa2-35fd-458b-8a84-69ccf7a6276f"
Sep 09 16:14:05.602096 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:05.602053    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjz7d" podUID="98c72774-4b40-43b2-b4dd-7590320acc60"
Sep 09 16:14:07.601390 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:07.601353    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:14:07.601771 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:07.601465    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slbqw" podUID="f1bd8aa2-35fd-458b-8a84-69ccf7a6276f"
Sep 09 16:14:07.601771 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:07.601364    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:14:07.601771 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:07.601353    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:14:07.601771 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:07.601562    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjz7d" podUID="98c72774-4b40-43b2-b4dd-7590320acc60"
Sep 09 16:14:07.601771 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:07.601620    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gcb4p" podUID="7334d4d3-9b79-4b0d-ab28-e0aec88c18cb"
Sep 09 16:14:09.602391 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:09.602351    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:14:09.602836 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:09.602356    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:14:09.602836 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:09.602485    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gcb4p" podUID="7334d4d3-9b79-4b0d-ab28-e0aec88c18cb"
Sep 09 16:14:09.602836 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:09.602563    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjz7d" podUID="98c72774-4b40-43b2-b4dd-7590320acc60"
Sep 09 16:14:09.602836 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:09.602356    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:14:09.602836 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:09.602652    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slbqw" podUID="f1bd8aa2-35fd-458b-8a84-69ccf7a6276f"
Sep 09 16:14:11.340592 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:11.340518    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98c72774-4b40-43b2-b4dd-7590320acc60-original-pull-secret\") pod \"global-pull-secret-syncer-hjz7d\" (UID: \"98c72774-4b40-43b2-b4dd-7590320acc60\") " pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:14:11.340592 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:11.340581    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs\") pod \"network-metrics-daemon-gcb4p\" (UID: \"7334d4d3-9b79-4b0d-ab28-e0aec88c18cb\") " pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:14:11.341106 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:11.340654    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 16:14:11.341106 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:11.340683    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:14:11.341106 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:11.340719    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98c72774-4b40-43b2-b4dd-7590320acc60-original-pull-secret podName:98c72774-4b40-43b2-b4dd-7590320acc60 nodeName:}" failed. No retries permitted until 2026-09-09 16:14:27.34070383 +0000 UTC m=+33.170911870 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98c72774-4b40-43b2-b4dd-7590320acc60-original-pull-secret") pod "global-pull-secret-syncer-hjz7d" (UID: "98c72774-4b40-43b2-b4dd-7590320acc60") : object "kube-system"/"original-pull-secret" not registered
Sep 09 16:14:11.341106 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:11.340735    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs podName:7334d4d3-9b79-4b0d-ab28-e0aec88c18cb nodeName:}" failed. No retries permitted until 2026-09-09 16:14:27.340728162 +0000 UTC m=+33.170936198 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs") pod "network-metrics-daemon-gcb4p" (UID: "7334d4d3-9b79-4b0d-ab28-e0aec88c18cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:14:11.441495 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:11.441461    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv7fn\" (UniqueName: \"kubernetes.io/projected/f1bd8aa2-35fd-458b-8a84-69ccf7a6276f-kube-api-access-cv7fn\") pod \"network-check-target-slbqw\" (UID: \"f1bd8aa2-35fd-458b-8a84-69ccf7a6276f\") " pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:14:11.441674 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:11.441620    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 16:14:11.441674 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:11.441641    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 16:14:11.441674 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:11.441651    2555 projected.go:196] Error preparing data for projected volume kube-api-access-cv7fn for pod openshift-network-diagnostics/network-check-target-slbqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:14:11.441829 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:11.441701    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1bd8aa2-35fd-458b-8a84-69ccf7a6276f-kube-api-access-cv7fn podName:f1bd8aa2-35fd-458b-8a84-69ccf7a6276f nodeName:}" failed. No retries permitted until 2026-09-09 16:14:27.441686375 +0000 UTC m=+33.271894425 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cv7fn" (UniqueName: "kubernetes.io/projected/f1bd8aa2-35fd-458b-8a84-69ccf7a6276f-kube-api-access-cv7fn") pod "network-check-target-slbqw" (UID: "f1bd8aa2-35fd-458b-8a84-69ccf7a6276f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:14:11.602428 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:11.602344    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:14:11.602428 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:11.602384    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:14:11.602637 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:11.602480    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gcb4p" podUID="7334d4d3-9b79-4b0d-ab28-e0aec88c18cb"
Sep 09 16:14:11.602637 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:11.602543    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slbqw" podUID="f1bd8aa2-35fd-458b-8a84-69ccf7a6276f"
Sep 09 16:14:11.602637 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:11.602605    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:14:11.602786 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:11.602684    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjz7d" podUID="98c72774-4b40-43b2-b4dd-7590320acc60"
Sep 09 16:14:13.601734 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:13.601695    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:14:13.602143 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:13.601695    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:14:13.602143 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:13.601832    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gcb4p" podUID="7334d4d3-9b79-4b0d-ab28-e0aec88c18cb"
Sep 09 16:14:13.602143 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:13.601705    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:14:13.602143 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:13.601894    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slbqw" podUID="f1bd8aa2-35fd-458b-8a84-69ccf7a6276f"
Sep 09 16:14:13.602143 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:13.601986    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjz7d" podUID="98c72774-4b40-43b2-b4dd-7590320acc60"
Sep 09 16:14:15.602121 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:15.601710    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:14:15.602480 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:15.601718    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:14:15.602480 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:15.602220    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slbqw" podUID="f1bd8aa2-35fd-458b-8a84-69ccf7a6276f"
Sep 09 16:14:15.602480 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:15.602253    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjz7d" podUID="98c72774-4b40-43b2-b4dd-7590320acc60"
Sep 09 16:14:15.602480 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:15.601723    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:14:15.602480 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:15.602338    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gcb4p" podUID="7334d4d3-9b79-4b0d-ab28-e0aec88c18cb"
Sep 09 16:14:15.804966 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:15.804932    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-92lzv" event={"ID":"fc6eae27-5d1d-4957-99b5-e87c1d177376","Type":"ContainerStarted","Data":"bca8825c74ed84c35a014aaf11e26d766e1996e886598c7fd593c69bddb94901"}
Sep 09 16:14:15.807381 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:15.807346    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9shcg" event={"ID":"68024951-17f4-4df9-b48a-7674411fd492","Type":"ContainerStarted","Data":"a0e3de3a400249006cb9f3c395317b5bc616089ec285d54839f67f8914c81b87"}
Sep 09 16:14:15.810138 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:15.810108    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phs9w" event={"ID":"e4fce212-3111-43ec-8fee-a58bd7ffdff9","Type":"ContainerStarted","Data":"9bbd84ec094cc3bf2736a76afa5aea6d8937405654894984688e6313175547ff"}
Sep 09 16:14:15.810189 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:15.810177    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phs9w" event={"ID":"e4fce212-3111-43ec-8fee-a58bd7ffdff9","Type":"ContainerStarted","Data":"2ea3849097aa942c8b60f1e9566244c5832c95181b5fe636c73860f3c63118b6"}
Sep 09 16:14:15.810240 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:15.810224    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phs9w" event={"ID":"e4fce212-3111-43ec-8fee-a58bd7ffdff9","Type":"ContainerStarted","Data":"40a02d68b7b2ee5405d6a515cfb70ea41ff4a7740e14b00047cd91df657a2728"}
Sep 09 16:14:15.810315 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:15.810266    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phs9w" event={"ID":"e4fce212-3111-43ec-8fee-a58bd7ffdff9","Type":"ContainerStarted","Data":"1fb191e61e89fcf86a9ad27b3b60f37f6b69642cb0fbf39d4cd3f3f743e4987d"}
Sep 09 16:14:15.811513 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:15.811494    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hkc7k" event={"ID":"3f37385e-b449-4281-b738-7edce088387c","Type":"ContainerStarted","Data":"c11b3ffccd2aae24ebf1c7df3a903402d4d9ea1ecc06ba48d5c1671255cf0d8c"}
Sep 09 16:14:15.813022 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:15.813001    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-trqnv" event={"ID":"afea1994-1cf2-4678-b897-aeaa41412fc1","Type":"ContainerStarted","Data":"48fd5cb4a478e9ffb646f549863606f673a4fbdd9787092c66ab20cb932d1798"}
Sep 09 16:14:15.814413 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:15.814384    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hgj9" event={"ID":"bd350005-0808-410e-bef5-9ecfd227869d","Type":"ContainerStarted","Data":"5ea1fb716930c934436f54a86a760625b576bf866822ef01a95a261bcb1b4f9b"}
Sep 09 16:14:15.815592 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:15.815574    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-202.ec2.internal" event={"ID":"a40d2b498bc21892b32d687ae581fc71","Type":"ContainerStarted","Data":"b0e00076297c5cbed018ff784e09e1c37b9388d6cf83bb9183da6427aecdf23f"}
Sep 09 16:14:15.816900 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:15.816881    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2d5w7" event={"ID":"bf8b3ddd-efd5-4277-a2dc-c5e93467f290","Type":"ContainerStarted","Data":"0e5d07bf4b986315aa0082853be2366a940f1da81b38ebffaea5e21f65839f83"}
Sep 09 16:14:15.818346 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:15.818322    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv" event={"ID":"3aa9556d-7d31-44de-8861-6ef97bc87b66","Type":"ContainerStarted","Data":"49e93974fe08f14f1a6f3c4a6e3044e4ac00d22683861dd44328309448a03966"}
Sep 09 16:14:15.819471 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:15.819427    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-92lzv" podStartSLOduration=1.693837549 podStartE2EDuration="20.819412058s" totalImagesPullingTime="19.125574509s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:15.819176021 +0000 UTC m=+21.649384080" watchObservedRunningTime="2026-09-09 16:14:15.819412058 +0000 UTC m=+21.649620116"
Sep 09 16:14:15.832184 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:15.832148    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-202.ec2.internal" podStartSLOduration=19.832136227 podStartE2EDuration="19.832136227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:15.831602848 +0000 UTC m=+21.661810905" watchObservedRunningTime="2026-09-09 16:14:15.832136227 +0000 UTC m=+21.662344285"
Sep 09 16:14:15.866559 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:15.866519    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-trqnv" podStartSLOduration=1.692688019 podStartE2EDuration="20.866505718s" totalImagesPullingTime="19.173817699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:15.866201278 +0000 UTC m=+21.696409336" watchObservedRunningTime="2026-09-09 16:14:15.866505718 +0000 UTC m=+21.696713776"
Sep 09 16:14:15.881333 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:15.881255    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9shcg" podStartSLOduration=1.770387506 podStartE2EDuration="20.881242561s" totalImagesPullingTime="19.110855055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:15.88093427 +0000 UTC m=+21.711142328" watchObservedRunningTime="2026-09-09 16:14:15.881242561 +0000 UTC m=+21.711450619"
Sep 09 16:14:15.896723 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:15.896686    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2d5w7" podStartSLOduration=1.742466688 podStartE2EDuration="20.896674544s" totalImagesPullingTime="19.154207856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:15.896545408 +0000 UTC m=+21.726753476" watchObservedRunningTime="2026-09-09 16:14:15.896674544 +0000 UTC m=+21.726882599"
Sep 09 16:14:15.909809 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:15.909773    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hkc7k" podStartSLOduration=1.7790170220000001 podStartE2EDuration="20.909759432s" totalImagesPullingTime="19.13074241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:15.909422482 +0000 UTC m=+21.739630540" watchObservedRunningTime="2026-09-09 16:14:15.909759432 +0000 UTC m=+21.739967489"
Sep 09 16:14:16.824728 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:16.824683    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phs9w" event={"ID":"e4fce212-3111-43ec-8fee-a58bd7ffdff9","Type":"ContainerStarted","Data":"0f890243fe9e5a5ee7897592cb5c229e3582cb5c56536a0ac0fbd14cf8f7f702"}
Sep 09 16:14:16.825589 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:16.824771    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phs9w" event={"ID":"e4fce212-3111-43ec-8fee-a58bd7ffdff9","Type":"ContainerStarted","Data":"13d8296415a25cc275a0a5a91030956973efcf7e1eba87c1a0f3c8dbd6e10dd6"}
Sep 09 16:14:16.826101 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:16.826081    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rhdkg" event={"ID":"f46d54d3-e17a-4ef8-9e1e-41cb4820d5fd","Type":"ContainerStarted","Data":"db1a99f6b26bb2a06afa253549552b2c9a013dec58554546cf457e36612c5db1"}
Sep 09 16:14:16.827529 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:16.827506    2555 generic.go:356] "Generic (PLEG): container finished" podID="bd350005-0808-410e-bef5-9ecfd227869d" containerID="5ea1fb716930c934436f54a86a760625b576bf866822ef01a95a261bcb1b4f9b" exitCode=0
Sep 09 16:14:16.827649 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:16.827575    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hgj9" event={"ID":"bd350005-0808-410e-bef5-9ecfd227869d","Type":"ContainerDied","Data":"5ea1fb716930c934436f54a86a760625b576bf866822ef01a95a261bcb1b4f9b"}
Sep 09 16:14:16.828979 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:16.828955    2555 generic.go:356] "Generic (PLEG): container finished" podID="1489a50f476055d1d230538a489fe1f4" containerID="e92f8be757ad648c37869dbddd7c058cf8605058fc7a02a1761ef8d4c5804127" exitCode=0
Sep 09 16:14:16.829122 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:16.829097    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-202.ec2.internal" event={"ID":"1489a50f476055d1d230538a489fe1f4","Type":"ContainerDied","Data":"e92f8be757ad648c37869dbddd7c058cf8605058fc7a02a1761ef8d4c5804127"}
Sep 09 16:14:16.840629 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:16.840562    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rhdkg" podStartSLOduration=2.644539076 podStartE2EDuration="21.840538771s" totalImagesPullingTime="19.195999695s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:16.839742431 +0000 UTC m=+22.669950500" watchObservedRunningTime="2026-09-09 16:14:16.840538771 +0000 UTC m=+22.670746831"
Sep 09 16:14:17.122240 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:17.122074    2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 16:14:17.571705 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:17.571597    2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T16:14:17.122228249Z","UUID":"d5f0c518-b8f3-4329-a46a-09a8beedf0ee","Handler":null,"Name":"","Endpoint":""}
Sep 09 16:14:17.573393 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:17.573368    2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 16:14:17.573532 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:17.573406    2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 16:14:17.601517 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:17.601490    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:14:17.601643 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:17.601490    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:14:17.601643 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:17.601609    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjz7d" podUID="98c72774-4b40-43b2-b4dd-7590320acc60"
Sep 09 16:14:17.601643 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:17.601498    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:14:17.601809 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:17.601704    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gcb4p" podUID="7334d4d3-9b79-4b0d-ab28-e0aec88c18cb"
Sep 09 16:14:17.601880 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:17.601806    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slbqw" podUID="f1bd8aa2-35fd-458b-8a84-69ccf7a6276f"
Sep 09 16:14:17.834543 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:17.834429    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv" event={"ID":"3aa9556d-7d31-44de-8861-6ef97bc87b66","Type":"ContainerStarted","Data":"2b1710e8a563861102345de04eb1285712716b27da572a123768c2029d66f95e"}
Sep 09 16:14:17.836763 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:17.836733    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-202.ec2.internal" event={"ID":"1489a50f476055d1d230538a489fe1f4","Type":"ContainerStarted","Data":"4f5f81add319924704dac1f84dc1e51661567ca661538512dc2f66ad02ab01a9"}
Sep 09 16:14:17.855880 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:17.855826    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-202.ec2.internal" podStartSLOduration=21.855806439 podStartE2EDuration="21.855806439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:17.855094371 +0000 UTC m=+23.685302431" watchObservedRunningTime="2026-09-09 16:14:17.855806439 +0000 UTC m=+23.686014499"
Sep 09 16:14:18.578304 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:18.578263    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9shcg"
Sep 09 16:14:18.841853 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:18.841768    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv" event={"ID":"3aa9556d-7d31-44de-8861-6ef97bc87b66","Type":"ContainerStarted","Data":"ac8007220a36799b0069fca3909def3e09129a89d21e4e3f6352f3abeea67081"}
Sep 09 16:14:18.848533 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:18.848498    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phs9w" event={"ID":"e4fce212-3111-43ec-8fee-a58bd7ffdff9","Type":"ContainerStarted","Data":"db4928d984340768f5e97c445fa274035d244fd988df6dba594225ee5e5a7b78"}
Sep 09 16:14:18.862715 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:18.862673    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsnkv" podStartSLOduration=2.377485723 podStartE2EDuration="23.862660724s" totalImagesPullingTime="21.485175001s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:13:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:18.862387324 +0000 UTC m=+24.692595384" watchObservedRunningTime="2026-09-09 16:14:18.862660724 +0000 UTC m=+24.692868812"
Sep 09 16:14:18.885513 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:18.885479    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9shcg"
Sep 09 16:14:18.886125 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:18.886108    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9shcg"
Sep 09 16:14:19.601394 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:19.601363    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:14:19.601394 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:19.601382    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:14:19.601723 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:19.601388    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:14:19.601723 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:19.601482    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjz7d" podUID="98c72774-4b40-43b2-b4dd-7590320acc60"
Sep 09 16:14:19.601723 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:19.601541    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slbqw" podUID="f1bd8aa2-35fd-458b-8a84-69ccf7a6276f"
Sep 09 16:14:19.601723 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:19.601624    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gcb4p" podUID="7334d4d3-9b79-4b0d-ab28-e0aec88c18cb"
Sep 09 16:14:19.852617 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:19.852378    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9shcg"
Sep 09 16:14:20.860144 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:20.859940    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phs9w" event={"ID":"e4fce212-3111-43ec-8fee-a58bd7ffdff9","Type":"ContainerStarted","Data":"97aef340a7be125bcd35e36e860c36fb25a9283804ca8545986282db2b25e90d"}
Sep 09 16:14:20.888006 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:20.887958    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-phs9w" podStartSLOduration=6.452630779 podStartE2EDuration="25.887941798s" totalImagesPullingTime="19.435311019s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:20.887272335 +0000 UTC m=+26.717480396" watchObservedRunningTime="2026-09-09 16:14:20.887941798 +0000 UTC m=+26.718149857"
Sep 09 16:14:21.607277 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:21.601414    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:14:21.607277 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:21.601565    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gcb4p" podUID="7334d4d3-9b79-4b0d-ab28-e0aec88c18cb"
Sep 09 16:14:21.607277 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:21.601986    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:14:21.607277 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:21.602081    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjz7d" podUID="98c72774-4b40-43b2-b4dd-7590320acc60"
Sep 09 16:14:21.607277 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:21.602122    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:14:21.607277 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:21.602186    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slbqw" podUID="f1bd8aa2-35fd-458b-8a84-69ccf7a6276f"
Sep 09 16:14:21.862945 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:21.862669    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:14:21.862945 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:21.862785    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:14:21.862945 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:21.862824    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:14:21.879160 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:21.879008    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:14:21.880903 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:21.880881    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:14:22.371281 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:22.371254    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hjz7d"]
Sep 09 16:14:22.371422 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:22.371403    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:14:22.371540 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:22.371520    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjz7d" podUID="98c72774-4b40-43b2-b4dd-7590320acc60"
Sep 09 16:14:22.371880 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:22.371719    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-slbqw"]
Sep 09 16:14:22.372006 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:22.371989    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:14:22.372110 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:22.372093    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slbqw" podUID="f1bd8aa2-35fd-458b-8a84-69ccf7a6276f"
Sep 09 16:14:22.372352 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:22.372326    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gcb4p"]
Sep 09 16:14:22.372445 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:22.372420    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:14:22.372531 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:22.372514    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gcb4p" podUID="7334d4d3-9b79-4b0d-ab28-e0aec88c18cb"
Sep 09 16:14:22.866006 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:22.865975    2555 generic.go:356] "Generic (PLEG): container finished" podID="bd350005-0808-410e-bef5-9ecfd227869d" containerID="8141467078e0fac0d5ae0d56071ca5e812385a6173105d8bb651d9ddef825714" exitCode=0
Sep 09 16:14:22.866941 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:22.866058    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hgj9" event={"ID":"bd350005-0808-410e-bef5-9ecfd227869d","Type":"ContainerDied","Data":"8141467078e0fac0d5ae0d56071ca5e812385a6173105d8bb651d9ddef825714"}
Sep 09 16:14:23.601581 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:23.601551    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:14:23.601782 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:23.601654    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slbqw" podUID="f1bd8aa2-35fd-458b-8a84-69ccf7a6276f"
Sep 09 16:14:24.604098 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:24.604075    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:14:24.604467 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:24.604159    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjz7d" podUID="98c72774-4b40-43b2-b4dd-7590320acc60"
Sep 09 16:14:24.604467 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:24.604210    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:14:24.604467 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:24.604317    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gcb4p" podUID="7334d4d3-9b79-4b0d-ab28-e0aec88c18cb"
Sep 09 16:14:24.871485 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:24.871269    2555 generic.go:356] "Generic (PLEG): container finished" podID="bd350005-0808-410e-bef5-9ecfd227869d" containerID="34911db9121755d90d66a14dbb3a31a3ea36ddf341fd932a278317b53c12e222" exitCode=0
Sep 09 16:14:24.871632 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:24.871323    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hgj9" event={"ID":"bd350005-0808-410e-bef5-9ecfd227869d","Type":"ContainerDied","Data":"34911db9121755d90d66a14dbb3a31a3ea36ddf341fd932a278317b53c12e222"}
Sep 09 16:14:25.601952 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:25.601922    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:14:25.602094 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:25.602024    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slbqw" podUID="f1bd8aa2-35fd-458b-8a84-69ccf7a6276f"
Sep 09 16:14:26.601590 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:26.601557    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:14:26.601931 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:26.601700    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gcb4p" podUID="7334d4d3-9b79-4b0d-ab28-e0aec88c18cb"
Sep 09 16:14:26.601931 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:26.601753    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:14:26.601931 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:26.601844    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjz7d" podUID="98c72774-4b40-43b2-b4dd-7590320acc60"
Sep 09 16:14:26.878393 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:26.878286    2555 generic.go:356] "Generic (PLEG): container finished" podID="bd350005-0808-410e-bef5-9ecfd227869d" containerID="f17b846e3fbb056e5bc8ec22a218ce060eb0e04b37ea7f597c0865a338d9a7f5" exitCode=0
Sep 09 16:14:26.878393 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:26.878363    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hgj9" event={"ID":"bd350005-0808-410e-bef5-9ecfd227869d","Type":"ContainerDied","Data":"f17b846e3fbb056e5bc8ec22a218ce060eb0e04b37ea7f597c0865a338d9a7f5"}
Sep 09 16:14:27.364554 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:27.364503    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98c72774-4b40-43b2-b4dd-7590320acc60-original-pull-secret\") pod \"global-pull-secret-syncer-hjz7d\" (UID: \"98c72774-4b40-43b2-b4dd-7590320acc60\") " pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:14:27.364737 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:27.364587    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs\") pod \"network-metrics-daemon-gcb4p\" (UID: \"7334d4d3-9b79-4b0d-ab28-e0aec88c18cb\") " pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:14:27.364737 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:27.364711    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:14:27.364737 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:27.364720    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 16:14:27.364885 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:27.364785    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs podName:7334d4d3-9b79-4b0d-ab28-e0aec88c18cb nodeName:}" failed. No retries permitted until 2026-09-09 16:14:59.364764182 +0000 UTC m=+65.194972233 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs") pod "network-metrics-daemon-gcb4p" (UID: "7334d4d3-9b79-4b0d-ab28-e0aec88c18cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:14:27.364885 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:27.364803    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98c72774-4b40-43b2-b4dd-7590320acc60-original-pull-secret podName:98c72774-4b40-43b2-b4dd-7590320acc60 nodeName:}" failed. No retries permitted until 2026-09-09 16:14:59.364793796 +0000 UTC m=+65.195001838 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98c72774-4b40-43b2-b4dd-7590320acc60-original-pull-secret") pod "global-pull-secret-syncer-hjz7d" (UID: "98c72774-4b40-43b2-b4dd-7590320acc60") : object "kube-system"/"original-pull-secret" not registered
Sep 09 16:14:27.465622 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:27.465586    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv7fn\" (UniqueName: \"kubernetes.io/projected/f1bd8aa2-35fd-458b-8a84-69ccf7a6276f-kube-api-access-cv7fn\") pod \"network-check-target-slbqw\" (UID: \"f1bd8aa2-35fd-458b-8a84-69ccf7a6276f\") " pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:14:27.465785 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:27.465733    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 16:14:27.465785 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:27.465753    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 16:14:27.465785 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:27.465762    2555 projected.go:196] Error preparing data for projected volume kube-api-access-cv7fn for pod openshift-network-diagnostics/network-check-target-slbqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:14:27.465907 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:27.465815    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1bd8aa2-35fd-458b-8a84-69ccf7a6276f-kube-api-access-cv7fn podName:f1bd8aa2-35fd-458b-8a84-69ccf7a6276f nodeName:}" failed. No retries permitted until 2026-09-09 16:14:59.46579768 +0000 UTC m=+65.296005720 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cv7fn" (UniqueName: "kubernetes.io/projected/f1bd8aa2-35fd-458b-8a84-69ccf7a6276f-kube-api-access-cv7fn") pod "network-check-target-slbqw" (UID: "f1bd8aa2-35fd-458b-8a84-69ccf7a6276f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:14:27.602125 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:27.602089    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:14:27.602679 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:27.602205    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-slbqw" podUID="f1bd8aa2-35fd-458b-8a84-69ccf7a6276f"
Sep 09 16:14:28.604735 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:28.604707    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:14:28.605082 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:28.604802    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hjz7d" podUID="98c72774-4b40-43b2-b4dd-7590320acc60"
Sep 09 16:14:28.605082 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:28.604850    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:14:28.605082 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:28.604941    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gcb4p" podUID="7334d4d3-9b79-4b0d-ab28-e0aec88c18cb"
Sep 09 16:14:28.995381 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:28.995349    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-202.ec2.internal" event="NodeReady"
Sep 09 16:14:28.995588 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:28.995491    2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 16:14:29.039155 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.039119    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xs54n"]
Sep 09 16:14:29.052807 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.052777    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-snd6v"]
Sep 09 16:14:29.052993 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.052906    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xs54n"
Sep 09 16:14:29.055420 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.055396    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 16:14:29.055587 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.055560    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 16:14:29.055689 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.055675    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zjfwh\""
Sep 09 16:14:29.071272 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.071250    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xs54n"]
Sep 09 16:14:29.071272 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.071284    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-snd6v"]
Sep 09 16:14:29.071429 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.071374    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-snd6v"
Sep 09 16:14:29.073831 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.073811    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 16:14:29.073943 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.073885    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 16:14:29.074025 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.073963    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 16:14:29.074075 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.074020    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vg786\""
Sep 09 16:14:29.180312 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.180218    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v97l\" (UniqueName: \"kubernetes.io/projected/b7c6ef20-237d-46e2-a83a-c20ac88ccfdd-kube-api-access-9v97l\") pod \"ingress-canary-snd6v\" (UID: \"b7c6ef20-237d-46e2-a83a-c20ac88ccfdd\") " pod="openshift-ingress-canary/ingress-canary-snd6v"
Sep 09 16:14:29.180312 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.180272    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7c6ef20-237d-46e2-a83a-c20ac88ccfdd-cert\") pod \"ingress-canary-snd6v\" (UID: \"b7c6ef20-237d-46e2-a83a-c20ac88ccfdd\") " pod="openshift-ingress-canary/ingress-canary-snd6v"
Sep 09 16:14:29.180559 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.180331    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b21b58de-81cf-4ff7-9352-40854e5e0246-config-volume\") pod \"dns-default-xs54n\" (UID: \"b21b58de-81cf-4ff7-9352-40854e5e0246\") " pod="openshift-dns/dns-default-xs54n"
Sep 09 16:14:29.180559 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.180391    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls\") pod \"dns-default-xs54n\" (UID: \"b21b58de-81cf-4ff7-9352-40854e5e0246\") " pod="openshift-dns/dns-default-xs54n"
Sep 09 16:14:29.180559 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.180467    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt6jv\" (UniqueName: \"kubernetes.io/projected/b21b58de-81cf-4ff7-9352-40854e5e0246-kube-api-access-jt6jv\") pod \"dns-default-xs54n\" (UID: \"b21b58de-81cf-4ff7-9352-40854e5e0246\") " pod="openshift-dns/dns-default-xs54n"
Sep 09 16:14:29.180693 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.180558    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b21b58de-81cf-4ff7-9352-40854e5e0246-tmp-dir\") pod \"dns-default-xs54n\" (UID: \"b21b58de-81cf-4ff7-9352-40854e5e0246\") " pod="openshift-dns/dns-default-xs54n"
Sep 09 16:14:29.281585 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.281546    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt6jv\" (UniqueName: \"kubernetes.io/projected/b21b58de-81cf-4ff7-9352-40854e5e0246-kube-api-access-jt6jv\") pod \"dns-default-xs54n\" (UID: \"b21b58de-81cf-4ff7-9352-40854e5e0246\") " pod="openshift-dns/dns-default-xs54n"
Sep 09 16:14:29.282090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.282068    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b21b58de-81cf-4ff7-9352-40854e5e0246-tmp-dir\") pod \"dns-default-xs54n\" (UID: \"b21b58de-81cf-4ff7-9352-40854e5e0246\") " pod="openshift-dns/dns-default-xs54n"
Sep 09 16:14:29.282238 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.282222    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v97l\" (UniqueName: \"kubernetes.io/projected/b7c6ef20-237d-46e2-a83a-c20ac88ccfdd-kube-api-access-9v97l\") pod \"ingress-canary-snd6v\" (UID: \"b7c6ef20-237d-46e2-a83a-c20ac88ccfdd\") " pod="openshift-ingress-canary/ingress-canary-snd6v"
Sep 09 16:14:29.282339 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.282269    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b21b58de-81cf-4ff7-9352-40854e5e0246-tmp-dir\") pod \"dns-default-xs54n\" (UID: \"b21b58de-81cf-4ff7-9352-40854e5e0246\") " pod="openshift-dns/dns-default-xs54n"
Sep 09 16:14:29.282339 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.282279    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7c6ef20-237d-46e2-a83a-c20ac88ccfdd-cert\") pod \"ingress-canary-snd6v\" (UID: \"b7c6ef20-237d-46e2-a83a-c20ac88ccfdd\") " pod="openshift-ingress-canary/ingress-canary-snd6v"
Sep 09 16:14:29.282445 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.282379    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b21b58de-81cf-4ff7-9352-40854e5e0246-config-volume\") pod \"dns-default-xs54n\" (UID: \"b21b58de-81cf-4ff7-9352-40854e5e0246\") " pod="openshift-dns/dns-default-xs54n"
Sep 09 16:14:29.282707 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.282642    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls\") pod \"dns-default-xs54n\" (UID: \"b21b58de-81cf-4ff7-9352-40854e5e0246\") " pod="openshift-dns/dns-default-xs54n"
Sep 09 16:14:29.283065 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.283045    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b21b58de-81cf-4ff7-9352-40854e5e0246-config-volume\") pod \"dns-default-xs54n\" (UID: \"b21b58de-81cf-4ff7-9352-40854e5e0246\") " pod="openshift-dns/dns-default-xs54n"
Sep 09 16:14:29.283307 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:29.283276    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:14:29.283377 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:29.283369    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls podName:b21b58de-81cf-4ff7-9352-40854e5e0246 nodeName:}" failed. No retries permitted until 2026-09-09 16:14:29.783346757 +0000 UTC m=+35.613554817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls") pod "dns-default-xs54n" (UID: "b21b58de-81cf-4ff7-9352-40854e5e0246") : secret "dns-default-metrics-tls" not found
Sep 09 16:14:29.286972 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.286946    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7c6ef20-237d-46e2-a83a-c20ac88ccfdd-cert\") pod \"ingress-canary-snd6v\" (UID: \"b7c6ef20-237d-46e2-a83a-c20ac88ccfdd\") " pod="openshift-ingress-canary/ingress-canary-snd6v"
Sep 09 16:14:29.290517 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.290490    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt6jv\" (UniqueName: \"kubernetes.io/projected/b21b58de-81cf-4ff7-9352-40854e5e0246-kube-api-access-jt6jv\") pod \"dns-default-xs54n\" (UID: \"b21b58de-81cf-4ff7-9352-40854e5e0246\") " pod="openshift-dns/dns-default-xs54n"
Sep 09 16:14:29.290844 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.290823    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v97l\" (UniqueName: \"kubernetes.io/projected/b7c6ef20-237d-46e2-a83a-c20ac88ccfdd-kube-api-access-9v97l\") pod \"ingress-canary-snd6v\" (UID: \"b7c6ef20-237d-46e2-a83a-c20ac88ccfdd\") " pod="openshift-ingress-canary/ingress-canary-snd6v"
Sep 09 16:14:29.379905 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.379862    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-snd6v"
Sep 09 16:14:29.602240 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.602037    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:14:29.605092 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.605063    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 16:14:29.605455 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.605187    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 16:14:29.605455 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.605312    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7wfvc\""
Sep 09 16:14:29.713688 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.713661    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-snd6v"]
Sep 09 16:14:29.717580 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:14:29.717545    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c6ef20_237d_46e2_a83a_c20ac88ccfdd.slice/crio-8089c9e535f3278e018034372322019262d36e39f9e706fe0feaaa6390651831 WatchSource:0}: Error finding container 8089c9e535f3278e018034372322019262d36e39f9e706fe0feaaa6390651831: Status 404 returned error can't find the container with id 8089c9e535f3278e018034372322019262d36e39f9e706fe0feaaa6390651831
Sep 09 16:14:29.787581 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.787550    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls\") pod \"dns-default-xs54n\" (UID: \"b21b58de-81cf-4ff7-9352-40854e5e0246\") " pod="openshift-dns/dns-default-xs54n"
Sep 09 16:14:29.787759 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:29.787728    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:14:29.787838 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:29.787824    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls podName:b21b58de-81cf-4ff7-9352-40854e5e0246 nodeName:}" failed. No retries permitted until 2026-09-09 16:14:30.787802047 +0000 UTC m=+36.618010099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls") pod "dns-default-xs54n" (UID: "b21b58de-81cf-4ff7-9352-40854e5e0246") : secret "dns-default-metrics-tls" not found
Sep 09 16:14:29.886807 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:29.886768    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-snd6v" event={"ID":"b7c6ef20-237d-46e2-a83a-c20ac88ccfdd","Type":"ContainerStarted","Data":"8089c9e535f3278e018034372322019262d36e39f9e706fe0feaaa6390651831"}
Sep 09 16:14:30.605805 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:30.605777    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:14:30.606390 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:30.605776    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:14:30.608802 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:30.608783    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 16:14:30.609881 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:30.609863    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 16:14:30.609970 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:30.609869    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cjzmn\""
Sep 09 16:14:30.796154 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:30.796117    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls\") pod \"dns-default-xs54n\" (UID: \"b21b58de-81cf-4ff7-9352-40854e5e0246\") " pod="openshift-dns/dns-default-xs54n"
Sep 09 16:14:30.796338 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:30.796256    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:14:30.796338 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:30.796329    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls podName:b21b58de-81cf-4ff7-9352-40854e5e0246 nodeName:}" failed. No retries permitted until 2026-09-09 16:14:32.796310188 +0000 UTC m=+38.626518242 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls") pod "dns-default-xs54n" (UID: "b21b58de-81cf-4ff7-9352-40854e5e0246") : secret "dns-default-metrics-tls" not found
Sep 09 16:14:31.894027 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:31.893994    2555 generic.go:356] "Generic (PLEG): container finished" podID="bd350005-0808-410e-bef5-9ecfd227869d" containerID="0a229450c38fd3447ab2582f77b6165e8d6ced966d387c012b1b81df3fd316df" exitCode=0
Sep 09 16:14:31.894817 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:31.894071    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hgj9" event={"ID":"bd350005-0808-410e-bef5-9ecfd227869d","Type":"ContainerDied","Data":"0a229450c38fd3447ab2582f77b6165e8d6ced966d387c012b1b81df3fd316df"}
Sep 09 16:14:32.813546 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:32.813495    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls\") pod \"dns-default-xs54n\" (UID: \"b21b58de-81cf-4ff7-9352-40854e5e0246\") " pod="openshift-dns/dns-default-xs54n"
Sep 09 16:14:32.813711 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:32.813667    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:14:32.813783 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:32.813745    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls podName:b21b58de-81cf-4ff7-9352-40854e5e0246 nodeName:}" failed. No retries permitted until 2026-09-09 16:14:36.813724348 +0000 UTC m=+42.643932407 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls") pod "dns-default-xs54n" (UID: "b21b58de-81cf-4ff7-9352-40854e5e0246") : secret "dns-default-metrics-tls" not found
Sep 09 16:14:32.898827 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:32.898789    2555 generic.go:356] "Generic (PLEG): container finished" podID="bd350005-0808-410e-bef5-9ecfd227869d" containerID="afac903cc11fc5edc0600574be58f60ff9e724a3e651a432df5a03f28cbfaad2" exitCode=0
Sep 09 16:14:32.899207 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:32.898834    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hgj9" event={"ID":"bd350005-0808-410e-bef5-9ecfd227869d","Type":"ContainerDied","Data":"afac903cc11fc5edc0600574be58f60ff9e724a3e651a432df5a03f28cbfaad2"}
Sep 09 16:14:33.904889 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:33.904859    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hgj9" event={"ID":"bd350005-0808-410e-bef5-9ecfd227869d","Type":"ContainerStarted","Data":"8b481c897e029b6fa3fc2089342936454b60b857a1b5de28c7b9af3624b719aa"}
Sep 09 16:14:33.926157 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:33.926078    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6hgj9" podStartSLOduration=7.671695537 podStartE2EDuration="38.926063902s" totalImagesPullingTime="31.254368365s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:13:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:33.924519373 +0000 UTC m=+39.754727443" watchObservedRunningTime="2026-09-09 16:14:33.926063902 +0000 UTC m=+39.756271962"
Sep 09 16:14:34.908972 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:34.908937    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-snd6v" event={"ID":"b7c6ef20-237d-46e2-a83a-c20ac88ccfdd","Type":"ContainerStarted","Data":"a4f435dd275b92504a9ca488b3e3497e8bd61b2db88a6d8c230877f3e66853ae"}
Sep 09 16:14:34.924469 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:34.924425    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-snd6v" podStartSLOduration=1.7667259199999998 podStartE2EDuration="5.924409587s" totalImagesPullingTime="4.157683667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:34.924010996 +0000 UTC m=+40.754219054" watchObservedRunningTime="2026-09-09 16:14:34.924409587 +0000 UTC m=+40.754617645"
Sep 09 16:14:36.837884 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:36.837844    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls\") pod \"dns-default-xs54n\" (UID: \"b21b58de-81cf-4ff7-9352-40854e5e0246\") " pod="openshift-dns/dns-default-xs54n"
Sep 09 16:14:36.841243 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:36.841217    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:14:36.841383 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:36.841283    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls podName:b21b58de-81cf-4ff7-9352-40854e5e0246 nodeName:}" failed. No retries permitted until 2026-09-09 16:14:44.84126733 +0000 UTC m=+50.671475366 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls") pod "dns-default-xs54n" (UID: "b21b58de-81cf-4ff7-9352-40854e5e0246") : secret "dns-default-metrics-tls" not found
Sep 09 16:14:44.889584 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:44.889518    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls\") pod \"dns-default-xs54n\" (UID: \"b21b58de-81cf-4ff7-9352-40854e5e0246\") " pod="openshift-dns/dns-default-xs54n"
Sep 09 16:14:44.890037 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:44.889645    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:14:44.890037 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:44.889701    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls podName:b21b58de-81cf-4ff7-9352-40854e5e0246 nodeName:}" failed. No retries permitted until 2026-09-09 16:15:00.889688631 +0000 UTC m=+66.719896667 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls") pod "dns-default-xs54n" (UID: "b21b58de-81cf-4ff7-9352-40854e5e0246") : secret "dns-default-metrics-tls" not found
Sep 09 16:14:53.886085 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:53.886055    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-phs9w"
Sep 09 16:14:59.387426 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:59.387385    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs\") pod \"network-metrics-daemon-gcb4p\" (UID: \"7334d4d3-9b79-4b0d-ab28-e0aec88c18cb\") " pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:14:59.387827 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:59.387536    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98c72774-4b40-43b2-b4dd-7590320acc60-original-pull-secret\") pod \"global-pull-secret-syncer-hjz7d\" (UID: \"98c72774-4b40-43b2-b4dd-7590320acc60\") " pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:14:59.390187 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:59.390145    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 16:14:59.390353 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:59.390334    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 16:14:59.398350 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:59.398336    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 16:14:59.398403 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:14:59.398390    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs podName:7334d4d3-9b79-4b0d-ab28-e0aec88c18cb nodeName:}" failed. No retries permitted until 2026-09-09 16:16:03.398373404 +0000 UTC m=+129.228581440 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs") pod "network-metrics-daemon-gcb4p" (UID: "7334d4d3-9b79-4b0d-ab28-e0aec88c18cb") : secret "metrics-daemon-secret" not found
Sep 09 16:14:59.401124 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:59.401106    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98c72774-4b40-43b2-b4dd-7590320acc60-original-pull-secret\") pod \"global-pull-secret-syncer-hjz7d\" (UID: \"98c72774-4b40-43b2-b4dd-7590320acc60\") " pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:14:59.423322 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:59.423289    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hjz7d"
Sep 09 16:14:59.488046 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:59.488014    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv7fn\" (UniqueName: \"kubernetes.io/projected/f1bd8aa2-35fd-458b-8a84-69ccf7a6276f-kube-api-access-cv7fn\") pod \"network-check-target-slbqw\" (UID: \"f1bd8aa2-35fd-458b-8a84-69ccf7a6276f\") " pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:14:59.491170 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:59.491153    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 16:14:59.500699 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:59.500679    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 16:14:59.510997 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:59.510969    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv7fn\" (UniqueName: \"kubernetes.io/projected/f1bd8aa2-35fd-458b-8a84-69ccf7a6276f-kube-api-access-cv7fn\") pod \"network-check-target-slbqw\" (UID: \"f1bd8aa2-35fd-458b-8a84-69ccf7a6276f\") " pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:14:59.615687 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:59.615662    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7wfvc\""
Sep 09 16:14:59.623925 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:59.623907    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:14:59.691212 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:59.691186    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hjz7d"]
Sep 09 16:14:59.695212 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:14:59.695185    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c72774_4b40_43b2_b4dd_7590320acc60.slice/crio-7e7e4d975f3b6eb46d81a17c2c62e13ee375f3b30e368a72dc86bde16a37fdea WatchSource:0}: Error finding container 7e7e4d975f3b6eb46d81a17c2c62e13ee375f3b30e368a72dc86bde16a37fdea: Status 404 returned error can't find the container with id 7e7e4d975f3b6eb46d81a17c2c62e13ee375f3b30e368a72dc86bde16a37fdea
Sep 09 16:14:59.888931 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:59.888900    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-slbqw"]
Sep 09 16:14:59.892242 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:14:59.892214    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1bd8aa2_35fd_458b_8a84_69ccf7a6276f.slice/crio-f60d7aaf3050916ff8964f8bb75e95e1ac589757e6042649e2c0d4360b1ebc80 WatchSource:0}: Error finding container f60d7aaf3050916ff8964f8bb75e95e1ac589757e6042649e2c0d4360b1ebc80: Status 404 returned error can't find the container with id f60d7aaf3050916ff8964f8bb75e95e1ac589757e6042649e2c0d4360b1ebc80
Sep 09 16:14:59.969306 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:59.969205    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-slbqw" event={"ID":"f1bd8aa2-35fd-458b-8a84-69ccf7a6276f","Type":"ContainerStarted","Data":"f60d7aaf3050916ff8964f8bb75e95e1ac589757e6042649e2c0d4360b1ebc80"}
Sep 09 16:14:59.970317 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:14:59.970269    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hjz7d" event={"ID":"98c72774-4b40-43b2-b4dd-7590320acc60","Type":"ContainerStarted","Data":"7e7e4d975f3b6eb46d81a17c2c62e13ee375f3b30e368a72dc86bde16a37fdea"}
Sep 09 16:15:00.896758 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:00.896716    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls\") pod \"dns-default-xs54n\" (UID: \"b21b58de-81cf-4ff7-9352-40854e5e0246\") " pod="openshift-dns/dns-default-xs54n"
Sep 09 16:15:00.897233 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:15:00.896913    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:15:00.897233 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:15:00.896994    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls podName:b21b58de-81cf-4ff7-9352-40854e5e0246 nodeName:}" failed. No retries permitted until 2026-09-09 16:15:32.896971095 +0000 UTC m=+98.727179135 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls") pod "dns-default-xs54n" (UID: "b21b58de-81cf-4ff7-9352-40854e5e0246") : secret "dns-default-metrics-tls" not found
Sep 09 16:15:03.422268 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.422229    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ffc886cc6-xhcmh"]
Sep 09 16:15:03.425186 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.425163    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ffc886cc6-xhcmh"
Sep 09 16:15:03.428106 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.428084    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 16:15:03.428106 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.428099    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-mljtq\""
Sep 09 16:15:03.428302 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.428105    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 16:15:03.429517 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.429496    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 09 16:15:03.429600 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.429528    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 16:15:03.434095 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.434075    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ffc886cc6-xhcmh"]
Sep 09 16:15:03.450920 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.450893    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-66bd86d448-frb9z"]
Sep 09 16:15:03.453735 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.453711    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66bd86d448-frb9z"
Sep 09 16:15:03.456602 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.456581    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 09 16:15:03.463250 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.463228    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-66bd86d448-frb9z"]
Sep 09 16:15:03.513395 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.513365    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/82ecdb79-906e-4fe7-b6e8-bf4d4dc53333-klusterlet-config\") pod \"klusterlet-addon-workmgr-66bd86d448-frb9z\" (UID: \"82ecdb79-906e-4fe7-b6e8-bf4d4dc53333\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66bd86d448-frb9z"
Sep 09 16:15:03.513563 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.513426    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4814345a-758e-4aa1-ba3f-0b71cd822584-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6ffc886cc6-xhcmh\" (UID: \"4814345a-758e-4aa1-ba3f-0b71cd822584\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ffc886cc6-xhcmh"
Sep 09 16:15:03.513563 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.513504    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82ecdb79-906e-4fe7-b6e8-bf4d4dc53333-tmp\") pod \"klusterlet-addon-workmgr-66bd86d448-frb9z\" (UID: \"82ecdb79-906e-4fe7-b6e8-bf4d4dc53333\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66bd86d448-frb9z"
Sep 09 16:15:03.513563 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.513540    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljhhj\" (UniqueName: \"kubernetes.io/projected/4814345a-758e-4aa1-ba3f-0b71cd822584-kube-api-access-ljhhj\") pod \"managed-serviceaccount-addon-agent-6ffc886cc6-xhcmh\" (UID: \"4814345a-758e-4aa1-ba3f-0b71cd822584\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ffc886cc6-xhcmh"
Sep 09 16:15:03.513690 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.513580    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25m9l\" (UniqueName: \"kubernetes.io/projected/82ecdb79-906e-4fe7-b6e8-bf4d4dc53333-kube-api-access-25m9l\") pod \"klusterlet-addon-workmgr-66bd86d448-frb9z\" (UID: \"82ecdb79-906e-4fe7-b6e8-bf4d4dc53333\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66bd86d448-frb9z"
Sep 09 16:15:03.614418 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.614374    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4814345a-758e-4aa1-ba3f-0b71cd822584-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6ffc886cc6-xhcmh\" (UID: \"4814345a-758e-4aa1-ba3f-0b71cd822584\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ffc886cc6-xhcmh"
Sep 09 16:15:03.614600 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.614429    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82ecdb79-906e-4fe7-b6e8-bf4d4dc53333-tmp\") pod \"klusterlet-addon-workmgr-66bd86d448-frb9z\" (UID: \"82ecdb79-906e-4fe7-b6e8-bf4d4dc53333\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66bd86d448-frb9z"
Sep 09 16:15:03.614600 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.614462    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljhhj\" (UniqueName: \"kubernetes.io/projected/4814345a-758e-4aa1-ba3f-0b71cd822584-kube-api-access-ljhhj\") pod \"managed-serviceaccount-addon-agent-6ffc886cc6-xhcmh\" (UID: \"4814345a-758e-4aa1-ba3f-0b71cd822584\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ffc886cc6-xhcmh"
Sep 09 16:15:03.614719 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.614653    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25m9l\" (UniqueName: \"kubernetes.io/projected/82ecdb79-906e-4fe7-b6e8-bf4d4dc53333-kube-api-access-25m9l\") pod \"klusterlet-addon-workmgr-66bd86d448-frb9z\" (UID: \"82ecdb79-906e-4fe7-b6e8-bf4d4dc53333\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66bd86d448-frb9z"
Sep 09 16:15:03.614772 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.614747    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/82ecdb79-906e-4fe7-b6e8-bf4d4dc53333-klusterlet-config\") pod \"klusterlet-addon-workmgr-66bd86d448-frb9z\" (UID: \"82ecdb79-906e-4fe7-b6e8-bf4d4dc53333\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66bd86d448-frb9z"
Sep 09 16:15:03.614942 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.614894    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82ecdb79-906e-4fe7-b6e8-bf4d4dc53333-tmp\") pod \"klusterlet-addon-workmgr-66bd86d448-frb9z\" (UID: \"82ecdb79-906e-4fe7-b6e8-bf4d4dc53333\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66bd86d448-frb9z"
Sep 09 16:15:03.617439 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.617416    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4814345a-758e-4aa1-ba3f-0b71cd822584-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6ffc886cc6-xhcmh\" (UID: \"4814345a-758e-4aa1-ba3f-0b71cd822584\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ffc886cc6-xhcmh"
Sep 09 16:15:03.617629 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.617603    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/82ecdb79-906e-4fe7-b6e8-bf4d4dc53333-klusterlet-config\") pod \"klusterlet-addon-workmgr-66bd86d448-frb9z\" (UID: \"82ecdb79-906e-4fe7-b6e8-bf4d4dc53333\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66bd86d448-frb9z"
Sep 09 16:15:03.622656 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.622614    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25m9l\" (UniqueName: \"kubernetes.io/projected/82ecdb79-906e-4fe7-b6e8-bf4d4dc53333-kube-api-access-25m9l\") pod \"klusterlet-addon-workmgr-66bd86d448-frb9z\" (UID: \"82ecdb79-906e-4fe7-b6e8-bf4d4dc53333\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66bd86d448-frb9z"
Sep 09 16:15:03.622749 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.622683    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljhhj\" (UniqueName: \"kubernetes.io/projected/4814345a-758e-4aa1-ba3f-0b71cd822584-kube-api-access-ljhhj\") pod \"managed-serviceaccount-addon-agent-6ffc886cc6-xhcmh\" (UID: \"4814345a-758e-4aa1-ba3f-0b71cd822584\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ffc886cc6-xhcmh"
Sep 09 16:15:03.744662 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.744629    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ffc886cc6-xhcmh"
Sep 09 16:15:03.764039 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.764004    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66bd86d448-frb9z"
Sep 09 16:15:03.982615 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.982579    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-slbqw" event={"ID":"f1bd8aa2-35fd-458b-8a84-69ccf7a6276f","Type":"ContainerStarted","Data":"3a4d2a4f67c0d37df06eb728740e379bba7678b3df977f3c46070396398750fc"}
Sep 09 16:15:03.982781 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.982660    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:15:03.998870 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:03.998763    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-slbqw" podStartSLOduration=65.911949279 podStartE2EDuration="1m8.998743558s" totalImagesPullingTime="3.086794279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:15:03.997679337 +0000 UTC m=+69.827887396" watchObservedRunningTime="2026-09-09 16:15:03.998743558 +0000 UTC m=+69.828951616"
Sep 09 16:15:04.643776 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:04.643694    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-66bd86d448-frb9z"]
Sep 09 16:15:04.646134 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:04.646108    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ffc886cc6-xhcmh"]
Sep 09 16:15:04.647042 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:15:04.647018    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ecdb79_906e_4fe7_b6e8_bf4d4dc53333.slice/crio-398979264bc91deccbd7194cb2021ad192ff51c129a3f0c958d95cab8ab7107a WatchSource:0}: Error finding container 398979264bc91deccbd7194cb2021ad192ff51c129a3f0c958d95cab8ab7107a: Status 404 returned error can't find the container with id 398979264bc91deccbd7194cb2021ad192ff51c129a3f0c958d95cab8ab7107a
Sep 09 16:15:04.649338 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:15:04.649288    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4814345a_758e_4aa1_ba3f_0b71cd822584.slice/crio-368c0e188e9d92d6b60dc92ea5d0f07740498c4d1f87b8573ec920239f725369 WatchSource:0}: Error finding container 368c0e188e9d92d6b60dc92ea5d0f07740498c4d1f87b8573ec920239f725369: Status 404 returned error can't find the container with id 368c0e188e9d92d6b60dc92ea5d0f07740498c4d1f87b8573ec920239f725369
Sep 09 16:15:04.986724 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:04.986631    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66bd86d448-frb9z" event={"ID":"82ecdb79-906e-4fe7-b6e8-bf4d4dc53333","Type":"ContainerStarted","Data":"398979264bc91deccbd7194cb2021ad192ff51c129a3f0c958d95cab8ab7107a"}
Sep 09 16:15:04.988323 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:04.988280    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hjz7d" event={"ID":"98c72774-4b40-43b2-b4dd-7590320acc60","Type":"ContainerStarted","Data":"f534fadfa0e55bc2d7d5bb60e0957b44fcd22258f5c321a081ef2b0b6f4bebc9"}
Sep 09 16:15:04.989363 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:04.989341    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ffc886cc6-xhcmh" event={"ID":"4814345a-758e-4aa1-ba3f-0b71cd822584","Type":"ContainerStarted","Data":"368c0e188e9d92d6b60dc92ea5d0f07740498c4d1f87b8573ec920239f725369"}
Sep 09 16:15:05.003748 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:05.003689    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hjz7d" podStartSLOduration=65.26605908 podStartE2EDuration="1m10.003670812s" totalImagesPullingTime="4.737611732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:15:05.003157093 +0000 UTC m=+70.833365180" watchObservedRunningTime="2026-09-09 16:15:05.003670812 +0000 UTC m=+70.833878873"
Sep 09 16:15:10.006830 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:10.006786    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ffc886cc6-xhcmh" event={"ID":"4814345a-758e-4aa1-ba3f-0b71cd822584","Type":"ContainerStarted","Data":"dc774e3367a1de5a4e3b77d2ab48e832275ea4bbe15d7d62d7d2379cbaa0c960"}
Sep 09 16:15:10.008283 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:10.008259    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66bd86d448-frb9z" event={"ID":"82ecdb79-906e-4fe7-b6e8-bf4d4dc53333","Type":"ContainerStarted","Data":"11b3d4c336f87cd4a10d88bf3f6405aced1a3c8f977352367bb2b0abbf0f2d2e"}
Sep 09 16:15:10.008466 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:10.008446    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66bd86d448-frb9z"
Sep 09 16:15:10.009764 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:10.009744    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66bd86d448-frb9z"
Sep 09 16:15:10.022864 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:10.022821    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ffc886cc6-xhcmh" podStartSLOduration=1.814951905 podStartE2EDuration="7.022808993s" totalImagesPullingTime="5.207857088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:15:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:15:10.02252542 +0000 UTC m=+75.852733478" watchObservedRunningTime="2026-09-09 16:15:10.022808993 +0000 UTC m=+75.853017047"
Sep 09 16:15:10.039602 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:10.039558    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66bd86d448-frb9z" podStartSLOduration=1.8122241460000001 podStartE2EDuration="7.039541596s" totalImagesPullingTime="5.22731745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:15:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:15:10.038868816 +0000 UTC m=+75.869076875" watchObservedRunningTime="2026-09-09 16:15:10.039541596 +0000 UTC m=+75.869749657"
Sep 09 16:15:32.917630 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:32.917580    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls\") pod \"dns-default-xs54n\" (UID: \"b21b58de-81cf-4ff7-9352-40854e5e0246\") " pod="openshift-dns/dns-default-xs54n"
Sep 09 16:15:32.918010 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:15:32.917713    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:15:32.918010 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:15:32.917780    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls podName:b21b58de-81cf-4ff7-9352-40854e5e0246 nodeName:}" failed. No retries permitted until 2026-09-09 16:16:36.917765618 +0000 UTC m=+162.747973659 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls") pod "dns-default-xs54n" (UID: "b21b58de-81cf-4ff7-9352-40854e5e0246") : secret "dns-default-metrics-tls" not found
Sep 09 16:15:34.991689 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:15:34.991654    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-slbqw"
Sep 09 16:16:03.422052 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:16:03.422002    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs\") pod \"network-metrics-daemon-gcb4p\" (UID: \"7334d4d3-9b79-4b0d-ab28-e0aec88c18cb\") " pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:16:03.422646 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:16:03.422189    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 16:16:03.422646 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:16:03.422308    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs podName:7334d4d3-9b79-4b0d-ab28-e0aec88c18cb nodeName:}" failed. No retries permitted until 2026-09-09 16:18:05.422269984 +0000 UTC m=+251.252478037 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs") pod "network-metrics-daemon-gcb4p" (UID: "7334d4d3-9b79-4b0d-ab28-e0aec88c18cb") : secret "metrics-daemon-secret" not found
Sep 09 16:16:32.062883 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:16:32.062831    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xs54n" podUID="b21b58de-81cf-4ff7-9352-40854e5e0246"
Sep 09 16:16:32.205656 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:16:32.205583    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xs54n"
Sep 09 16:16:33.617662 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:16:33.617624    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-gcb4p" podUID="7334d4d3-9b79-4b0d-ab28-e0aec88c18cb"
Sep 09 16:16:36.949757 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:16:36.949715    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls\") pod \"dns-default-xs54n\" (UID: \"b21b58de-81cf-4ff7-9352-40854e5e0246\") " pod="openshift-dns/dns-default-xs54n"
Sep 09 16:16:36.950201 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:16:36.949871    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:16:36.950201 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:16:36.949946    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls podName:b21b58de-81cf-4ff7-9352-40854e5e0246 nodeName:}" failed. No retries permitted until 2026-09-09 16:18:38.949927137 +0000 UTC m=+284.780135179 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls") pod "dns-default-xs54n" (UID: "b21b58de-81cf-4ff7-9352-40854e5e0246") : secret "dns-default-metrics-tls" not found
Sep 09 16:16:46.601861 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:16:46.601818    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:17:10.008957 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:17:10.008918    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66bd86d448-frb9z" podUID="82ecdb79-906e-4fe7-b6e8-bf4d4dc53333" containerName="acm-agent" probeResult="failure" output="Get \"http://10.134.0.9:8000/readyz\": dial tcp 10.134.0.9:8000: connect: connection refused"
Sep 09 16:17:10.295548 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:17:10.295518    2555 generic.go:356] "Generic (PLEG): container finished" podID="4814345a-758e-4aa1-ba3f-0b71cd822584" containerID="dc774e3367a1de5a4e3b77d2ab48e832275ea4bbe15d7d62d7d2379cbaa0c960" exitCode=255
Sep 09 16:17:10.295749 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:17:10.295600    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ffc886cc6-xhcmh" event={"ID":"4814345a-758e-4aa1-ba3f-0b71cd822584","Type":"ContainerDied","Data":"dc774e3367a1de5a4e3b77d2ab48e832275ea4bbe15d7d62d7d2379cbaa0c960"}
Sep 09 16:17:10.296911 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:17:10.296889    2555 generic.go:356] "Generic (PLEG): container finished" podID="82ecdb79-906e-4fe7-b6e8-bf4d4dc53333" containerID="11b3d4c336f87cd4a10d88bf3f6405aced1a3c8f977352367bb2b0abbf0f2d2e" exitCode=1
Sep 09 16:17:10.297011 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:17:10.296943    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66bd86d448-frb9z" event={"ID":"82ecdb79-906e-4fe7-b6e8-bf4d4dc53333","Type":"ContainerDied","Data":"11b3d4c336f87cd4a10d88bf3f6405aced1a3c8f977352367bb2b0abbf0f2d2e"}
Sep 09 16:17:10.297201 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:17:10.297189    2555 scope.go:117] "RemoveContainer" containerID="11b3d4c336f87cd4a10d88bf3f6405aced1a3c8f977352367bb2b0abbf0f2d2e"
Sep 09 16:17:10.301604 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:17:10.301582    2555 scope.go:117] "RemoveContainer" containerID="dc774e3367a1de5a4e3b77d2ab48e832275ea4bbe15d7d62d7d2379cbaa0c960"
Sep 09 16:17:11.302308 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:17:11.302261    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6ffc886cc6-xhcmh" event={"ID":"4814345a-758e-4aa1-ba3f-0b71cd822584","Type":"ContainerStarted","Data":"fc4a74ff5edde9aaca35203eeff5c0d3579102ecdef72210796ec0374983975f"}
Sep 09 16:17:11.303889 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:17:11.303868    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66bd86d448-frb9z" event={"ID":"82ecdb79-906e-4fe7-b6e8-bf4d4dc53333","Type":"ContainerStarted","Data":"f4346c4c6956444c79a310674e0d630a9742ae42ef6c72e46eb9020cbaa6223d"}
Sep 09 16:17:11.304135 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:17:11.304119    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66bd86d448-frb9z"
Sep 09 16:17:11.304725 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:17:11.304710    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66bd86d448-frb9z"
Sep 09 16:18:05.448115 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:05.448069    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs\") pod \"network-metrics-daemon-gcb4p\" (UID: \"7334d4d3-9b79-4b0d-ab28-e0aec88c18cb\") " pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:18:05.448747 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:05.448265    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 16:18:05.448747 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:05.448403    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs podName:7334d4d3-9b79-4b0d-ab28-e0aec88c18cb nodeName:}" failed. No retries permitted until 2026-09-09 16:20:07.448377125 +0000 UTC m=+373.278585175 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs") pod "network-metrics-daemon-gcb4p" (UID: "7334d4d3-9b79-4b0d-ab28-e0aec88c18cb") : secret "metrics-daemon-secret" not found
Sep 09 16:18:06.792718 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.792682    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-zgcjl"]
Sep 09 16:18:06.794514 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.794493    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-zgcjl"
Sep 09 16:18:06.797187 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.797162    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 09 16:18:06.797334 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.797160    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 09 16:18:06.797334 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.797318    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 09 16:18:06.798501 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.798484    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-sxl4q\""
Sep 09 16:18:06.798605 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.798565    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 09 16:18:06.803455 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.803434    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 09 16:18:06.807544 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.807524    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-zgcjl"]
Sep 09 16:18:06.858278 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.858233    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/561d9dd7-04c3-42dc-bcf6-2c92fd61706b-trusted-ca\") pod \"console-operator-85b454874b-zgcjl\" (UID: \"561d9dd7-04c3-42dc-bcf6-2c92fd61706b\") " pod="openshift-console-operator/console-operator-85b454874b-zgcjl"
Sep 09 16:18:06.858495 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.858318    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/561d9dd7-04c3-42dc-bcf6-2c92fd61706b-config\") pod \"console-operator-85b454874b-zgcjl\" (UID: \"561d9dd7-04c3-42dc-bcf6-2c92fd61706b\") " pod="openshift-console-operator/console-operator-85b454874b-zgcjl"
Sep 09 16:18:06.858495 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.858375    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhz7s\" (UniqueName: \"kubernetes.io/projected/561d9dd7-04c3-42dc-bcf6-2c92fd61706b-kube-api-access-fhz7s\") pod \"console-operator-85b454874b-zgcjl\" (UID: \"561d9dd7-04c3-42dc-bcf6-2c92fd61706b\") " pod="openshift-console-operator/console-operator-85b454874b-zgcjl"
Sep 09 16:18:06.858495 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.858412    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/561d9dd7-04c3-42dc-bcf6-2c92fd61706b-serving-cert\") pod \"console-operator-85b454874b-zgcjl\" (UID: \"561d9dd7-04c3-42dc-bcf6-2c92fd61706b\") " pod="openshift-console-operator/console-operator-85b454874b-zgcjl"
Sep 09 16:18:06.880633 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.880602    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-srgtn"]
Sep 09 16:18:06.882993 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.882965    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn"
Sep 09 16:18:06.886770 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.886747    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 09 16:18:06.886886 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.886864    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 16:18:06.890372 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.890352    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 16:18:06.890486 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.890351    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 09 16:18:06.890486 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.890358    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-b8q58\""
Sep 09 16:18:06.893271 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.893250    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 09 16:18:06.903532 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.903507    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-srgtn"]
Sep 09 16:18:06.959174 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.959119    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/561d9dd7-04c3-42dc-bcf6-2c92fd61706b-trusted-ca\") pod \"console-operator-85b454874b-zgcjl\" (UID: \"561d9dd7-04c3-42dc-bcf6-2c92fd61706b\") " pod="openshift-console-operator/console-operator-85b454874b-zgcjl"
Sep 09 16:18:06.959401 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.959198    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/561d9dd7-04c3-42dc-bcf6-2c92fd61706b-config\") pod \"console-operator-85b454874b-zgcjl\" (UID: \"561d9dd7-04c3-42dc-bcf6-2c92fd61706b\") " pod="openshift-console-operator/console-operator-85b454874b-zgcjl"
Sep 09 16:18:06.959401 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.959236    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhz7s\" (UniqueName: \"kubernetes.io/projected/561d9dd7-04c3-42dc-bcf6-2c92fd61706b-kube-api-access-fhz7s\") pod \"console-operator-85b454874b-zgcjl\" (UID: \"561d9dd7-04c3-42dc-bcf6-2c92fd61706b\") " pod="openshift-console-operator/console-operator-85b454874b-zgcjl"
Sep 09 16:18:06.959401 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.959263    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca9bcf27-b906-4735-bab2-b17bffa33bce-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-srgtn\" (UID: \"ca9bcf27-b906-4735-bab2-b17bffa33bce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn"
Sep 09 16:18:06.959401 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.959318    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca9bcf27-b906-4735-bab2-b17bffa33bce-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-srgtn\" (UID: \"ca9bcf27-b906-4735-bab2-b17bffa33bce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn"
Sep 09 16:18:06.959615 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.959405    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca9bcf27-b906-4735-bab2-b17bffa33bce-tmp\") pod \"insights-operator-76bcd6ccf5-srgtn\" (UID: \"ca9bcf27-b906-4735-bab2-b17bffa33bce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn"
Sep 09 16:18:06.959615 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.959455    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/561d9dd7-04c3-42dc-bcf6-2c92fd61706b-serving-cert\") pod \"console-operator-85b454874b-zgcjl\" (UID: \"561d9dd7-04c3-42dc-bcf6-2c92fd61706b\") " pod="openshift-console-operator/console-operator-85b454874b-zgcjl"
Sep 09 16:18:06.959615 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.959484    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ca9bcf27-b906-4735-bab2-b17bffa33bce-snapshots\") pod \"insights-operator-76bcd6ccf5-srgtn\" (UID: \"ca9bcf27-b906-4735-bab2-b17bffa33bce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn"
Sep 09 16:18:06.959615 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.959515    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ca9bcf27-b906-4735-bab2-b17bffa33bce-serving-cert\") pod \"insights-operator-76bcd6ccf5-srgtn\" (UID: \"ca9bcf27-b906-4735-bab2-b17bffa33bce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn"
Sep 09 16:18:06.959615 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.959542    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhmkp\" (UniqueName: \"kubernetes.io/projected/ca9bcf27-b906-4735-bab2-b17bffa33bce-kube-api-access-dhmkp\") pod \"insights-operator-76bcd6ccf5-srgtn\" (UID: \"ca9bcf27-b906-4735-bab2-b17bffa33bce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn"
Sep 09 16:18:06.959968 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.959948    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/561d9dd7-04c3-42dc-bcf6-2c92fd61706b-config\") pod \"console-operator-85b454874b-zgcjl\" (UID: \"561d9dd7-04c3-42dc-bcf6-2c92fd61706b\") " pod="openshift-console-operator/console-operator-85b454874b-zgcjl"
Sep 09 16:18:06.960227 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.960207    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/561d9dd7-04c3-42dc-bcf6-2c92fd61706b-trusted-ca\") pod \"console-operator-85b454874b-zgcjl\" (UID: \"561d9dd7-04c3-42dc-bcf6-2c92fd61706b\") " pod="openshift-console-operator/console-operator-85b454874b-zgcjl"
Sep 09 16:18:06.961780 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.961757    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/561d9dd7-04c3-42dc-bcf6-2c92fd61706b-serving-cert\") pod \"console-operator-85b454874b-zgcjl\" (UID: \"561d9dd7-04c3-42dc-bcf6-2c92fd61706b\") " pod="openshift-console-operator/console-operator-85b454874b-zgcjl"
Sep 09 16:18:06.974989 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:06.974955    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhz7s\" (UniqueName: \"kubernetes.io/projected/561d9dd7-04c3-42dc-bcf6-2c92fd61706b-kube-api-access-fhz7s\") pod \"console-operator-85b454874b-zgcjl\" (UID: \"561d9dd7-04c3-42dc-bcf6-2c92fd61706b\") " pod="openshift-console-operator/console-operator-85b454874b-zgcjl"
Sep 09 16:18:07.060531 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:07.060435    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca9bcf27-b906-4735-bab2-b17bffa33bce-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-srgtn\" (UID: \"ca9bcf27-b906-4735-bab2-b17bffa33bce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn"
Sep 09 16:18:07.060531 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:07.060475    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca9bcf27-b906-4735-bab2-b17bffa33bce-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-srgtn\" (UID: \"ca9bcf27-b906-4735-bab2-b17bffa33bce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn"
Sep 09 16:18:07.060531 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:07.060502    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca9bcf27-b906-4735-bab2-b17bffa33bce-tmp\") pod \"insights-operator-76bcd6ccf5-srgtn\" (UID: \"ca9bcf27-b906-4735-bab2-b17bffa33bce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn"
Sep 09 16:18:07.060816 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:07.060546    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ca9bcf27-b906-4735-bab2-b17bffa33bce-snapshots\") pod \"insights-operator-76bcd6ccf5-srgtn\" (UID: \"ca9bcf27-b906-4735-bab2-b17bffa33bce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn"
Sep 09 16:18:07.060816 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:07.060568    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ca9bcf27-b906-4735-bab2-b17bffa33bce-serving-cert\") pod \"insights-operator-76bcd6ccf5-srgtn\" (UID: \"ca9bcf27-b906-4735-bab2-b17bffa33bce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn"
Sep 09 16:18:07.060816 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:07.060603    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhmkp\" (UniqueName: \"kubernetes.io/projected/ca9bcf27-b906-4735-bab2-b17bffa33bce-kube-api-access-dhmkp\") pod \"insights-operator-76bcd6ccf5-srgtn\" (UID: \"ca9bcf27-b906-4735-bab2-b17bffa33bce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn"
Sep 09 16:18:07.060973 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:07.060864    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca9bcf27-b906-4735-bab2-b17bffa33bce-tmp\") pod \"insights-operator-76bcd6ccf5-srgtn\" (UID: \"ca9bcf27-b906-4735-bab2-b17bffa33bce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn"
Sep 09 16:18:07.061131 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:07.061108    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca9bcf27-b906-4735-bab2-b17bffa33bce-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-srgtn\" (UID: \"ca9bcf27-b906-4735-bab2-b17bffa33bce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn"
Sep 09 16:18:07.061208 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:07.061174    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ca9bcf27-b906-4735-bab2-b17bffa33bce-snapshots\") pod \"insights-operator-76bcd6ccf5-srgtn\" (UID: \"ca9bcf27-b906-4735-bab2-b17bffa33bce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn"
Sep 09 16:18:07.061323 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:07.061283    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca9bcf27-b906-4735-bab2-b17bffa33bce-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-srgtn\" (UID: \"ca9bcf27-b906-4735-bab2-b17bffa33bce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn"
Sep 09 16:18:07.062937 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:07.062921    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ca9bcf27-b906-4735-bab2-b17bffa33bce-serving-cert\") pod \"insights-operator-76bcd6ccf5-srgtn\" (UID: \"ca9bcf27-b906-4735-bab2-b17bffa33bce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn"
Sep 09 16:18:07.068456 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:07.068425    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhmkp\" (UniqueName: \"kubernetes.io/projected/ca9bcf27-b906-4735-bab2-b17bffa33bce-kube-api-access-dhmkp\") pod \"insights-operator-76bcd6ccf5-srgtn\" (UID: \"ca9bcf27-b906-4735-bab2-b17bffa33bce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn"
Sep 09 16:18:07.104160 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:07.104124    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-zgcjl"
Sep 09 16:18:07.194856 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:07.194814    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn"
Sep 09 16:18:07.372575 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:07.372490    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-zgcjl"]
Sep 09 16:18:07.375932 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:18:07.375902    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod561d9dd7_04c3_42dc_bcf6_2c92fd61706b.slice/crio-e0005725594235d270d50c1d46b22b66a0a89a766c3361ff4ef51dd460b57c29 WatchSource:0}: Error finding container e0005725594235d270d50c1d46b22b66a0a89a766c3361ff4ef51dd460b57c29: Status 404 returned error can't find the container with id e0005725594235d270d50c1d46b22b66a0a89a766c3361ff4ef51dd460b57c29
Sep 09 16:18:07.461610 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:07.461572    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-srgtn"]
Sep 09 16:18:07.465203 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:18:07.465169    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9bcf27_b906_4735_bab2_b17bffa33bce.slice/crio-47732aebd0b8ef754dbb1657fe0d52629eab33919c17249be5a3cb85376996e6 WatchSource:0}: Error finding container 47732aebd0b8ef754dbb1657fe0d52629eab33919c17249be5a3cb85376996e6: Status 404 returned error can't find the container with id 47732aebd0b8ef754dbb1657fe0d52629eab33919c17249be5a3cb85376996e6
Sep 09 16:18:07.468644 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:07.468620    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-zgcjl" event={"ID":"561d9dd7-04c3-42dc-bcf6-2c92fd61706b","Type":"ContainerStarted","Data":"e0005725594235d270d50c1d46b22b66a0a89a766c3361ff4ef51dd460b57c29"}
Sep 09 16:18:08.474644 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:08.474598    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn" event={"ID":"ca9bcf27-b906-4735-bab2-b17bffa33bce","Type":"ContainerStarted","Data":"47732aebd0b8ef754dbb1657fe0d52629eab33919c17249be5a3cb85376996e6"}
Sep 09 16:18:10.483145 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:10.483035    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn" event={"ID":"ca9bcf27-b906-4735-bab2-b17bffa33bce","Type":"ContainerStarted","Data":"06bd533cc23313e19f9293a4c6423a671d6cbd86c725dfcd824d36d4046d563d"}
Sep 09 16:18:10.484735 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:10.484713    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgcjl_561d9dd7-04c3-42dc-bcf6-2c92fd61706b/console-operator/0.log"
Sep 09 16:18:10.484834 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:10.484754    2555 generic.go:356] "Generic (PLEG): container finished" podID="561d9dd7-04c3-42dc-bcf6-2c92fd61706b" containerID="2134f8fe4b4e26f04333c41a0c29b4e0684a46458e0cb481dfe70a514f271e59" exitCode=255
Sep 09 16:18:10.484834 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:10.484784    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-zgcjl" event={"ID":"561d9dd7-04c3-42dc-bcf6-2c92fd61706b","Type":"ContainerDied","Data":"2134f8fe4b4e26f04333c41a0c29b4e0684a46458e0cb481dfe70a514f271e59"}
Sep 09 16:18:10.485020 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:10.485006    2555 scope.go:117] "RemoveContainer" containerID="2134f8fe4b4e26f04333c41a0c29b4e0684a46458e0cb481dfe70a514f271e59"
Sep 09 16:18:10.498161 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:10.498111    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn" podStartSLOduration=1.933258285 podStartE2EDuration="4.498095917s" totalImagesPullingTime="2.564837632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:18:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:18:10.49732487 +0000 UTC m=+256.327532944" watchObservedRunningTime="2026-09-09 16:18:10.498095917 +0000 UTC m=+256.328303977"
Sep 09 16:18:11.396726 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.396692    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5wzqv"]
Sep 09 16:18:11.399712 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.399686    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:11.402105 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.402075    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 16:18:11.402234 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.402144    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q2x6b\""
Sep 09 16:18:11.403457 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.403438    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 16:18:11.411934 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.411906    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5wzqv"]
Sep 09 16:18:11.489369 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.489340    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgcjl_561d9dd7-04c3-42dc-bcf6-2c92fd61706b/console-operator/1.log"
Sep 09 16:18:11.489769 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.489756    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgcjl_561d9dd7-04c3-42dc-bcf6-2c92fd61706b/console-operator/0.log"
Sep 09 16:18:11.489811 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.489794    2555 generic.go:356] "Generic (PLEG): container finished" podID="561d9dd7-04c3-42dc-bcf6-2c92fd61706b" containerID="59abd7079828436bb5f99769c8711d04df9001199c7a745747708721c9484f07" exitCode=255
Sep 09 16:18:11.489904 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.489881    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-zgcjl" event={"ID":"561d9dd7-04c3-42dc-bcf6-2c92fd61706b","Type":"ContainerDied","Data":"59abd7079828436bb5f99769c8711d04df9001199c7a745747708721c9484f07"}
Sep 09 16:18:11.489955 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.489944    2555 scope.go:117] "RemoveContainer" containerID="2134f8fe4b4e26f04333c41a0c29b4e0684a46458e0cb481dfe70a514f271e59"
Sep 09 16:18:11.490170 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.490154    2555 scope.go:117] "RemoveContainer" containerID="59abd7079828436bb5f99769c8711d04df9001199c7a745747708721c9484f07"
Sep 09 16:18:11.490381 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:11.490362    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-zgcjl_openshift-console-operator(561d9dd7-04c3-42dc-bcf6-2c92fd61706b)\"" pod="openshift-console-operator/console-operator-85b454874b-zgcjl" podUID="561d9dd7-04c3-42dc-bcf6-2c92fd61706b"
Sep 09 16:18:11.496709 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.496684    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c931a390-ae5a-451b-9b76-258aa5ff5bba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:11.496820 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.496722    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnhnf\" (UniqueName: \"kubernetes.io/projected/c931a390-ae5a-451b-9b76-258aa5ff5bba-kube-api-access-rnhnf\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:11.496820 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.496743    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c931a390-ae5a-451b-9b76-258aa5ff5bba-crio-socket\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:11.496820 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.496760    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c931a390-ae5a-451b-9b76-258aa5ff5bba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:11.496968 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.496880    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c931a390-ae5a-451b-9b76-258aa5ff5bba-data-volume\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:11.598039 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.598002    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c931a390-ae5a-451b-9b76-258aa5ff5bba-data-volume\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:11.598238 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.598098    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c931a390-ae5a-451b-9b76-258aa5ff5bba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:11.598238 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.598197    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnhnf\" (UniqueName: \"kubernetes.io/projected/c931a390-ae5a-451b-9b76-258aa5ff5bba-kube-api-access-rnhnf\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:11.598403 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.598266    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c931a390-ae5a-451b-9b76-258aa5ff5bba-crio-socket\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:11.598403 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.598318    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c931a390-ae5a-451b-9b76-258aa5ff5bba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:11.598513 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:11.598426    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 16:18:11.598513 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.598431    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c931a390-ae5a-451b-9b76-258aa5ff5bba-data-volume\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:11.598513 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.598425    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c931a390-ae5a-451b-9b76-258aa5ff5bba-crio-socket\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:11.598630 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:11.598566    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c931a390-ae5a-451b-9b76-258aa5ff5bba-insights-runtime-extractor-tls podName:c931a390-ae5a-451b-9b76-258aa5ff5bba nodeName:}" failed. No retries permitted until 2026-09-09 16:18:12.098541998 +0000 UTC m=+257.928750051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c931a390-ae5a-451b-9b76-258aa5ff5bba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5wzqv" (UID: "c931a390-ae5a-451b-9b76-258aa5ff5bba") : secret "insights-runtime-extractor-tls" not found
Sep 09 16:18:11.598630 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:11.598595    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 16:18:11.598706 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:11.598671    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c931a390-ae5a-451b-9b76-258aa5ff5bba-kube-rbac-proxy-cm podName:c931a390-ae5a-451b-9b76-258aa5ff5bba nodeName:}" failed. No retries permitted until 2026-09-09 16:18:12.098655295 +0000 UTC m=+257.928863349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c931a390-ae5a-451b-9b76-258aa5ff5bba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5wzqv" (UID: "c931a390-ae5a-451b-9b76-258aa5ff5bba") : configmap "kube-rbac-proxy" not found
Sep 09 16:18:11.606896 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:11.606864    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnhnf\" (UniqueName: \"kubernetes.io/projected/c931a390-ae5a-451b-9b76-258aa5ff5bba-kube-api-access-rnhnf\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:12.102005 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:12.101946    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c931a390-ae5a-451b-9b76-258aa5ff5bba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:12.102220 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:12.102027    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c931a390-ae5a-451b-9b76-258aa5ff5bba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:12.102220 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:12.102076    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 16:18:12.102220 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:12.102153    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 16:18:12.102220 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:12.102183    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c931a390-ae5a-451b-9b76-258aa5ff5bba-kube-rbac-proxy-cm podName:c931a390-ae5a-451b-9b76-258aa5ff5bba nodeName:}" failed. No retries permitted until 2026-09-09 16:18:13.102167535 +0000 UTC m=+258.932375585 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c931a390-ae5a-451b-9b76-258aa5ff5bba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5wzqv" (UID: "c931a390-ae5a-451b-9b76-258aa5ff5bba") : configmap "kube-rbac-proxy" not found
Sep 09 16:18:12.102220 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:12.102212    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c931a390-ae5a-451b-9b76-258aa5ff5bba-insights-runtime-extractor-tls podName:c931a390-ae5a-451b-9b76-258aa5ff5bba nodeName:}" failed. No retries permitted until 2026-09-09 16:18:13.102197412 +0000 UTC m=+258.932405466 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c931a390-ae5a-451b-9b76-258aa5ff5bba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5wzqv" (UID: "c931a390-ae5a-451b-9b76-258aa5ff5bba") : secret "insights-runtime-extractor-tls" not found
Sep 09 16:18:12.494682 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:12.494653    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgcjl_561d9dd7-04c3-42dc-bcf6-2c92fd61706b/console-operator/1.log"
Sep 09 16:18:12.495058 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:12.495039    2555 scope.go:117] "RemoveContainer" containerID="59abd7079828436bb5f99769c8711d04df9001199c7a745747708721c9484f07"
Sep 09 16:18:12.495249 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:12.495228    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-zgcjl_openshift-console-operator(561d9dd7-04c3-42dc-bcf6-2c92fd61706b)\"" pod="openshift-console-operator/console-operator-85b454874b-zgcjl" podUID="561d9dd7-04c3-42dc-bcf6-2c92fd61706b"
Sep 09 16:18:12.926919 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:12.926836    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-drfcs"]
Sep 09 16:18:12.931089 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:12.931069    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-drfcs"
Sep 09 16:18:12.933644 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:12.933612    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 09 16:18:12.933778 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:12.933619    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-k2gqw\""
Sep 09 16:18:12.934910 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:12.934889    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 09 16:18:12.934910 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:12.934889    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 09 16:18:12.935086 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:12.934988    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 09 16:18:12.936771 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:12.936748    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-drfcs"]
Sep 09 16:18:13.009514 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:13.009458    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d05e58f4-ad7e-4876-8939-413905b94562-signing-key\") pod \"service-ca-75d99c484-drfcs\" (UID: \"d05e58f4-ad7e-4876-8939-413905b94562\") " pod="openshift-service-ca/service-ca-75d99c484-drfcs"
Sep 09 16:18:13.009514 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:13.009517    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcwgd\" (UniqueName: \"kubernetes.io/projected/d05e58f4-ad7e-4876-8939-413905b94562-kube-api-access-hcwgd\") pod \"service-ca-75d99c484-drfcs\" (UID: \"d05e58f4-ad7e-4876-8939-413905b94562\") " pod="openshift-service-ca/service-ca-75d99c484-drfcs"
Sep 09 16:18:13.009732 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:13.009557    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d05e58f4-ad7e-4876-8939-413905b94562-signing-cabundle\") pod \"service-ca-75d99c484-drfcs\" (UID: \"d05e58f4-ad7e-4876-8939-413905b94562\") " pod="openshift-service-ca/service-ca-75d99c484-drfcs"
Sep 09 16:18:13.110796 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:13.110752    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c931a390-ae5a-451b-9b76-258aa5ff5bba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:13.110796 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:13.110802    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c931a390-ae5a-451b-9b76-258aa5ff5bba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:13.110996 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:13.110886    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 16:18:13.110996 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:13.110951    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d05e58f4-ad7e-4876-8939-413905b94562-signing-key\") pod \"service-ca-75d99c484-drfcs\" (UID: \"d05e58f4-ad7e-4876-8939-413905b94562\") " pod="openshift-service-ca/service-ca-75d99c484-drfcs"
Sep 09 16:18:13.110996 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:13.110972    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c931a390-ae5a-451b-9b76-258aa5ff5bba-kube-rbac-proxy-cm podName:c931a390-ae5a-451b-9b76-258aa5ff5bba nodeName:}" failed. No retries permitted until 2026-09-09 16:18:15.11095382 +0000 UTC m=+260.941161870 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c931a390-ae5a-451b-9b76-258aa5ff5bba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5wzqv" (UID: "c931a390-ae5a-451b-9b76-258aa5ff5bba") : configmap "kube-rbac-proxy" not found
Sep 09 16:18:13.111131 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:13.111013    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcwgd\" (UniqueName: \"kubernetes.io/projected/d05e58f4-ad7e-4876-8939-413905b94562-kube-api-access-hcwgd\") pod \"service-ca-75d99c484-drfcs\" (UID: \"d05e58f4-ad7e-4876-8939-413905b94562\") " pod="openshift-service-ca/service-ca-75d99c484-drfcs"
Sep 09 16:18:13.111131 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:13.111044    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 16:18:13.111131 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:13.111050    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d05e58f4-ad7e-4876-8939-413905b94562-signing-cabundle\") pod \"service-ca-75d99c484-drfcs\" (UID: \"d05e58f4-ad7e-4876-8939-413905b94562\") " pod="openshift-service-ca/service-ca-75d99c484-drfcs"
Sep 09 16:18:13.111131 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:13.111083    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c931a390-ae5a-451b-9b76-258aa5ff5bba-insights-runtime-extractor-tls podName:c931a390-ae5a-451b-9b76-258aa5ff5bba nodeName:}" failed. No retries permitted until 2026-09-09 16:18:15.111072536 +0000 UTC m=+260.941280573 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c931a390-ae5a-451b-9b76-258aa5ff5bba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5wzqv" (UID: "c931a390-ae5a-451b-9b76-258aa5ff5bba") : secret "insights-runtime-extractor-tls" not found
Sep 09 16:18:13.111855 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:13.111830    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d05e58f4-ad7e-4876-8939-413905b94562-signing-cabundle\") pod \"service-ca-75d99c484-drfcs\" (UID: \"d05e58f4-ad7e-4876-8939-413905b94562\") " pod="openshift-service-ca/service-ca-75d99c484-drfcs"
Sep 09 16:18:13.113605 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:13.113582    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d05e58f4-ad7e-4876-8939-413905b94562-signing-key\") pod \"service-ca-75d99c484-drfcs\" (UID: \"d05e58f4-ad7e-4876-8939-413905b94562\") " pod="openshift-service-ca/service-ca-75d99c484-drfcs"
Sep 09 16:18:13.124375 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:13.124347    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcwgd\" (UniqueName: \"kubernetes.io/projected/d05e58f4-ad7e-4876-8939-413905b94562-kube-api-access-hcwgd\") pod \"service-ca-75d99c484-drfcs\" (UID: \"d05e58f4-ad7e-4876-8939-413905b94562\") " pod="openshift-service-ca/service-ca-75d99c484-drfcs"
Sep 09 16:18:13.241391 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:13.241349    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-drfcs"
Sep 09 16:18:13.365028 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:13.365001    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-92lzv_fc6eae27-5d1d-4957-99b5-e87c1d177376/dns-node-resolver/0.log"
Sep 09 16:18:13.511744 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:13.511655    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-drfcs"]
Sep 09 16:18:13.514712 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:18:13.514675    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05e58f4_ad7e_4876_8939_413905b94562.slice/crio-dcdc5f1d30da7a8654d3f75fd0b19d2060c81f2121c0ad7a9cc84a3fbfa1c141 WatchSource:0}: Error finding container dcdc5f1d30da7a8654d3f75fd0b19d2060c81f2121c0ad7a9cc84a3fbfa1c141: Status 404 returned error can't find the container with id dcdc5f1d30da7a8654d3f75fd0b19d2060c81f2121c0ad7a9cc84a3fbfa1c141
Sep 09 16:18:14.365547 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:14.365514    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hkc7k_3f37385e-b449-4281-b738-7edce088387c/node-ca/0.log"
Sep 09 16:18:14.502203 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:14.502161    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-drfcs" event={"ID":"d05e58f4-ad7e-4876-8939-413905b94562","Type":"ContainerStarted","Data":"dcdc5f1d30da7a8654d3f75fd0b19d2060c81f2121c0ad7a9cc84a3fbfa1c141"}
Sep 09 16:18:15.128301 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:15.128265    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c931a390-ae5a-451b-9b76-258aa5ff5bba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:15.128706 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:15.128377    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c931a390-ae5a-451b-9b76-258aa5ff5bba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:15.128706 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:15.128442    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 16:18:15.128706 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:15.128451    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 16:18:15.128706 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:15.128530    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c931a390-ae5a-451b-9b76-258aa5ff5bba-kube-rbac-proxy-cm podName:c931a390-ae5a-451b-9b76-258aa5ff5bba nodeName:}" failed. No retries permitted until 2026-09-09 16:18:19.128509131 +0000 UTC m=+264.958717170 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c931a390-ae5a-451b-9b76-258aa5ff5bba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5wzqv" (UID: "c931a390-ae5a-451b-9b76-258aa5ff5bba") : configmap "kube-rbac-proxy" not found
Sep 09 16:18:15.128706 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:15.128556    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c931a390-ae5a-451b-9b76-258aa5ff5bba-insights-runtime-extractor-tls podName:c931a390-ae5a-451b-9b76-258aa5ff5bba nodeName:}" failed. No retries permitted until 2026-09-09 16:18:19.128540676 +0000 UTC m=+264.958748713 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c931a390-ae5a-451b-9b76-258aa5ff5bba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5wzqv" (UID: "c931a390-ae5a-451b-9b76-258aa5ff5bba") : secret "insights-runtime-extractor-tls" not found
Sep 09 16:18:15.365465 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:15.365426    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-snd6v_b7c6ef20-237d-46e2-a83a-c20ac88ccfdd/serve-healthcheck-canary/0.log"
Sep 09 16:18:16.510250 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:16.510205    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-drfcs" event={"ID":"d05e58f4-ad7e-4876-8939-413905b94562","Type":"ContainerStarted","Data":"766656a3dfcba64e4b97b32f18a54a2cde1f00d4cde727acd88f352aacf3a3f4"}
Sep 09 16:18:17.105104 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:17.104959    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-zgcjl"
Sep 09 16:18:17.105104 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:17.105024    2555 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-zgcjl"
Sep 09 16:18:17.105484 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:17.105467    2555 scope.go:117] "RemoveContainer" containerID="59abd7079828436bb5f99769c8711d04df9001199c7a745747708721c9484f07"
Sep 09 16:18:17.105662 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:17.105640    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-zgcjl_openshift-console-operator(561d9dd7-04c3-42dc-bcf6-2c92fd61706b)\"" pod="openshift-console-operator/console-operator-85b454874b-zgcjl" podUID="561d9dd7-04c3-42dc-bcf6-2c92fd61706b"
Sep 09 16:18:19.163437 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:19.163388    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c931a390-ae5a-451b-9b76-258aa5ff5bba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:19.163814 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:19.163456    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c931a390-ae5a-451b-9b76-258aa5ff5bba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:19.163814 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:19.163517    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 16:18:19.163814 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:19.163583    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 16:18:19.163814 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:19.163604    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c931a390-ae5a-451b-9b76-258aa5ff5bba-kube-rbac-proxy-cm podName:c931a390-ae5a-451b-9b76-258aa5ff5bba nodeName:}" failed. No retries permitted until 2026-09-09 16:18:27.163583675 +0000 UTC m=+272.993791720 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c931a390-ae5a-451b-9b76-258aa5ff5bba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5wzqv" (UID: "c931a390-ae5a-451b-9b76-258aa5ff5bba") : configmap "kube-rbac-proxy" not found
Sep 09 16:18:19.163814 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:19.163629    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c931a390-ae5a-451b-9b76-258aa5ff5bba-insights-runtime-extractor-tls podName:c931a390-ae5a-451b-9b76-258aa5ff5bba nodeName:}" failed. No retries permitted until 2026-09-09 16:18:27.163619504 +0000 UTC m=+272.993827559 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c931a390-ae5a-451b-9b76-258aa5ff5bba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5wzqv" (UID: "c931a390-ae5a-451b-9b76-258aa5ff5bba") : secret "insights-runtime-extractor-tls" not found
Sep 09 16:18:27.230457 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:27.230403    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c931a390-ae5a-451b-9b76-258aa5ff5bba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:27.230457 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:27.230471    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c931a390-ae5a-451b-9b76-258aa5ff5bba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:27.230911 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:27.230584    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 16:18:27.230911 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:27.230637    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c931a390-ae5a-451b-9b76-258aa5ff5bba-insights-runtime-extractor-tls podName:c931a390-ae5a-451b-9b76-258aa5ff5bba nodeName:}" failed. No retries permitted until 2026-09-09 16:18:43.230624661 +0000 UTC m=+289.060832698 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c931a390-ae5a-451b-9b76-258aa5ff5bba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5wzqv" (UID: "c931a390-ae5a-451b-9b76-258aa5ff5bba") : secret "insights-runtime-extractor-tls" not found
Sep 09 16:18:27.231084 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:27.231057    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c931a390-ae5a-451b-9b76-258aa5ff5bba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:28.601938 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:28.601911    2555 scope.go:117] "RemoveContainer" containerID="59abd7079828436bb5f99769c8711d04df9001199c7a745747708721c9484f07"
Sep 09 16:18:29.553244 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:29.553199    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgcjl_561d9dd7-04c3-42dc-bcf6-2c92fd61706b/console-operator/1.log"
Sep 09 16:18:29.553442 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:29.553312    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-zgcjl" event={"ID":"561d9dd7-04c3-42dc-bcf6-2c92fd61706b","Type":"ContainerStarted","Data":"bc39edc39e3e491456becfff5998b7535892bf4da99e708a7ce237cd68a50b30"}
Sep 09 16:18:29.553701 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:29.553679    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-zgcjl"
Sep 09 16:18:29.558888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:29.558830    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-zgcjl"
Sep 09 16:18:29.570634 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:29.570551    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-drfcs" podStartSLOduration=15.479250112999999 podStartE2EDuration="17.570525347s" totalImagesPullingTime="2.091275234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:18:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:18:16.526031679 +0000 UTC m=+262.356239739" watchObservedRunningTime="2026-09-09 16:18:29.570525347 +0000 UTC m=+275.400733406"
Sep 09 16:18:29.571523 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:29.571474    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-zgcjl" podStartSLOduration=20.915506512 podStartE2EDuration="23.571458956s" totalImagesPullingTime="2.655952444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:18:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:18:29.569680093 +0000 UTC m=+275.399888153" watchObservedRunningTime="2026-09-09 16:18:29.571458956 +0000 UTC m=+275.401667015"
Sep 09 16:18:33.780710 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.780677    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cc689c778-vg2sn"]
Sep 09 16:18:33.807492 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.807458    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cc689c778-vg2sn"]
Sep 09 16:18:33.807666 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.807626    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:33.810420 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.810395    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 16:18:33.810567 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.810446    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8q4lv\""
Sep 09 16:18:33.810567 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.810512    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 16:18:33.810670 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.810580    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 16:18:33.816272 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.816242    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 16:18:33.881088 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.881050    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fc4dce56-bf35-434f-9b43-c3ea87ef83e3-ca-trust-extracted\") pod \"image-registry-cc689c778-vg2sn\" (UID: \"fc4dce56-bf35-434f-9b43-c3ea87ef83e3\") " pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:33.881088 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.881103    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc4dce56-bf35-434f-9b43-c3ea87ef83e3-bound-sa-token\") pod \"image-registry-cc689c778-vg2sn\" (UID: \"fc4dce56-bf35-434f-9b43-c3ea87ef83e3\") " pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:33.881357 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.881207    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fc4dce56-bf35-434f-9b43-c3ea87ef83e3-image-registry-private-configuration\") pod \"image-registry-cc689c778-vg2sn\" (UID: \"fc4dce56-bf35-434f-9b43-c3ea87ef83e3\") " pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:33.881357 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.881251    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fc4dce56-bf35-434f-9b43-c3ea87ef83e3-registry-certificates\") pod \"image-registry-cc689c778-vg2sn\" (UID: \"fc4dce56-bf35-434f-9b43-c3ea87ef83e3\") " pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:33.881357 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.881285    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpk4j\" (UniqueName: \"kubernetes.io/projected/fc4dce56-bf35-434f-9b43-c3ea87ef83e3-kube-api-access-dpk4j\") pod \"image-registry-cc689c778-vg2sn\" (UID: \"fc4dce56-bf35-434f-9b43-c3ea87ef83e3\") " pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:33.881357 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.881339    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc4dce56-bf35-434f-9b43-c3ea87ef83e3-registry-tls\") pod \"image-registry-cc689c778-vg2sn\" (UID: \"fc4dce56-bf35-434f-9b43-c3ea87ef83e3\") " pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:33.881515 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.881410    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fc4dce56-bf35-434f-9b43-c3ea87ef83e3-trusted-ca\") pod \"image-registry-cc689c778-vg2sn\" (UID: \"fc4dce56-bf35-434f-9b43-c3ea87ef83e3\") " pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:33.881515 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.881451    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fc4dce56-bf35-434f-9b43-c3ea87ef83e3-installation-pull-secrets\") pod \"image-registry-cc689c778-vg2sn\" (UID: \"fc4dce56-bf35-434f-9b43-c3ea87ef83e3\") " pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:33.982535 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.982492    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc4dce56-bf35-434f-9b43-c3ea87ef83e3-bound-sa-token\") pod \"image-registry-cc689c778-vg2sn\" (UID: \"fc4dce56-bf35-434f-9b43-c3ea87ef83e3\") " pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:33.982737 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.982571    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fc4dce56-bf35-434f-9b43-c3ea87ef83e3-image-registry-private-configuration\") pod \"image-registry-cc689c778-vg2sn\" (UID: \"fc4dce56-bf35-434f-9b43-c3ea87ef83e3\") " pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:33.982737 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.982600    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fc4dce56-bf35-434f-9b43-c3ea87ef83e3-registry-certificates\") pod \"image-registry-cc689c778-vg2sn\" (UID: \"fc4dce56-bf35-434f-9b43-c3ea87ef83e3\") " pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:33.982737 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.982647    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpk4j\" (UniqueName: \"kubernetes.io/projected/fc4dce56-bf35-434f-9b43-c3ea87ef83e3-kube-api-access-dpk4j\") pod \"image-registry-cc689c778-vg2sn\" (UID: \"fc4dce56-bf35-434f-9b43-c3ea87ef83e3\") " pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:33.982737 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.982690    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc4dce56-bf35-434f-9b43-c3ea87ef83e3-registry-tls\") pod \"image-registry-cc689c778-vg2sn\" (UID: \"fc4dce56-bf35-434f-9b43-c3ea87ef83e3\") " pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:33.982737 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.982720    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fc4dce56-bf35-434f-9b43-c3ea87ef83e3-trusted-ca\") pod \"image-registry-cc689c778-vg2sn\" (UID: \"fc4dce56-bf35-434f-9b43-c3ea87ef83e3\") " pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:33.982984 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.982764    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fc4dce56-bf35-434f-9b43-c3ea87ef83e3-installation-pull-secrets\") pod \"image-registry-cc689c778-vg2sn\" (UID: \"fc4dce56-bf35-434f-9b43-c3ea87ef83e3\") " pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:33.982984 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.982793    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fc4dce56-bf35-434f-9b43-c3ea87ef83e3-ca-trust-extracted\") pod \"image-registry-cc689c778-vg2sn\" (UID: \"fc4dce56-bf35-434f-9b43-c3ea87ef83e3\") " pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:33.983178 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.983155    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fc4dce56-bf35-434f-9b43-c3ea87ef83e3-ca-trust-extracted\") pod \"image-registry-cc689c778-vg2sn\" (UID: \"fc4dce56-bf35-434f-9b43-c3ea87ef83e3\") " pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:33.983553 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.983526    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fc4dce56-bf35-434f-9b43-c3ea87ef83e3-registry-certificates\") pod \"image-registry-cc689c778-vg2sn\" (UID: \"fc4dce56-bf35-434f-9b43-c3ea87ef83e3\") " pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:33.984152 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.984128    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fc4dce56-bf35-434f-9b43-c3ea87ef83e3-trusted-ca\") pod \"image-registry-cc689c778-vg2sn\" (UID: \"fc4dce56-bf35-434f-9b43-c3ea87ef83e3\") " pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:33.985155 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.985128    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fc4dce56-bf35-434f-9b43-c3ea87ef83e3-image-registry-private-configuration\") pod \"image-registry-cc689c778-vg2sn\" (UID: \"fc4dce56-bf35-434f-9b43-c3ea87ef83e3\") " pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:33.985650 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.985625    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fc4dce56-bf35-434f-9b43-c3ea87ef83e3-installation-pull-secrets\") pod \"image-registry-cc689c778-vg2sn\" (UID: \"fc4dce56-bf35-434f-9b43-c3ea87ef83e3\") " pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:33.985750 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.985716    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fc4dce56-bf35-434f-9b43-c3ea87ef83e3-registry-tls\") pod \"image-registry-cc689c778-vg2sn\" (UID: \"fc4dce56-bf35-434f-9b43-c3ea87ef83e3\") " pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:33.990781 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.990742    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc4dce56-bf35-434f-9b43-c3ea87ef83e3-bound-sa-token\") pod \"image-registry-cc689c778-vg2sn\" (UID: \"fc4dce56-bf35-434f-9b43-c3ea87ef83e3\") " pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:33.990970 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:33.990935    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpk4j\" (UniqueName: \"kubernetes.io/projected/fc4dce56-bf35-434f-9b43-c3ea87ef83e3-kube-api-access-dpk4j\") pod \"image-registry-cc689c778-vg2sn\" (UID: \"fc4dce56-bf35-434f-9b43-c3ea87ef83e3\") " pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:34.116913 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:34.116812    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:34.406870 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:34.406843    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cc689c778-vg2sn"]
Sep 09 16:18:34.409396 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:18:34.409360    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4dce56_bf35_434f_9b43_c3ea87ef83e3.slice/crio-70f01ab5c1fc79f850b560efa5dc934edd75182c4c6028d2f2a4410a280cde86 WatchSource:0}: Error finding container 70f01ab5c1fc79f850b560efa5dc934edd75182c4c6028d2f2a4410a280cde86: Status 404 returned error can't find the container with id 70f01ab5c1fc79f850b560efa5dc934edd75182c4c6028d2f2a4410a280cde86
Sep 09 16:18:34.570813 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:34.570771    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cc689c778-vg2sn" event={"ID":"fc4dce56-bf35-434f-9b43-c3ea87ef83e3","Type":"ContainerStarted","Data":"6a6d2d525311a3a798b86abfd81e0f7966bcc0aa80d9d9d9c35f49cd9a7a1559"}
Sep 09 16:18:34.570978 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:34.570823    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cc689c778-vg2sn" event={"ID":"fc4dce56-bf35-434f-9b43-c3ea87ef83e3","Type":"ContainerStarted","Data":"70f01ab5c1fc79f850b560efa5dc934edd75182c4c6028d2f2a4410a280cde86"}
Sep 09 16:18:34.570978 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:34.570853    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:34.591324 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:34.590287    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-cc689c778-vg2sn" podStartSLOduration=1.590264796 podStartE2EDuration="1.590264796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:18:34.587331752 +0000 UTC m=+280.417539813" watchObservedRunningTime="2026-09-09 16:18:34.590264796 +0000 UTC m=+280.420472856"
Sep 09 16:18:35.206458 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:35.206406    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xs54n" podUID="b21b58de-81cf-4ff7-9352-40854e5e0246"
Sep 09 16:18:35.574136 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:35.574101    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xs54n"
Sep 09 16:18:39.027718 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:39.027671    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls\") pod \"dns-default-xs54n\" (UID: \"b21b58de-81cf-4ff7-9352-40854e5e0246\") " pod="openshift-dns/dns-default-xs54n"
Sep 09 16:18:39.030005 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:39.029977    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b21b58de-81cf-4ff7-9352-40854e5e0246-metrics-tls\") pod \"dns-default-xs54n\" (UID: \"b21b58de-81cf-4ff7-9352-40854e5e0246\") " pod="openshift-dns/dns-default-xs54n"
Sep 09 16:18:39.177529 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:39.177494    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zjfwh\""
Sep 09 16:18:39.185721 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:39.185686    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xs54n"
Sep 09 16:18:39.457246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:39.457167    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xs54n"]
Sep 09 16:18:39.460368 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:18:39.460338    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb21b58de_81cf_4ff7_9352_40854e5e0246.slice/crio-0a87ffe197942f57fce69eb7d41a8d3619317fa514ba6d8af67a6ba516696f7f WatchSource:0}: Error finding container 0a87ffe197942f57fce69eb7d41a8d3619317fa514ba6d8af67a6ba516696f7f: Status 404 returned error can't find the container with id 0a87ffe197942f57fce69eb7d41a8d3619317fa514ba6d8af67a6ba516696f7f
Sep 09 16:18:39.587893 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:39.587857    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xs54n" event={"ID":"b21b58de-81cf-4ff7-9352-40854e5e0246","Type":"ContainerStarted","Data":"0a87ffe197942f57fce69eb7d41a8d3619317fa514ba6d8af67a6ba516696f7f"}
Sep 09 16:18:41.458351 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.458324    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cdbb5756c-m4dkn"]
Sep 09 16:18:41.461246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.461227    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:18:41.465129 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.465001    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 16:18:41.465129 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.465089    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 16:18:41.465129 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.465094    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 16:18:41.465391 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.465165    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 16:18:41.465391 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.465187    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 16:18:41.465391 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.465110    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mnlm5\""
Sep 09 16:18:41.465391 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.465242    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 16:18:41.465593 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.465533    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 16:18:41.473215 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.473174    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cdbb5756c-m4dkn"]
Sep 09 16:18:41.549893 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.549862    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-service-ca\") pod \"console-6cdbb5756c-m4dkn\" (UID: \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\") " pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:18:41.550044 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.549908    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-console-serving-cert\") pod \"console-6cdbb5756c-m4dkn\" (UID: \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\") " pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:18:41.550044 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.550020    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-console-config\") pod \"console-6cdbb5756c-m4dkn\" (UID: \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\") " pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:18:41.550164 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.550095    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-oauth-serving-cert\") pod \"console-6cdbb5756c-m4dkn\" (UID: \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\") " pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:18:41.550164 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.550142    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjmdd\" (UniqueName: \"kubernetes.io/projected/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-kube-api-access-bjmdd\") pod \"console-6cdbb5756c-m4dkn\" (UID: \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\") " pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:18:41.550259 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.550192    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-console-oauth-config\") pod \"console-6cdbb5756c-m4dkn\" (UID: \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\") " pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:18:41.598718 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.598633    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xs54n" event={"ID":"b21b58de-81cf-4ff7-9352-40854e5e0246","Type":"ContainerStarted","Data":"2d1b4dbaabf3a93b0dff1dbea8ef32fab6e1a45ca8d6d124b03c1908fac2748e"}
Sep 09 16:18:41.651268 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.651231    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-service-ca\") pod \"console-6cdbb5756c-m4dkn\" (UID: \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\") " pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:18:41.651268 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.651279    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-console-serving-cert\") pod \"console-6cdbb5756c-m4dkn\" (UID: \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\") " pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:18:41.651544 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.651361    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-console-config\") pod \"console-6cdbb5756c-m4dkn\" (UID: \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\") " pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:18:41.651544 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.651423    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-oauth-serving-cert\") pod \"console-6cdbb5756c-m4dkn\" (UID: \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\") " pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:18:41.651544 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.651466    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjmdd\" (UniqueName: \"kubernetes.io/projected/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-kube-api-access-bjmdd\") pod \"console-6cdbb5756c-m4dkn\" (UID: \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\") " pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:18:41.651544 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.651499    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-console-oauth-config\") pod \"console-6cdbb5756c-m4dkn\" (UID: \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\") " pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:18:41.652110 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.652078    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-service-ca\") pod \"console-6cdbb5756c-m4dkn\" (UID: \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\") " pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:18:41.652227 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.652116    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-console-config\") pod \"console-6cdbb5756c-m4dkn\" (UID: \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\") " pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:18:41.652408 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.652385    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-oauth-serving-cert\") pod \"console-6cdbb5756c-m4dkn\" (UID: \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\") " pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:18:41.654038 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.654015    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-console-oauth-config\") pod \"console-6cdbb5756c-m4dkn\" (UID: \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\") " pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:18:41.654416 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.654395    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-console-serving-cert\") pod \"console-6cdbb5756c-m4dkn\" (UID: \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\") " pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:18:41.659420 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.659386    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjmdd\" (UniqueName: \"kubernetes.io/projected/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-kube-api-access-bjmdd\") pod \"console-6cdbb5756c-m4dkn\" (UID: \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\") " pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:18:41.779080 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:41.779040    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:18:42.049282 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:42.049255    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cdbb5756c-m4dkn"]
Sep 09 16:18:42.051844 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:18:42.051809    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f555fa_866d_4ec8_87bf_bc67cc00bfb6.slice/crio-71244e208da29886f48907a48fc5575655b162e74c2e306223e8cefcb52b3468 WatchSource:0}: Error finding container 71244e208da29886f48907a48fc5575655b162e74c2e306223e8cefcb52b3468: Status 404 returned error can't find the container with id 71244e208da29886f48907a48fc5575655b162e74c2e306223e8cefcb52b3468
Sep 09 16:18:42.605684 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:42.605654    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xs54n"
Sep 09 16:18:42.606059 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:42.605698    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cdbb5756c-m4dkn" event={"ID":"30f555fa-866d-4ec8-87bf-bc67cc00bfb6","Type":"ContainerStarted","Data":"71244e208da29886f48907a48fc5575655b162e74c2e306223e8cefcb52b3468"}
Sep 09 16:18:42.606059 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:42.605718    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xs54n" event={"ID":"b21b58de-81cf-4ff7-9352-40854e5e0246","Type":"ContainerStarted","Data":"06c7a4de0a0d4c510d0c8c627c0759c2bcd422e9014af45cacfdf5d9d7a8ec3c"}
Sep 09 16:18:42.622400 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:42.622343    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xs54n" podStartSLOduration=251.629548767 podStartE2EDuration="4m13.622325953s" totalImagesPullingTime="1.992777186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:18:42.620515089 +0000 UTC m=+288.450723147" watchObservedRunningTime="2026-09-09 16:18:42.622325953 +0000 UTC m=+288.452534012"
Sep 09 16:18:43.269014 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:43.268970    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c931a390-ae5a-451b-9b76-258aa5ff5bba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:43.271638 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:43.271608    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c931a390-ae5a-451b-9b76-258aa5ff5bba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5wzqv\" (UID: \"c931a390-ae5a-451b-9b76-258aa5ff5bba\") " pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:43.513588 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:43.513542    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5wzqv"
Sep 09 16:18:43.797321 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:43.797208    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5wzqv"]
Sep 09 16:18:43.802352 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:18:43.802313    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc931a390_ae5a_451b_9b76_258aa5ff5bba.slice/crio-06e8f3c20cdab2377dbe68f2da8bfd05b0ed61f0ced8e130745bb23884186b2e WatchSource:0}: Error finding container 06e8f3c20cdab2377dbe68f2da8bfd05b0ed61f0ced8e130745bb23884186b2e: Status 404 returned error can't find the container with id 06e8f3c20cdab2377dbe68f2da8bfd05b0ed61f0ced8e130745bb23884186b2e
Sep 09 16:18:44.619835 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:44.619798    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5wzqv" event={"ID":"c931a390-ae5a-451b-9b76-258aa5ff5bba","Type":"ContainerStarted","Data":"2a9907c7bebfca2ea8c298d154e847a15c8f4a13e7f9c7facbbcf5ad9eb6237c"}
Sep 09 16:18:44.620023 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:44.619902    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5wzqv" event={"ID":"c931a390-ae5a-451b-9b76-258aa5ff5bba","Type":"ContainerStarted","Data":"06e8f3c20cdab2377dbe68f2da8bfd05b0ed61f0ced8e130745bb23884186b2e"}
Sep 09 16:18:45.624834 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:45.624799    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cdbb5756c-m4dkn" event={"ID":"30f555fa-866d-4ec8-87bf-bc67cc00bfb6","Type":"ContainerStarted","Data":"22a544d9b512b85e247dfe6012683b14177600209c8852cede4ba79b98e6eb7d"}
Sep 09 16:18:45.626714 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:45.626687    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5wzqv" event={"ID":"c931a390-ae5a-451b-9b76-258aa5ff5bba","Type":"ContainerStarted","Data":"ecfadc273720d9fa86dd301e617c55125d715618c7922314ed6b28b0ce3c4fd7"}
Sep 09 16:18:45.641431 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:45.641379    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6cdbb5756c-m4dkn" podStartSLOduration=1.494287926 podStartE2EDuration="4.641362801s" totalImagesPullingTime="3.147074875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:18:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:18:45.64020299 +0000 UTC m=+291.470411052" watchObservedRunningTime="2026-09-09 16:18:45.641362801 +0000 UTC m=+291.471570850"
Sep 09 16:18:47.629272 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.629237    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67b86bdb74-svv8p"]
Sep 09 16:18:47.632308 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.632275    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:47.641470 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.641445    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 16:18:47.645966 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.645939    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67b86bdb74-svv8p"]
Sep 09 16:18:47.706019 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.705990    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24fe585c-09f3-452f-a843-8f946af4bd32-service-ca\") pod \"console-67b86bdb74-svv8p\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") " pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:47.706178 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.706030    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24fe585c-09f3-452f-a843-8f946af4bd32-trusted-ca-bundle\") pod \"console-67b86bdb74-svv8p\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") " pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:47.706178 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.706105    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24fe585c-09f3-452f-a843-8f946af4bd32-console-oauth-config\") pod \"console-67b86bdb74-svv8p\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") " pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:47.706178 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.706144    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24fe585c-09f3-452f-a843-8f946af4bd32-console-config\") pod \"console-67b86bdb74-svv8p\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") " pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:47.706178 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.706162    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlj5n\" (UniqueName: \"kubernetes.io/projected/24fe585c-09f3-452f-a843-8f946af4bd32-kube-api-access-qlj5n\") pod \"console-67b86bdb74-svv8p\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") " pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:47.706400 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.706181    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24fe585c-09f3-452f-a843-8f946af4bd32-console-serving-cert\") pod \"console-67b86bdb74-svv8p\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") " pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:47.706400 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.706282    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24fe585c-09f3-452f-a843-8f946af4bd32-oauth-serving-cert\") pod \"console-67b86bdb74-svv8p\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") " pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:47.806704 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.806654    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24fe585c-09f3-452f-a843-8f946af4bd32-service-ca\") pod \"console-67b86bdb74-svv8p\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") " pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:47.806704 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.806706    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24fe585c-09f3-452f-a843-8f946af4bd32-trusted-ca-bundle\") pod \"console-67b86bdb74-svv8p\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") " pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:47.806921 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.806749    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24fe585c-09f3-452f-a843-8f946af4bd32-console-oauth-config\") pod \"console-67b86bdb74-svv8p\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") " pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:47.806921 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.806776    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24fe585c-09f3-452f-a843-8f946af4bd32-console-config\") pod \"console-67b86bdb74-svv8p\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") " pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:47.806921 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.806799    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlj5n\" (UniqueName: \"kubernetes.io/projected/24fe585c-09f3-452f-a843-8f946af4bd32-kube-api-access-qlj5n\") pod \"console-67b86bdb74-svv8p\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") " pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:47.806921 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.806827    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24fe585c-09f3-452f-a843-8f946af4bd32-console-serving-cert\") pod \"console-67b86bdb74-svv8p\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") " pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:47.807106 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.806986    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24fe585c-09f3-452f-a843-8f946af4bd32-oauth-serving-cert\") pod \"console-67b86bdb74-svv8p\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") " pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:47.807569 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.807541    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24fe585c-09f3-452f-a843-8f946af4bd32-service-ca\") pod \"console-67b86bdb74-svv8p\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") " pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:47.807717 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.807541    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24fe585c-09f3-452f-a843-8f946af4bd32-console-config\") pod \"console-67b86bdb74-svv8p\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") " pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:47.807717 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.807673    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24fe585c-09f3-452f-a843-8f946af4bd32-trusted-ca-bundle\") pod \"console-67b86bdb74-svv8p\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") " pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:47.807839 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.807751    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24fe585c-09f3-452f-a843-8f946af4bd32-oauth-serving-cert\") pod \"console-67b86bdb74-svv8p\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") " pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:47.809327 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.809305    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24fe585c-09f3-452f-a843-8f946af4bd32-console-serving-cert\") pod \"console-67b86bdb74-svv8p\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") " pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:47.809458 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.809442    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24fe585c-09f3-452f-a843-8f946af4bd32-console-oauth-config\") pod \"console-67b86bdb74-svv8p\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") " pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:47.815644 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.815623    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlj5n\" (UniqueName: \"kubernetes.io/projected/24fe585c-09f3-452f-a843-8f946af4bd32-kube-api-access-qlj5n\") pod \"console-67b86bdb74-svv8p\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") " pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:47.942086 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:47.941991    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:48.215636 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:48.215593    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67b86bdb74-svv8p"]
Sep 09 16:18:48.218140 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:18:48.218108    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24fe585c_09f3_452f_a843_8f946af4bd32.slice/crio-7792657eeb414c71b0fb733a99d3470dd5122fc97496a637089a754ccaa66b26 WatchSource:0}: Error finding container 7792657eeb414c71b0fb733a99d3470dd5122fc97496a637089a754ccaa66b26: Status 404 returned error can't find the container with id 7792657eeb414c71b0fb733a99d3470dd5122fc97496a637089a754ccaa66b26
Sep 09 16:18:48.640719 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:48.640673    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67b86bdb74-svv8p" event={"ID":"24fe585c-09f3-452f-a843-8f946af4bd32","Type":"ContainerStarted","Data":"32a73dadc60deb7493d3886444d6317ab185f46864a16bc1094e2b2cd0fa4986"}
Sep 09 16:18:48.641183 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:48.640743    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67b86bdb74-svv8p" event={"ID":"24fe585c-09f3-452f-a843-8f946af4bd32","Type":"ContainerStarted","Data":"7792657eeb414c71b0fb733a99d3470dd5122fc97496a637089a754ccaa66b26"}
Sep 09 16:18:48.659313 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:48.659239    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67b86bdb74-svv8p" podStartSLOduration=1.6592146589999999 podStartE2EDuration="1.659214659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:18:48.65819061 +0000 UTC m=+294.488398670" watchObservedRunningTime="2026-09-09 16:18:48.659214659 +0000 UTC m=+294.489422721"
Sep 09 16:18:49.603125 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:18:49.603088    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-gcb4p" podUID="7334d4d3-9b79-4b0d-ab28-e0aec88c18cb"
Sep 09 16:18:49.646153 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:49.646116    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5wzqv" event={"ID":"c931a390-ae5a-451b-9b76-258aa5ff5bba","Type":"ContainerStarted","Data":"51fb89356194beea97416ee779aeb51d13fecd79843951a12625b6f81f066ac8"}
Sep 09 16:18:49.666959 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:49.666903    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5wzqv" podStartSLOduration=33.100286587 podStartE2EDuration="38.666885842s" totalImagesPullingTime="5.566599255s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:18:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:18:49.665611851 +0000 UTC m=+295.495819911" watchObservedRunningTime="2026-09-09 16:18:49.666885842 +0000 UTC m=+295.497093900"
Sep 09 16:18:51.779763 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:51.779723    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:18:51.780238 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:51.779792    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:18:51.787060 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:51.787030    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:18:52.613048 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:52.613014    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xs54n"
Sep 09 16:18:52.661284 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:52.661245    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:18:54.517419 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:54.517383    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgcjl_561d9dd7-04c3-42dc-bcf6-2c92fd61706b/console-operator/1.log"
Sep 09 16:18:54.517921 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:54.517642    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgcjl_561d9dd7-04c3-42dc-bcf6-2c92fd61706b/console-operator/1.log"
Sep 09 16:18:54.529045 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:54.529016    2555 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 16:18:55.578911 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:55.578882    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-cc689c778-vg2sn"
Sep 09 16:18:57.942939 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:57.942899    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:57.943367 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:57.942972    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:57.947868 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:57.947845    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:58.680270 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:58.680242    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:18:58.737360 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:58.737321    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cdbb5756c-m4dkn"]
Sep 09 16:18:59.026109 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.026066    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5zhcs"]
Sep 09 16:18:59.029998 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.029971    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.038484 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.038454    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 16:18:59.038701 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.038454    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 16:18:59.038792 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.038477    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 16:18:59.038905 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.038563    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 16:18:59.038969 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.038578    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 16:18:59.039035 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.038616    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zlvb9\""
Sep 09 16:18:59.039642 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.039621    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 16:18:59.214756 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.214721    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c1fc3122-4351-4a27-9fca-ba12043e0f2a-root\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.214756 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.214758    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c1fc3122-4351-4a27-9fca-ba12043e0f2a-node-exporter-wtmp\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.214981 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.214841    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wlzt\" (UniqueName: \"kubernetes.io/projected/c1fc3122-4351-4a27-9fca-ba12043e0f2a-kube-api-access-5wlzt\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.214981 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.214866    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1fc3122-4351-4a27-9fca-ba12043e0f2a-metrics-client-ca\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.214981 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.214959    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1fc3122-4351-4a27-9fca-ba12043e0f2a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.215079 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.214992    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c1fc3122-4351-4a27-9fca-ba12043e0f2a-node-exporter-textfile\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.215079 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.215021    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c1fc3122-4351-4a27-9fca-ba12043e0f2a-node-exporter-tls\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.215079 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.215048    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c1fc3122-4351-4a27-9fca-ba12043e0f2a-node-exporter-accelerators-collector-config\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.215168 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.215102    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1fc3122-4351-4a27-9fca-ba12043e0f2a-sys\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.315790 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.315695    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wlzt\" (UniqueName: \"kubernetes.io/projected/c1fc3122-4351-4a27-9fca-ba12043e0f2a-kube-api-access-5wlzt\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.315790 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.315737    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1fc3122-4351-4a27-9fca-ba12043e0f2a-metrics-client-ca\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.315973 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.315863    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1fc3122-4351-4a27-9fca-ba12043e0f2a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.315973 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.315904    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c1fc3122-4351-4a27-9fca-ba12043e0f2a-node-exporter-textfile\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.315973 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.315928    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c1fc3122-4351-4a27-9fca-ba12043e0f2a-node-exporter-tls\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.315973 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.315960    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c1fc3122-4351-4a27-9fca-ba12043e0f2a-node-exporter-accelerators-collector-config\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.316163 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.316137    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1fc3122-4351-4a27-9fca-ba12043e0f2a-sys\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.316222 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.316192    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c1fc3122-4351-4a27-9fca-ba12043e0f2a-root\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.316279 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.316236    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c1fc3122-4351-4a27-9fca-ba12043e0f2a-node-exporter-wtmp\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.316279 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.316254    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c1fc3122-4351-4a27-9fca-ba12043e0f2a-root\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.316423 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.316268    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c1fc3122-4351-4a27-9fca-ba12043e0f2a-node-exporter-textfile\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.316423 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.316323    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1fc3122-4351-4a27-9fca-ba12043e0f2a-sys\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.316519 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.316463    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c1fc3122-4351-4a27-9fca-ba12043e0f2a-node-exporter-wtmp\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.316519 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.316469    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1fc3122-4351-4a27-9fca-ba12043e0f2a-metrics-client-ca\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.316598 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.316518    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c1fc3122-4351-4a27-9fca-ba12043e0f2a-node-exporter-accelerators-collector-config\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.318479 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.318449    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1fc3122-4351-4a27-9fca-ba12043e0f2a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.318479 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.318460    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c1fc3122-4351-4a27-9fca-ba12043e0f2a-node-exporter-tls\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.324419 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.324393    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wlzt\" (UniqueName: \"kubernetes.io/projected/c1fc3122-4351-4a27-9fca-ba12043e0f2a-kube-api-access-5wlzt\") pod \"node-exporter-5zhcs\" (UID: \"c1fc3122-4351-4a27-9fca-ba12043e0f2a\") " pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.347084 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.347040    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5zhcs"
Sep 09 16:18:59.357731 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:18:59.357686    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1fc3122_4351_4a27_9fca_ba12043e0f2a.slice/crio-e549636af19ae6a5c1137746e0430c9972b8a3c5cf8fa996ecac3d58bf82fddc WatchSource:0}: Error finding container e549636af19ae6a5c1137746e0430c9972b8a3c5cf8fa996ecac3d58bf82fddc: Status 404 returned error can't find the container with id e549636af19ae6a5c1137746e0430c9972b8a3c5cf8fa996ecac3d58bf82fddc
Sep 09 16:18:59.359709 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.359689    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 16:18:59.681237 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:18:59.681148    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5zhcs" event={"ID":"c1fc3122-4351-4a27-9fca-ba12043e0f2a","Type":"ContainerStarted","Data":"e549636af19ae6a5c1137746e0430c9972b8a3c5cf8fa996ecac3d58bf82fddc"}
Sep 09 16:19:01.689043 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:01.689006    2555 generic.go:356] "Generic (PLEG): container finished" podID="c1fc3122-4351-4a27-9fca-ba12043e0f2a" containerID="cb3f28f43278f0079e1b96276305ba54cda856aba5beba4fb468a34152a8f1e0" exitCode=0
Sep 09 16:19:01.689446 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:01.689097    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5zhcs" event={"ID":"c1fc3122-4351-4a27-9fca-ba12043e0f2a","Type":"ContainerDied","Data":"cb3f28f43278f0079e1b96276305ba54cda856aba5beba4fb468a34152a8f1e0"}
Sep 09 16:19:02.602080 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:02.602046    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:19:02.695090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:02.695041    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5zhcs" event={"ID":"c1fc3122-4351-4a27-9fca-ba12043e0f2a","Type":"ContainerStarted","Data":"84d5688a8a12e2200b546fbb74b14d4909801105eb40d3971173836ac83a63a0"}
Sep 09 16:19:02.695465 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:02.695109    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5zhcs" event={"ID":"c1fc3122-4351-4a27-9fca-ba12043e0f2a","Type":"ContainerStarted","Data":"0f6352bcfb5fdd53a1248cf2e9127f02cc62dadbcfc33678368bc475b8349c2b"}
Sep 09 16:19:02.716119 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:02.716064    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5zhcs" podStartSLOduration=3.339843049 podStartE2EDuration="4.716048528s" totalImagesPullingTime="1.313379732s" totalInitContainerRuntime="62.825747ms" isStatefulPod=true podCreationTimestamp="2026-09-09 16:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:19:02.714421796 +0000 UTC m=+308.544629880" watchObservedRunningTime="2026-09-09 16:19:02.716048528 +0000 UTC m=+308.546256587"
Sep 09 16:19:03.830954 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:03.830918    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-797b4dc79b-fxfh8"]
Sep 09 16:19:03.834065 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:03.834045    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:03.844392 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:03.844366    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-797b4dc79b-fxfh8"]
Sep 09 16:19:03.859690 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:03.859653    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9fa0138-7f51-4edd-a486-e06e8fea92d1-console-oauth-config\") pod \"console-797b4dc79b-fxfh8\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") " pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:03.859826 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:03.859697    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9fa0138-7f51-4edd-a486-e06e8fea92d1-trusted-ca-bundle\") pod \"console-797b4dc79b-fxfh8\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") " pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:03.859826 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:03.859724    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fa0138-7f51-4edd-a486-e06e8fea92d1-service-ca\") pod \"console-797b4dc79b-fxfh8\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") " pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:03.859826 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:03.859761    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9fa0138-7f51-4edd-a486-e06e8fea92d1-oauth-serving-cert\") pod \"console-797b4dc79b-fxfh8\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") " pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:03.859826 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:03.859825    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9fa0138-7f51-4edd-a486-e06e8fea92d1-console-serving-cert\") pod \"console-797b4dc79b-fxfh8\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") " pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:03.859987 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:03.859839    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9fa0138-7f51-4edd-a486-e06e8fea92d1-console-config\") pod \"console-797b4dc79b-fxfh8\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") " pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:03.859987 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:03.859910    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prpc6\" (UniqueName: \"kubernetes.io/projected/a9fa0138-7f51-4edd-a486-e06e8fea92d1-kube-api-access-prpc6\") pod \"console-797b4dc79b-fxfh8\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") " pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:03.960699 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:03.960654    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9fa0138-7f51-4edd-a486-e06e8fea92d1-oauth-serving-cert\") pod \"console-797b4dc79b-fxfh8\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") " pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:03.960886 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:03.960739    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9fa0138-7f51-4edd-a486-e06e8fea92d1-console-serving-cert\") pod \"console-797b4dc79b-fxfh8\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") " pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:03.960886 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:03.960757    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9fa0138-7f51-4edd-a486-e06e8fea92d1-console-config\") pod \"console-797b4dc79b-fxfh8\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") " pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:03.960886 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:03.960776    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prpc6\" (UniqueName: \"kubernetes.io/projected/a9fa0138-7f51-4edd-a486-e06e8fea92d1-kube-api-access-prpc6\") pod \"console-797b4dc79b-fxfh8\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") " pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:03.960886 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:03.960803    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9fa0138-7f51-4edd-a486-e06e8fea92d1-console-oauth-config\") pod \"console-797b4dc79b-fxfh8\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") " pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:03.960886 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:03.960826    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9fa0138-7f51-4edd-a486-e06e8fea92d1-trusted-ca-bundle\") pod \"console-797b4dc79b-fxfh8\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") " pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:03.960886 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:03.960850    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fa0138-7f51-4edd-a486-e06e8fea92d1-service-ca\") pod \"console-797b4dc79b-fxfh8\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") " pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:03.961591 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:03.961560    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9fa0138-7f51-4edd-a486-e06e8fea92d1-oauth-serving-cert\") pod \"console-797b4dc79b-fxfh8\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") " pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:03.961722 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:03.961593    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9fa0138-7f51-4edd-a486-e06e8fea92d1-console-config\") pod \"console-797b4dc79b-fxfh8\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") " pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:03.961722 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:03.961606    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fa0138-7f51-4edd-a486-e06e8fea92d1-service-ca\") pod \"console-797b4dc79b-fxfh8\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") " pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:03.961845 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:03.961757    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9fa0138-7f51-4edd-a486-e06e8fea92d1-trusted-ca-bundle\") pod \"console-797b4dc79b-fxfh8\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") " pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:03.963326 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:03.963272    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9fa0138-7f51-4edd-a486-e06e8fea92d1-console-serving-cert\") pod \"console-797b4dc79b-fxfh8\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") " pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:03.963445 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:03.963420    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9fa0138-7f51-4edd-a486-e06e8fea92d1-console-oauth-config\") pod \"console-797b4dc79b-fxfh8\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") " pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:03.968509 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:03.968483    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prpc6\" (UniqueName: \"kubernetes.io/projected/a9fa0138-7f51-4edd-a486-e06e8fea92d1-kube-api-access-prpc6\") pod \"console-797b4dc79b-fxfh8\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") " pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:04.144649 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:04.144534    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:04.417923 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:04.417844    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-797b4dc79b-fxfh8"]
Sep 09 16:19:04.421267 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:19:04.421235    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9fa0138_7f51_4edd_a486_e06e8fea92d1.slice/crio-bcb8487d1583ed964fad8e2a8a5d41b985b7e7cc6c1db4d665770c0df2484b38 WatchSource:0}: Error finding container bcb8487d1583ed964fad8e2a8a5d41b985b7e7cc6c1db4d665770c0df2484b38: Status 404 returned error can't find the container with id bcb8487d1583ed964fad8e2a8a5d41b985b7e7cc6c1db4d665770c0df2484b38
Sep 09 16:19:04.703196 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:04.703115    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-797b4dc79b-fxfh8" event={"ID":"a9fa0138-7f51-4edd-a486-e06e8fea92d1","Type":"ContainerStarted","Data":"17836725ac653437342a5e1b6306ef1cd6f32a47087c9e726d3a002e67940827"}
Sep 09 16:19:04.703196 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:04.703163    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-797b4dc79b-fxfh8" event={"ID":"a9fa0138-7f51-4edd-a486-e06e8fea92d1","Type":"ContainerStarted","Data":"bcb8487d1583ed964fad8e2a8a5d41b985b7e7cc6c1db4d665770c0df2484b38"}
Sep 09 16:19:04.720219 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:04.720164    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-797b4dc79b-fxfh8" podStartSLOduration=1.7201498069999999 podStartE2EDuration="1.720149807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:19:04.720005074 +0000 UTC m=+310.550213147" watchObservedRunningTime="2026-09-09 16:19:04.720149807 +0000 UTC m=+310.550357867"
Sep 09 16:19:14.145353 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:14.145279    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:14.145823 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:14.145380    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:14.149751 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:14.149731    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:14.741285 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:14.741259    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:19:14.785616 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:14.785587    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67b86bdb74-svv8p"]
Sep 09 16:19:23.758579 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:23.758535    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6cdbb5756c-m4dkn" podUID="30f555fa-866d-4ec8-87bf-bc67cc00bfb6" containerName="console" containerID="cri-o://22a544d9b512b85e247dfe6012683b14177600209c8852cede4ba79b98e6eb7d" gracePeriod=15
Sep 09 16:19:23.996020 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:23.995994    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:19:24.092569 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.092483    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-console-oauth-config\") pod \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\" (UID: \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\") "
Sep 09 16:19:24.092569 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.092557    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-oauth-serving-cert\") pod \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\" (UID: \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\") "
Sep 09 16:19:24.092746 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.092586    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-console-serving-cert\") pod \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\" (UID: \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\") "
Sep 09 16:19:24.092746 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.092605    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-console-config\") pod \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\" (UID: \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\") "
Sep 09 16:19:24.092746 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.092627    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjmdd\" (UniqueName: \"kubernetes.io/projected/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-kube-api-access-bjmdd\") pod \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\" (UID: \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\") "
Sep 09 16:19:24.092746 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.092658    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-service-ca\") pod \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\" (UID: \"30f555fa-866d-4ec8-87bf-bc67cc00bfb6\") "
Sep 09 16:19:24.092938 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.092913    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "30f555fa-866d-4ec8-87bf-bc67cc00bfb6" (UID: "30f555fa-866d-4ec8-87bf-bc67cc00bfb6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:19:24.093032 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.092995    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-console-config" (OuterVolumeSpecName: "console-config") pod "30f555fa-866d-4ec8-87bf-bc67cc00bfb6" (UID: "30f555fa-866d-4ec8-87bf-bc67cc00bfb6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:19:24.093146 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.093072    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-service-ca" (OuterVolumeSpecName: "service-ca") pod "30f555fa-866d-4ec8-87bf-bc67cc00bfb6" (UID: "30f555fa-866d-4ec8-87bf-bc67cc00bfb6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:19:24.094777 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.094749    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "30f555fa-866d-4ec8-87bf-bc67cc00bfb6" (UID: "30f555fa-866d-4ec8-87bf-bc67cc00bfb6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:19:24.094854 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.094796    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "30f555fa-866d-4ec8-87bf-bc67cc00bfb6" (UID: "30f555fa-866d-4ec8-87bf-bc67cc00bfb6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:19:24.094854 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.094809    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-kube-api-access-bjmdd" (OuterVolumeSpecName: "kube-api-access-bjmdd") pod "30f555fa-866d-4ec8-87bf-bc67cc00bfb6" (UID: "30f555fa-866d-4ec8-87bf-bc67cc00bfb6"). InnerVolumeSpecName "kube-api-access-bjmdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:19:24.193207 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.193175    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjmdd\" (UniqueName: \"kubernetes.io/projected/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-kube-api-access-bjmdd\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:19:24.193207 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.193199    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-service-ca\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:19:24.193207 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.193209    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-console-oauth-config\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:19:24.193417 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.193217    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-oauth-serving-cert\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:19:24.193417 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.193226    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-console-serving-cert\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:19:24.193417 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.193234    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30f555fa-866d-4ec8-87bf-bc67cc00bfb6-console-config\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:19:24.773666 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.773631    2555 generic.go:356] "Generic (PLEG): container finished" podID="30f555fa-866d-4ec8-87bf-bc67cc00bfb6" containerID="22a544d9b512b85e247dfe6012683b14177600209c8852cede4ba79b98e6eb7d" exitCode=0
Sep 09 16:19:24.774108 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.773675    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cdbb5756c-m4dkn"
Sep 09 16:19:24.774108 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.773714    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cdbb5756c-m4dkn" event={"ID":"30f555fa-866d-4ec8-87bf-bc67cc00bfb6","Type":"ContainerDied","Data":"22a544d9b512b85e247dfe6012683b14177600209c8852cede4ba79b98e6eb7d"}
Sep 09 16:19:24.774108 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.773762    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cdbb5756c-m4dkn" event={"ID":"30f555fa-866d-4ec8-87bf-bc67cc00bfb6","Type":"ContainerDied","Data":"71244e208da29886f48907a48fc5575655b162e74c2e306223e8cefcb52b3468"}
Sep 09 16:19:24.774108 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.773784    2555 scope.go:117] "RemoveContainer" containerID="22a544d9b512b85e247dfe6012683b14177600209c8852cede4ba79b98e6eb7d"
Sep 09 16:19:24.782109 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.782091    2555 scope.go:117] "RemoveContainer" containerID="22a544d9b512b85e247dfe6012683b14177600209c8852cede4ba79b98e6eb7d"
Sep 09 16:19:24.782412 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:19:24.782388    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22a544d9b512b85e247dfe6012683b14177600209c8852cede4ba79b98e6eb7d\": container with ID starting with 22a544d9b512b85e247dfe6012683b14177600209c8852cede4ba79b98e6eb7d not found: ID does not exist" containerID="22a544d9b512b85e247dfe6012683b14177600209c8852cede4ba79b98e6eb7d"
Sep 09 16:19:24.782463 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.782427    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22a544d9b512b85e247dfe6012683b14177600209c8852cede4ba79b98e6eb7d"} err="failed to get container status \"22a544d9b512b85e247dfe6012683b14177600209c8852cede4ba79b98e6eb7d\": rpc error: code = NotFound desc = could not find container \"22a544d9b512b85e247dfe6012683b14177600209c8852cede4ba79b98e6eb7d\": container with ID starting with 22a544d9b512b85e247dfe6012683b14177600209c8852cede4ba79b98e6eb7d not found: ID does not exist"
Sep 09 16:19:24.791353 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.791332    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cdbb5756c-m4dkn"]
Sep 09 16:19:24.795058 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:24.795037    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cdbb5756c-m4dkn"]
Sep 09 16:19:26.606396 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:26.606353    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30f555fa-866d-4ec8-87bf-bc67cc00bfb6" path="/var/lib/kubelet/pods/30f555fa-866d-4ec8-87bf-bc67cc00bfb6/volumes"
Sep 09 16:19:39.806160 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:39.806087    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-67b86bdb74-svv8p" podUID="24fe585c-09f3-452f-a843-8f946af4bd32" containerName="console" containerID="cri-o://32a73dadc60deb7493d3886444d6317ab185f46864a16bc1094e2b2cd0fa4986" gracePeriod=15
Sep 09 16:19:40.043464 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.043439    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:19:40.108756 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.108699    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24fe585c-09f3-452f-a843-8f946af4bd32-console-config\") pod \"24fe585c-09f3-452f-a843-8f946af4bd32\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") "
Sep 09 16:19:40.108756 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.108730    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24fe585c-09f3-452f-a843-8f946af4bd32-console-oauth-config\") pod \"24fe585c-09f3-452f-a843-8f946af4bd32\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") "
Sep 09 16:19:40.108925 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.108804    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24fe585c-09f3-452f-a843-8f946af4bd32-console-serving-cert\") pod \"24fe585c-09f3-452f-a843-8f946af4bd32\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") "
Sep 09 16:19:40.108925 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.108822    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24fe585c-09f3-452f-a843-8f946af4bd32-service-ca\") pod \"24fe585c-09f3-452f-a843-8f946af4bd32\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") "
Sep 09 16:19:40.108925 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.108846    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24fe585c-09f3-452f-a843-8f946af4bd32-trusted-ca-bundle\") pod \"24fe585c-09f3-452f-a843-8f946af4bd32\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") "
Sep 09 16:19:40.108925 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.108871    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24fe585c-09f3-452f-a843-8f946af4bd32-oauth-serving-cert\") pod \"24fe585c-09f3-452f-a843-8f946af4bd32\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") "
Sep 09 16:19:40.108925 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.108915    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlj5n\" (UniqueName: \"kubernetes.io/projected/24fe585c-09f3-452f-a843-8f946af4bd32-kube-api-access-qlj5n\") pod \"24fe585c-09f3-452f-a843-8f946af4bd32\" (UID: \"24fe585c-09f3-452f-a843-8f946af4bd32\") "
Sep 09 16:19:40.109144 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.109062    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24fe585c-09f3-452f-a843-8f946af4bd32-console-config" (OuterVolumeSpecName: "console-config") pod "24fe585c-09f3-452f-a843-8f946af4bd32" (UID: "24fe585c-09f3-452f-a843-8f946af4bd32"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:19:40.109144 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.109124    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24fe585c-09f3-452f-a843-8f946af4bd32-console-config\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:19:40.109357 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.109335    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24fe585c-09f3-452f-a843-8f946af4bd32-service-ca" (OuterVolumeSpecName: "service-ca") pod "24fe585c-09f3-452f-a843-8f946af4bd32" (UID: "24fe585c-09f3-452f-a843-8f946af4bd32"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:19:40.109426 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.109354    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24fe585c-09f3-452f-a843-8f946af4bd32-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "24fe585c-09f3-452f-a843-8f946af4bd32" (UID: "24fe585c-09f3-452f-a843-8f946af4bd32"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:19:40.109426 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.109358    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24fe585c-09f3-452f-a843-8f946af4bd32-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "24fe585c-09f3-452f-a843-8f946af4bd32" (UID: "24fe585c-09f3-452f-a843-8f946af4bd32"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:19:40.111002 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.110974    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24fe585c-09f3-452f-a843-8f946af4bd32-kube-api-access-qlj5n" (OuterVolumeSpecName: "kube-api-access-qlj5n") pod "24fe585c-09f3-452f-a843-8f946af4bd32" (UID: "24fe585c-09f3-452f-a843-8f946af4bd32"). InnerVolumeSpecName "kube-api-access-qlj5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:19:40.111397 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.111366    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24fe585c-09f3-452f-a843-8f946af4bd32-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "24fe585c-09f3-452f-a843-8f946af4bd32" (UID: "24fe585c-09f3-452f-a843-8f946af4bd32"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:19:40.111474 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.111428    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24fe585c-09f3-452f-a843-8f946af4bd32-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "24fe585c-09f3-452f-a843-8f946af4bd32" (UID: "24fe585c-09f3-452f-a843-8f946af4bd32"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:19:40.209867 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.209842    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24fe585c-09f3-452f-a843-8f946af4bd32-console-serving-cert\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:19:40.209867 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.209862    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24fe585c-09f3-452f-a843-8f946af4bd32-service-ca\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:19:40.209867 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.209871    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24fe585c-09f3-452f-a843-8f946af4bd32-trusted-ca-bundle\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:19:40.210029 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.209880    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24fe585c-09f3-452f-a843-8f946af4bd32-oauth-serving-cert\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:19:40.210029 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.209888    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlj5n\" (UniqueName: \"kubernetes.io/projected/24fe585c-09f3-452f-a843-8f946af4bd32-kube-api-access-qlj5n\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:19:40.210029 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.209897    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24fe585c-09f3-452f-a843-8f946af4bd32-console-oauth-config\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:19:40.830161 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.830126    2555 generic.go:356] "Generic (PLEG): container finished" podID="24fe585c-09f3-452f-a843-8f946af4bd32" containerID="32a73dadc60deb7493d3886444d6317ab185f46864a16bc1094e2b2cd0fa4986" exitCode=0
Sep 09 16:19:40.830557 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.830173    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67b86bdb74-svv8p"
Sep 09 16:19:40.830557 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.830217    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67b86bdb74-svv8p" event={"ID":"24fe585c-09f3-452f-a843-8f946af4bd32","Type":"ContainerDied","Data":"32a73dadc60deb7493d3886444d6317ab185f46864a16bc1094e2b2cd0fa4986"}
Sep 09 16:19:40.830557 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.830268    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67b86bdb74-svv8p" event={"ID":"24fe585c-09f3-452f-a843-8f946af4bd32","Type":"ContainerDied","Data":"7792657eeb414c71b0fb733a99d3470dd5122fc97496a637089a754ccaa66b26"}
Sep 09 16:19:40.830557 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.830314    2555 scope.go:117] "RemoveContainer" containerID="32a73dadc60deb7493d3886444d6317ab185f46864a16bc1094e2b2cd0fa4986"
Sep 09 16:19:40.831917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.831897    2555 generic.go:356] "Generic (PLEG): container finished" podID="ca9bcf27-b906-4735-bab2-b17bffa33bce" containerID="06bd533cc23313e19f9293a4c6423a671d6cbd86c725dfcd824d36d4046d563d" exitCode=0
Sep 09 16:19:40.832014 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.831964    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn" event={"ID":"ca9bcf27-b906-4735-bab2-b17bffa33bce","Type":"ContainerDied","Data":"06bd533cc23313e19f9293a4c6423a671d6cbd86c725dfcd824d36d4046d563d"}
Sep 09 16:19:40.832338 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.832321    2555 scope.go:117] "RemoveContainer" containerID="06bd533cc23313e19f9293a4c6423a671d6cbd86c725dfcd824d36d4046d563d"
Sep 09 16:19:40.838431 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.838415    2555 scope.go:117] "RemoveContainer" containerID="32a73dadc60deb7493d3886444d6317ab185f46864a16bc1094e2b2cd0fa4986"
Sep 09 16:19:40.838693 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:19:40.838673    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32a73dadc60deb7493d3886444d6317ab185f46864a16bc1094e2b2cd0fa4986\": container with ID starting with 32a73dadc60deb7493d3886444d6317ab185f46864a16bc1094e2b2cd0fa4986 not found: ID does not exist" containerID="32a73dadc60deb7493d3886444d6317ab185f46864a16bc1094e2b2cd0fa4986"
Sep 09 16:19:40.838751 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.838699    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32a73dadc60deb7493d3886444d6317ab185f46864a16bc1094e2b2cd0fa4986"} err="failed to get container status \"32a73dadc60deb7493d3886444d6317ab185f46864a16bc1094e2b2cd0fa4986\": rpc error: code = NotFound desc = could not find container \"32a73dadc60deb7493d3886444d6317ab185f46864a16bc1094e2b2cd0fa4986\": container with ID starting with 32a73dadc60deb7493d3886444d6317ab185f46864a16bc1094e2b2cd0fa4986 not found: ID does not exist"
Sep 09 16:19:40.866784 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.866748    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67b86bdb74-svv8p"]
Sep 09 16:19:40.872748 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:40.872724    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67b86bdb74-svv8p"]
Sep 09 16:19:41.837753 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:41.837720    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-srgtn" event={"ID":"ca9bcf27-b906-4735-bab2-b17bffa33bce","Type":"ContainerStarted","Data":"1297e276eca8dfe3cc405db92a63c16e30a5fc696b7aa141cf46b06513b85f41"}
Sep 09 16:19:42.605627 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:19:42.605598    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24fe585c-09f3-452f-a843-8f946af4bd32" path="/var/lib/kubelet/pods/24fe585c-09f3-452f-a843-8f946af4bd32/volumes"
Sep 09 16:20:07.540704 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:07.540658    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs\") pod \"network-metrics-daemon-gcb4p\" (UID: \"7334d4d3-9b79-4b0d-ab28-e0aec88c18cb\") " pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:20:07.543053 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:07.543031    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7334d4d3-9b79-4b0d-ab28-e0aec88c18cb-metrics-certs\") pod \"network-metrics-daemon-gcb4p\" (UID: \"7334d4d3-9b79-4b0d-ab28-e0aec88c18cb\") " pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:20:07.706171 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:07.706136    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cjzmn\""
Sep 09 16:20:07.714514 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:07.714477    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gcb4p"
Sep 09 16:20:07.984954 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:07.984772    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gcb4p"]
Sep 09 16:20:07.987742 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:20:07.987714    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7334d4d3_9b79_4b0d_ab28_e0aec88c18cb.slice/crio-7bf8d7e6f7420f3c6f29dd9beb82c9b6cd6a8b9e92a901a80d10cfa003ffce4b WatchSource:0}: Error finding container 7bf8d7e6f7420f3c6f29dd9beb82c9b6cd6a8b9e92a901a80d10cfa003ffce4b: Status 404 returned error can't find the container with id 7bf8d7e6f7420f3c6f29dd9beb82c9b6cd6a8b9e92a901a80d10cfa003ffce4b
Sep 09 16:20:08.935606 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:08.935559    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gcb4p" event={"ID":"7334d4d3-9b79-4b0d-ab28-e0aec88c18cb","Type":"ContainerStarted","Data":"7bf8d7e6f7420f3c6f29dd9beb82c9b6cd6a8b9e92a901a80d10cfa003ffce4b"}
Sep 09 16:20:09.940778 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:09.940736    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gcb4p" event={"ID":"7334d4d3-9b79-4b0d-ab28-e0aec88c18cb","Type":"ContainerStarted","Data":"0e46eb8277aaa0334ffd803855000978812fcf3098cd65539945775c37995384"}
Sep 09 16:20:09.941243 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:09.940802    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gcb4p" event={"ID":"7334d4d3-9b79-4b0d-ab28-e0aec88c18cb","Type":"ContainerStarted","Data":"17bf61385ed7361a1459c4b9d8eabe3ba156b2356b37d245ac37510365b3bf3d"}
Sep 09 16:20:09.957406 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:09.957350    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gcb4p" podStartSLOduration=373.96229388 podStartE2EDuration="6m14.957335779s" totalImagesPullingTime="995.041899ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:20:09.956203548 +0000 UTC m=+375.786411607" watchObservedRunningTime="2026-09-09 16:20:09.957335779 +0000 UTC m=+375.787543839"
Sep 09 16:20:29.664126 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.664079    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6756975fbd-nw9zx"]
Sep 09 16:20:29.664657 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.664402    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30f555fa-866d-4ec8-87bf-bc67cc00bfb6" containerName="console"
Sep 09 16:20:29.664657 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.664417    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="30f555fa-866d-4ec8-87bf-bc67cc00bfb6" containerName="console"
Sep 09 16:20:29.664657 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.664425    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24fe585c-09f3-452f-a843-8f946af4bd32" containerName="console"
Sep 09 16:20:29.664657 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.664430    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="24fe585c-09f3-452f-a843-8f946af4bd32" containerName="console"
Sep 09 16:20:29.664657 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.664484    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="30f555fa-866d-4ec8-87bf-bc67cc00bfb6" containerName="console"
Sep 09 16:20:29.664657 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.664495    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="24fe585c-09f3-452f-a843-8f946af4bd32" containerName="console"
Sep 09 16:20:29.668464 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.668437    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:29.679028 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.678996    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6756975fbd-nw9zx"]
Sep 09 16:20:29.811002 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.810961    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8d786ed-c24f-4c5d-83cf-73120381e0c4-console-serving-cert\") pod \"console-6756975fbd-nw9zx\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") " pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:29.811002 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.811000    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8d786ed-c24f-4c5d-83cf-73120381e0c4-service-ca\") pod \"console-6756975fbd-nw9zx\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") " pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:29.811225 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.811021    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n9nn\" (UniqueName: \"kubernetes.io/projected/c8d786ed-c24f-4c5d-83cf-73120381e0c4-kube-api-access-4n9nn\") pod \"console-6756975fbd-nw9zx\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") " pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:29.811225 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.811104    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d786ed-c24f-4c5d-83cf-73120381e0c4-trusted-ca-bundle\") pod \"console-6756975fbd-nw9zx\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") " pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:29.811225 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.811131    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8d786ed-c24f-4c5d-83cf-73120381e0c4-oauth-serving-cert\") pod \"console-6756975fbd-nw9zx\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") " pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:29.811225 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.811156    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8d786ed-c24f-4c5d-83cf-73120381e0c4-console-config\") pod \"console-6756975fbd-nw9zx\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") " pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:29.811225 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.811214    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8d786ed-c24f-4c5d-83cf-73120381e0c4-console-oauth-config\") pod \"console-6756975fbd-nw9zx\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") " pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:29.911823 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.911773    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8d786ed-c24f-4c5d-83cf-73120381e0c4-console-config\") pod \"console-6756975fbd-nw9zx\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") " pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:29.912028 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.911845    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8d786ed-c24f-4c5d-83cf-73120381e0c4-console-oauth-config\") pod \"console-6756975fbd-nw9zx\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") " pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:29.912028 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.911878    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8d786ed-c24f-4c5d-83cf-73120381e0c4-console-serving-cert\") pod \"console-6756975fbd-nw9zx\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") " pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:29.912028 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.911893    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8d786ed-c24f-4c5d-83cf-73120381e0c4-service-ca\") pod \"console-6756975fbd-nw9zx\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") " pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:29.912028 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.911911    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n9nn\" (UniqueName: \"kubernetes.io/projected/c8d786ed-c24f-4c5d-83cf-73120381e0c4-kube-api-access-4n9nn\") pod \"console-6756975fbd-nw9zx\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") " pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:29.912028 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.911937    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d786ed-c24f-4c5d-83cf-73120381e0c4-trusted-ca-bundle\") pod \"console-6756975fbd-nw9zx\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") " pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:29.912028 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.911955    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8d786ed-c24f-4c5d-83cf-73120381e0c4-oauth-serving-cert\") pod \"console-6756975fbd-nw9zx\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") " pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:29.912665 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.912642    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8d786ed-c24f-4c5d-83cf-73120381e0c4-console-config\") pod \"console-6756975fbd-nw9zx\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") " pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:29.912770 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.912707    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8d786ed-c24f-4c5d-83cf-73120381e0c4-service-ca\") pod \"console-6756975fbd-nw9zx\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") " pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:29.912923 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.912897    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8d786ed-c24f-4c5d-83cf-73120381e0c4-oauth-serving-cert\") pod \"console-6756975fbd-nw9zx\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") " pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:29.912990 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.912908    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d786ed-c24f-4c5d-83cf-73120381e0c4-trusted-ca-bundle\") pod \"console-6756975fbd-nw9zx\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") " pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:29.914609 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.914547    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8d786ed-c24f-4c5d-83cf-73120381e0c4-console-oauth-config\") pod \"console-6756975fbd-nw9zx\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") " pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:29.914692 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.914607    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8d786ed-c24f-4c5d-83cf-73120381e0c4-console-serving-cert\") pod \"console-6756975fbd-nw9zx\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") " pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:29.919718 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.919698    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n9nn\" (UniqueName: \"kubernetes.io/projected/c8d786ed-c24f-4c5d-83cf-73120381e0c4-kube-api-access-4n9nn\") pod \"console-6756975fbd-nw9zx\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") " pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:29.982657 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:29.982608    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:30.255016 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:30.254801    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6756975fbd-nw9zx"]
Sep 09 16:20:30.257737 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:20:30.257709    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d786ed_c24f_4c5d_83cf_73120381e0c4.slice/crio-7135a8b7d8582a126a0453242c7b0c6f5483378efa68f1115a2c2bac6b814c9d WatchSource:0}: Error finding container 7135a8b7d8582a126a0453242c7b0c6f5483378efa68f1115a2c2bac6b814c9d: Status 404 returned error can't find the container with id 7135a8b7d8582a126a0453242c7b0c6f5483378efa68f1115a2c2bac6b814c9d
Sep 09 16:20:31.018008 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:31.017973    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6756975fbd-nw9zx" event={"ID":"c8d786ed-c24f-4c5d-83cf-73120381e0c4","Type":"ContainerStarted","Data":"017b6a25541ed391a70416f92ac1efbd6ecb0c59ad9e88bec87da8e5dfb61d32"}
Sep 09 16:20:31.018438 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:31.018024    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6756975fbd-nw9zx" event={"ID":"c8d786ed-c24f-4c5d-83cf-73120381e0c4","Type":"ContainerStarted","Data":"7135a8b7d8582a126a0453242c7b0c6f5483378efa68f1115a2c2bac6b814c9d"}
Sep 09 16:20:31.035158 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:31.035091    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6756975fbd-nw9zx" podStartSLOduration=2.035074758 podStartE2EDuration="2.035074758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:20:31.033544007 +0000 UTC m=+396.863752066" watchObservedRunningTime="2026-09-09 16:20:31.035074758 +0000 UTC m=+396.865282817"
Sep 09 16:20:39.983175 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:39.983123    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:39.983688 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:39.983250    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:39.988042 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:39.988010    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:40.056816 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:40.056782    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:20:40.103834 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:20:40.103794    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-797b4dc79b-fxfh8"]
Sep 09 16:21:05.126805 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:05.126744    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-797b4dc79b-fxfh8" podUID="a9fa0138-7f51-4edd-a486-e06e8fea92d1" containerName="console" containerID="cri-o://17836725ac653437342a5e1b6306ef1cd6f32a47087c9e726d3a002e67940827" gracePeriod=15
Sep 09 16:21:05.373133 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:05.373106    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:21:05.482856 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:05.482760    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fa0138-7f51-4edd-a486-e06e8fea92d1-service-ca\") pod \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") "
Sep 09 16:21:05.482856 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:05.482816    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9fa0138-7f51-4edd-a486-e06e8fea92d1-oauth-serving-cert\") pod \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") "
Sep 09 16:21:05.482856 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:05.482845    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9fa0138-7f51-4edd-a486-e06e8fea92d1-console-oauth-config\") pod \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") "
Sep 09 16:21:05.483105 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:05.482877    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9fa0138-7f51-4edd-a486-e06e8fea92d1-console-config\") pod \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") "
Sep 09 16:21:05.483105 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:05.482906    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9fa0138-7f51-4edd-a486-e06e8fea92d1-trusted-ca-bundle\") pod \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") "
Sep 09 16:21:05.483105 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:05.482930    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prpc6\" (UniqueName: \"kubernetes.io/projected/a9fa0138-7f51-4edd-a486-e06e8fea92d1-kube-api-access-prpc6\") pod \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") "
Sep 09 16:21:05.483105 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:05.483090    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9fa0138-7f51-4edd-a486-e06e8fea92d1-console-serving-cert\") pod \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\" (UID: \"a9fa0138-7f51-4edd-a486-e06e8fea92d1\") "
Sep 09 16:21:05.483328 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:05.483205    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9fa0138-7f51-4edd-a486-e06e8fea92d1-service-ca" (OuterVolumeSpecName: "service-ca") pod "a9fa0138-7f51-4edd-a486-e06e8fea92d1" (UID: "a9fa0138-7f51-4edd-a486-e06e8fea92d1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:21:05.483412 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:05.483384    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fa0138-7f51-4edd-a486-e06e8fea92d1-service-ca\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:21:05.483412 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:05.483382    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9fa0138-7f51-4edd-a486-e06e8fea92d1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a9fa0138-7f51-4edd-a486-e06e8fea92d1" (UID: "a9fa0138-7f51-4edd-a486-e06e8fea92d1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:21:05.483570 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:05.483516    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9fa0138-7f51-4edd-a486-e06e8fea92d1-console-config" (OuterVolumeSpecName: "console-config") pod "a9fa0138-7f51-4edd-a486-e06e8fea92d1" (UID: "a9fa0138-7f51-4edd-a486-e06e8fea92d1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:21:05.483570 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:05.483536    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9fa0138-7f51-4edd-a486-e06e8fea92d1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a9fa0138-7f51-4edd-a486-e06e8fea92d1" (UID: "a9fa0138-7f51-4edd-a486-e06e8fea92d1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:21:05.485116 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:05.485085    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9fa0138-7f51-4edd-a486-e06e8fea92d1-kube-api-access-prpc6" (OuterVolumeSpecName: "kube-api-access-prpc6") pod "a9fa0138-7f51-4edd-a486-e06e8fea92d1" (UID: "a9fa0138-7f51-4edd-a486-e06e8fea92d1"). InnerVolumeSpecName "kube-api-access-prpc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:21:05.485229 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:05.485210    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9fa0138-7f51-4edd-a486-e06e8fea92d1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a9fa0138-7f51-4edd-a486-e06e8fea92d1" (UID: "a9fa0138-7f51-4edd-a486-e06e8fea92d1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:21:05.485286 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:05.485231    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9fa0138-7f51-4edd-a486-e06e8fea92d1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a9fa0138-7f51-4edd-a486-e06e8fea92d1" (UID: "a9fa0138-7f51-4edd-a486-e06e8fea92d1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:21:05.584307 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:05.584269    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9fa0138-7f51-4edd-a486-e06e8fea92d1-oauth-serving-cert\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:21:05.584481 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:05.584318    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9fa0138-7f51-4edd-a486-e06e8fea92d1-console-oauth-config\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:21:05.584481 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:05.584333    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9fa0138-7f51-4edd-a486-e06e8fea92d1-console-config\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:21:05.584481 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:05.584345    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9fa0138-7f51-4edd-a486-e06e8fea92d1-trusted-ca-bundle\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:21:05.584481 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:05.584357    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prpc6\" (UniqueName: \"kubernetes.io/projected/a9fa0138-7f51-4edd-a486-e06e8fea92d1-kube-api-access-prpc6\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:21:05.584481 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:05.584368    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9fa0138-7f51-4edd-a486-e06e8fea92d1-console-serving-cert\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:21:06.147207 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:06.147165    2555 generic.go:356] "Generic (PLEG): container finished" podID="a9fa0138-7f51-4edd-a486-e06e8fea92d1" containerID="17836725ac653437342a5e1b6306ef1cd6f32a47087c9e726d3a002e67940827" exitCode=0
Sep 09 16:21:06.147627 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:06.147216    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-797b4dc79b-fxfh8"
Sep 09 16:21:06.147627 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:06.147259    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-797b4dc79b-fxfh8" event={"ID":"a9fa0138-7f51-4edd-a486-e06e8fea92d1","Type":"ContainerDied","Data":"17836725ac653437342a5e1b6306ef1cd6f32a47087c9e726d3a002e67940827"}
Sep 09 16:21:06.147627 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:06.147330    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-797b4dc79b-fxfh8" event={"ID":"a9fa0138-7f51-4edd-a486-e06e8fea92d1","Type":"ContainerDied","Data":"bcb8487d1583ed964fad8e2a8a5d41b985b7e7cc6c1db4d665770c0df2484b38"}
Sep 09 16:21:06.147627 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:06.147356    2555 scope.go:117] "RemoveContainer" containerID="17836725ac653437342a5e1b6306ef1cd6f32a47087c9e726d3a002e67940827"
Sep 09 16:21:06.156342 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:06.156323    2555 scope.go:117] "RemoveContainer" containerID="17836725ac653437342a5e1b6306ef1cd6f32a47087c9e726d3a002e67940827"
Sep 09 16:21:06.156695 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:21:06.156673    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17836725ac653437342a5e1b6306ef1cd6f32a47087c9e726d3a002e67940827\": container with ID starting with 17836725ac653437342a5e1b6306ef1cd6f32a47087c9e726d3a002e67940827 not found: ID does not exist" containerID="17836725ac653437342a5e1b6306ef1cd6f32a47087c9e726d3a002e67940827"
Sep 09 16:21:06.156767 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:06.156703    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17836725ac653437342a5e1b6306ef1cd6f32a47087c9e726d3a002e67940827"} err="failed to get container status \"17836725ac653437342a5e1b6306ef1cd6f32a47087c9e726d3a002e67940827\": rpc error: code = NotFound desc = could not find container \"17836725ac653437342a5e1b6306ef1cd6f32a47087c9e726d3a002e67940827\": container with ID starting with 17836725ac653437342a5e1b6306ef1cd6f32a47087c9e726d3a002e67940827 not found: ID does not exist"
Sep 09 16:21:06.169822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:06.169788    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-797b4dc79b-fxfh8"]
Sep 09 16:21:06.175427 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:06.175398    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-797b4dc79b-fxfh8"]
Sep 09 16:21:06.606742 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:06.606706    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9fa0138-7f51-4edd-a486-e06e8fea92d1" path="/var/lib/kubelet/pods/a9fa0138-7f51-4edd-a486-e06e8fea92d1/volumes"
Sep 09 16:21:35.400412 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.400365    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cd7c6fcc8-64bpg"]
Sep 09 16:21:35.400852 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.400656    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9fa0138-7f51-4edd-a486-e06e8fea92d1" containerName="console"
Sep 09 16:21:35.400852 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.400672    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9fa0138-7f51-4edd-a486-e06e8fea92d1" containerName="console"
Sep 09 16:21:35.400852 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.400722    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9fa0138-7f51-4edd-a486-e06e8fea92d1" containerName="console"
Sep 09 16:21:35.403603 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.403575    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:35.414229 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.414202    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cd7c6fcc8-64bpg"]
Sep 09 16:21:35.516710 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.516649    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8dk6\" (UniqueName: \"kubernetes.io/projected/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-kube-api-access-w8dk6\") pod \"console-cd7c6fcc8-64bpg\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") " pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:35.516710 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.516711    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-console-serving-cert\") pod \"console-cd7c6fcc8-64bpg\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") " pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:35.516939 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.516735    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-trusted-ca-bundle\") pod \"console-cd7c6fcc8-64bpg\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") " pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:35.516939 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.516751    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-service-ca\") pod \"console-cd7c6fcc8-64bpg\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") " pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:35.516939 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.516767    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-oauth-serving-cert\") pod \"console-cd7c6fcc8-64bpg\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") " pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:35.516939 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.516813    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-console-config\") pod \"console-cd7c6fcc8-64bpg\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") " pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:35.516939 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.516868    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-console-oauth-config\") pod \"console-cd7c6fcc8-64bpg\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") " pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:35.617819 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.617772    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8dk6\" (UniqueName: \"kubernetes.io/projected/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-kube-api-access-w8dk6\") pod \"console-cd7c6fcc8-64bpg\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") " pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:35.617819 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.617821    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-console-serving-cert\") pod \"console-cd7c6fcc8-64bpg\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") " pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:35.618081 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.617853    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-trusted-ca-bundle\") pod \"console-cd7c6fcc8-64bpg\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") " pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:35.618081 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.617877    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-service-ca\") pod \"console-cd7c6fcc8-64bpg\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") " pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:35.618081 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.617899    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-oauth-serving-cert\") pod \"console-cd7c6fcc8-64bpg\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") " pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:35.618081 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.617935    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-console-config\") pod \"console-cd7c6fcc8-64bpg\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") " pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:35.618278 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.618080    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-console-oauth-config\") pod \"console-cd7c6fcc8-64bpg\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") " pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:35.618687 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.618662    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-console-config\") pod \"console-cd7c6fcc8-64bpg\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") " pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:35.618818 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.618797    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-service-ca\") pod \"console-cd7c6fcc8-64bpg\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") " pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:35.618866 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.618842    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-oauth-serving-cert\") pod \"console-cd7c6fcc8-64bpg\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") " pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:35.618903 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.618873    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-trusted-ca-bundle\") pod \"console-cd7c6fcc8-64bpg\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") " pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:35.620463 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.620436    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-console-oauth-config\") pod \"console-cd7c6fcc8-64bpg\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") " pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:35.620463 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.620445    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-console-serving-cert\") pod \"console-cd7c6fcc8-64bpg\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") " pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:35.625484 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.625459    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8dk6\" (UniqueName: \"kubernetes.io/projected/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-kube-api-access-w8dk6\") pod \"console-cd7c6fcc8-64bpg\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") " pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:35.714422 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.714323    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:35.984444 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:35.984417    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cd7c6fcc8-64bpg"]
Sep 09 16:21:35.987083 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:21:35.987045    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88706fbd_166c_4f5e_8524_d7c1b3bfc2ae.slice/crio-9e5d0c4104b229e40f1689e4016143ecb5939f002e23a4f761be21dd4c163cdb WatchSource:0}: Error finding container 9e5d0c4104b229e40f1689e4016143ecb5939f002e23a4f761be21dd4c163cdb: Status 404 returned error can't find the container with id 9e5d0c4104b229e40f1689e4016143ecb5939f002e23a4f761be21dd4c163cdb
Sep 09 16:21:36.255726 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:36.255688    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd7c6fcc8-64bpg" event={"ID":"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae","Type":"ContainerStarted","Data":"f1fd63515339b1d141a24c658073efde10a6e2cb3731942b67d9b7010349a00f"}
Sep 09 16:21:36.255899 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:36.255745    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd7c6fcc8-64bpg" event={"ID":"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae","Type":"ContainerStarted","Data":"9e5d0c4104b229e40f1689e4016143ecb5939f002e23a4f761be21dd4c163cdb"}
Sep 09 16:21:36.273053 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:36.272994    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cd7c6fcc8-64bpg" podStartSLOduration=1.272977828 podStartE2EDuration="1.272977828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:21:36.271875453 +0000 UTC m=+462.102083512" watchObservedRunningTime="2026-09-09 16:21:36.272977828 +0000 UTC m=+462.103185964"
Sep 09 16:21:45.715482 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:45.715432    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:45.715853 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:45.715521    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:45.720453 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:45.720422    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:46.297739 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:46.297706    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:21:46.341032 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:21:46.340988    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6756975fbd-nw9zx"]
Sep 09 16:22:11.365409 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:11.365336    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6756975fbd-nw9zx" podUID="c8d786ed-c24f-4c5d-83cf-73120381e0c4" containerName="console" containerID="cri-o://017b6a25541ed391a70416f92ac1efbd6ecb0c59ad9e88bec87da8e5dfb61d32" gracePeriod=15
Sep 09 16:22:11.616622 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:11.616557    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:22:11.706249 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:11.706203    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d786ed-c24f-4c5d-83cf-73120381e0c4-trusted-ca-bundle\") pod \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") "
Sep 09 16:22:11.706481 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:11.706273    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8d786ed-c24f-4c5d-83cf-73120381e0c4-oauth-serving-cert\") pod \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") "
Sep 09 16:22:11.706481 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:11.706340    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4n9nn\" (UniqueName: \"kubernetes.io/projected/c8d786ed-c24f-4c5d-83cf-73120381e0c4-kube-api-access-4n9nn\") pod \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") "
Sep 09 16:22:11.706481 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:11.706368    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8d786ed-c24f-4c5d-83cf-73120381e0c4-console-oauth-config\") pod \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") "
Sep 09 16:22:11.706481 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:11.706420    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8d786ed-c24f-4c5d-83cf-73120381e0c4-service-ca\") pod \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") "
Sep 09 16:22:11.706481 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:11.706461    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8d786ed-c24f-4c5d-83cf-73120381e0c4-console-config\") pod \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") "
Sep 09 16:22:11.706716 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:11.706489    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8d786ed-c24f-4c5d-83cf-73120381e0c4-console-serving-cert\") pod \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\" (UID: \"c8d786ed-c24f-4c5d-83cf-73120381e0c4\") "
Sep 09 16:22:11.706753 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:11.706716    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8d786ed-c24f-4c5d-83cf-73120381e0c4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c8d786ed-c24f-4c5d-83cf-73120381e0c4" (UID: "c8d786ed-c24f-4c5d-83cf-73120381e0c4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:22:11.706817 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:11.706746    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8d786ed-c24f-4c5d-83cf-73120381e0c4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c8d786ed-c24f-4c5d-83cf-73120381e0c4" (UID: "c8d786ed-c24f-4c5d-83cf-73120381e0c4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:22:11.706898 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:11.706872    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8d786ed-c24f-4c5d-83cf-73120381e0c4-console-config" (OuterVolumeSpecName: "console-config") pod "c8d786ed-c24f-4c5d-83cf-73120381e0c4" (UID: "c8d786ed-c24f-4c5d-83cf-73120381e0c4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:22:11.707020 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:11.706996    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8d786ed-c24f-4c5d-83cf-73120381e0c4-service-ca" (OuterVolumeSpecName: "service-ca") pod "c8d786ed-c24f-4c5d-83cf-73120381e0c4" (UID: "c8d786ed-c24f-4c5d-83cf-73120381e0c4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:22:11.708559 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:11.708530    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d786ed-c24f-4c5d-83cf-73120381e0c4-kube-api-access-4n9nn" (OuterVolumeSpecName: "kube-api-access-4n9nn") pod "c8d786ed-c24f-4c5d-83cf-73120381e0c4" (UID: "c8d786ed-c24f-4c5d-83cf-73120381e0c4"). InnerVolumeSpecName "kube-api-access-4n9nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:22:11.708676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:11.708607    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8d786ed-c24f-4c5d-83cf-73120381e0c4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c8d786ed-c24f-4c5d-83cf-73120381e0c4" (UID: "c8d786ed-c24f-4c5d-83cf-73120381e0c4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:22:11.708725 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:11.708670    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8d786ed-c24f-4c5d-83cf-73120381e0c4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c8d786ed-c24f-4c5d-83cf-73120381e0c4" (UID: "c8d786ed-c24f-4c5d-83cf-73120381e0c4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:22:11.807225 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:11.807178    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8d786ed-c24f-4c5d-83cf-73120381e0c4-oauth-serving-cert\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:22:11.807225 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:11.807215    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n9nn\" (UniqueName: \"kubernetes.io/projected/c8d786ed-c24f-4c5d-83cf-73120381e0c4-kube-api-access-4n9nn\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:22:11.807225 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:11.807227    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8d786ed-c24f-4c5d-83cf-73120381e0c4-console-oauth-config\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:22:11.807225 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:11.807237    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8d786ed-c24f-4c5d-83cf-73120381e0c4-service-ca\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:22:11.807500 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:11.807245    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8d786ed-c24f-4c5d-83cf-73120381e0c4-console-config\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:22:11.807500 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:11.807253    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8d786ed-c24f-4c5d-83cf-73120381e0c4-console-serving-cert\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:22:11.807500 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:11.807261    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d786ed-c24f-4c5d-83cf-73120381e0c4-trusted-ca-bundle\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:22:12.391676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:12.391641    2555 generic.go:356] "Generic (PLEG): container finished" podID="c8d786ed-c24f-4c5d-83cf-73120381e0c4" containerID="017b6a25541ed391a70416f92ac1efbd6ecb0c59ad9e88bec87da8e5dfb61d32" exitCode=0
Sep 09 16:22:12.392152 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:12.391696    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6756975fbd-nw9zx"
Sep 09 16:22:12.392152 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:12.391723    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6756975fbd-nw9zx" event={"ID":"c8d786ed-c24f-4c5d-83cf-73120381e0c4","Type":"ContainerDied","Data":"017b6a25541ed391a70416f92ac1efbd6ecb0c59ad9e88bec87da8e5dfb61d32"}
Sep 09 16:22:12.392152 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:12.391779    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6756975fbd-nw9zx" event={"ID":"c8d786ed-c24f-4c5d-83cf-73120381e0c4","Type":"ContainerDied","Data":"7135a8b7d8582a126a0453242c7b0c6f5483378efa68f1115a2c2bac6b814c9d"}
Sep 09 16:22:12.392152 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:12.391799    2555 scope.go:117] "RemoveContainer" containerID="017b6a25541ed391a70416f92ac1efbd6ecb0c59ad9e88bec87da8e5dfb61d32"
Sep 09 16:22:12.399831 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:12.399808    2555 scope.go:117] "RemoveContainer" containerID="017b6a25541ed391a70416f92ac1efbd6ecb0c59ad9e88bec87da8e5dfb61d32"
Sep 09 16:22:12.400125 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:22:12.400102    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"017b6a25541ed391a70416f92ac1efbd6ecb0c59ad9e88bec87da8e5dfb61d32\": container with ID starting with 017b6a25541ed391a70416f92ac1efbd6ecb0c59ad9e88bec87da8e5dfb61d32 not found: ID does not exist" containerID="017b6a25541ed391a70416f92ac1efbd6ecb0c59ad9e88bec87da8e5dfb61d32"
Sep 09 16:22:12.400194 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:12.400139    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"017b6a25541ed391a70416f92ac1efbd6ecb0c59ad9e88bec87da8e5dfb61d32"} err="failed to get container status \"017b6a25541ed391a70416f92ac1efbd6ecb0c59ad9e88bec87da8e5dfb61d32\": rpc error: code = NotFound desc = could not find container \"017b6a25541ed391a70416f92ac1efbd6ecb0c59ad9e88bec87da8e5dfb61d32\": container with ID starting with 017b6a25541ed391a70416f92ac1efbd6ecb0c59ad9e88bec87da8e5dfb61d32 not found: ID does not exist"
Sep 09 16:22:12.424087 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:12.424043    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6756975fbd-nw9zx"]
Sep 09 16:22:12.428821 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:12.428789    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6756975fbd-nw9zx"]
Sep 09 16:22:12.606598 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:22:12.606562    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8d786ed-c24f-4c5d-83cf-73120381e0c4" path="/var/lib/kubelet/pods/c8d786ed-c24f-4c5d-83cf-73120381e0c4/volumes"
Sep 09 16:23:05.934233 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:05.934193    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj"]
Sep 09 16:23:05.934702 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:05.934461    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8d786ed-c24f-4c5d-83cf-73120381e0c4" containerName="console"
Sep 09 16:23:05.934702 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:05.934472    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8d786ed-c24f-4c5d-83cf-73120381e0c4" containerName="console"
Sep 09 16:23:05.934702 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:05.934518    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8d786ed-c24f-4c5d-83cf-73120381e0c4" containerName="console"
Sep 09 16:23:05.937748 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:05.937729    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj"
Sep 09 16:23:05.940554 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:05.940533    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 16:23:05.940678 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:05.940658    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 16:23:05.941790 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:05.941771    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ptw5b\""
Sep 09 16:23:05.945406 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:05.945383    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj"]
Sep 09 16:23:06.024227 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:06.024187    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8wh2\" (UniqueName: \"kubernetes.io/projected/c072e0f9-988c-4abf-b108-8e057b284c85-kube-api-access-h8wh2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj\" (UID: \"c072e0f9-988c-4abf-b108-8e057b284c85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj"
Sep 09 16:23:06.024433 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:06.024250    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c072e0f9-988c-4abf-b108-8e057b284c85-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj\" (UID: \"c072e0f9-988c-4abf-b108-8e057b284c85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj"
Sep 09 16:23:06.024433 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:06.024342    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c072e0f9-988c-4abf-b108-8e057b284c85-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj\" (UID: \"c072e0f9-988c-4abf-b108-8e057b284c85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj"
Sep 09 16:23:06.125670 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:06.125628    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c072e0f9-988c-4abf-b108-8e057b284c85-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj\" (UID: \"c072e0f9-988c-4abf-b108-8e057b284c85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj"
Sep 09 16:23:06.125859 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:06.125692    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c072e0f9-988c-4abf-b108-8e057b284c85-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj\" (UID: \"c072e0f9-988c-4abf-b108-8e057b284c85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj"
Sep 09 16:23:06.125859 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:06.125743    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8wh2\" (UniqueName: \"kubernetes.io/projected/c072e0f9-988c-4abf-b108-8e057b284c85-kube-api-access-h8wh2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj\" (UID: \"c072e0f9-988c-4abf-b108-8e057b284c85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj"
Sep 09 16:23:06.126034 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:06.126015    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c072e0f9-988c-4abf-b108-8e057b284c85-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj\" (UID: \"c072e0f9-988c-4abf-b108-8e057b284c85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj"
Sep 09 16:23:06.126096 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:06.126043    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c072e0f9-988c-4abf-b108-8e057b284c85-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj\" (UID: \"c072e0f9-988c-4abf-b108-8e057b284c85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj"
Sep 09 16:23:06.133761 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:06.133735    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8wh2\" (UniqueName: \"kubernetes.io/projected/c072e0f9-988c-4abf-b108-8e057b284c85-kube-api-access-h8wh2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj\" (UID: \"c072e0f9-988c-4abf-b108-8e057b284c85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj"
Sep 09 16:23:06.248033 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:06.247995    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj"
Sep 09 16:23:06.516222 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:06.516143    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj"]
Sep 09 16:23:06.519257 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:23:06.519226    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc072e0f9_988c_4abf_b108_8e057b284c85.slice/crio-1412eb185b880b432fdd966f57e0facc002c926b9649d89fe079ec40989b9cad WatchSource:0}: Error finding container 1412eb185b880b432fdd966f57e0facc002c926b9649d89fe079ec40989b9cad: Status 404 returned error can't find the container with id 1412eb185b880b432fdd966f57e0facc002c926b9649d89fe079ec40989b9cad
Sep 09 16:23:06.586141 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:06.586106    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj" event={"ID":"c072e0f9-988c-4abf-b108-8e057b284c85","Type":"ContainerStarted","Data":"1412eb185b880b432fdd966f57e0facc002c926b9649d89fe079ec40989b9cad"}
Sep 09 16:23:13.614774 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:13.614729    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj" event={"ID":"c072e0f9-988c-4abf-b108-8e057b284c85","Type":"ContainerStarted","Data":"7fae9dff5d28b2e1449142f2788f95dc767db68ef3f7f73ecafa864f8da0b79f"}
Sep 09 16:23:14.618912 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:14.618871    2555 generic.go:356] "Generic (PLEG): container finished" podID="c072e0f9-988c-4abf-b108-8e057b284c85" containerID="7fae9dff5d28b2e1449142f2788f95dc767db68ef3f7f73ecafa864f8da0b79f" exitCode=0
Sep 09 16:23:14.619346 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:14.618939    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj" event={"ID":"c072e0f9-988c-4abf-b108-8e057b284c85","Type":"ContainerDied","Data":"7fae9dff5d28b2e1449142f2788f95dc767db68ef3f7f73ecafa864f8da0b79f"}
Sep 09 16:23:17.632008 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:17.631964    2555 generic.go:356] "Generic (PLEG): container finished" podID="c072e0f9-988c-4abf-b108-8e057b284c85" containerID="c9056f706d320700bb7b2ba491a41d8cdc6b922bb7f10c3a97f2311821d7bcf4" exitCode=0
Sep 09 16:23:17.632439 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:17.632053    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj" event={"ID":"c072e0f9-988c-4abf-b108-8e057b284c85","Type":"ContainerDied","Data":"c9056f706d320700bb7b2ba491a41d8cdc6b922bb7f10c3a97f2311821d7bcf4"}
Sep 09 16:23:24.662151 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:24.662111    2555 generic.go:356] "Generic (PLEG): container finished" podID="c072e0f9-988c-4abf-b108-8e057b284c85" containerID="2e277a1be8a09ed560087509f0713b8ea34fec13641cc21817c292b179ca2091" exitCode=0
Sep 09 16:23:24.662602 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:24.662199    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj" event={"ID":"c072e0f9-988c-4abf-b108-8e057b284c85","Type":"ContainerDied","Data":"2e277a1be8a09ed560087509f0713b8ea34fec13641cc21817c292b179ca2091"}
Sep 09 16:23:25.837822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:25.837786    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj"
Sep 09 16:23:25.999806 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:25.999768    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c072e0f9-988c-4abf-b108-8e057b284c85-bundle\") pod \"c072e0f9-988c-4abf-b108-8e057b284c85\" (UID: \"c072e0f9-988c-4abf-b108-8e057b284c85\") "
Sep 09 16:23:25.999989 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:25.999826    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8wh2\" (UniqueName: \"kubernetes.io/projected/c072e0f9-988c-4abf-b108-8e057b284c85-kube-api-access-h8wh2\") pod \"c072e0f9-988c-4abf-b108-8e057b284c85\" (UID: \"c072e0f9-988c-4abf-b108-8e057b284c85\") "
Sep 09 16:23:25.999989 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:25.999858    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c072e0f9-988c-4abf-b108-8e057b284c85-util\") pod \"c072e0f9-988c-4abf-b108-8e057b284c85\" (UID: \"c072e0f9-988c-4abf-b108-8e057b284c85\") "
Sep 09 16:23:26.000491 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:26.000456    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c072e0f9-988c-4abf-b108-8e057b284c85-bundle" (OuterVolumeSpecName: "bundle") pod "c072e0f9-988c-4abf-b108-8e057b284c85" (UID: "c072e0f9-988c-4abf-b108-8e057b284c85"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:23:26.002006 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:26.001982    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c072e0f9-988c-4abf-b108-8e057b284c85-kube-api-access-h8wh2" (OuterVolumeSpecName: "kube-api-access-h8wh2") pod "c072e0f9-988c-4abf-b108-8e057b284c85" (UID: "c072e0f9-988c-4abf-b108-8e057b284c85"). InnerVolumeSpecName "kube-api-access-h8wh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:23:26.005659 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:26.005631    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c072e0f9-988c-4abf-b108-8e057b284c85-util" (OuterVolumeSpecName: "util") pod "c072e0f9-988c-4abf-b108-8e057b284c85" (UID: "c072e0f9-988c-4abf-b108-8e057b284c85"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:23:26.101106 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:26.101056    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c072e0f9-988c-4abf-b108-8e057b284c85-bundle\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:23:26.101106 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:26.101096    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8wh2\" (UniqueName: \"kubernetes.io/projected/c072e0f9-988c-4abf-b108-8e057b284c85-kube-api-access-h8wh2\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:23:26.101106 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:26.101109    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c072e0f9-988c-4abf-b108-8e057b284c85-util\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:23:26.671046 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:26.671009    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj" event={"ID":"c072e0f9-988c-4abf-b108-8e057b284c85","Type":"ContainerDied","Data":"1412eb185b880b432fdd966f57e0facc002c926b9649d89fe079ec40989b9cad"}
Sep 09 16:23:26.671046 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:26.671041    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409dmcj"
Sep 09 16:23:26.671246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:26.671060    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1412eb185b880b432fdd966f57e0facc002c926b9649d89fe079ec40989b9cad"
Sep 09 16:23:33.873093 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:33.873056    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p79mf"]
Sep 09 16:23:33.873509 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:33.873345    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c072e0f9-988c-4abf-b108-8e057b284c85" containerName="util"
Sep 09 16:23:33.873509 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:33.873358    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c072e0f9-988c-4abf-b108-8e057b284c85" containerName="util"
Sep 09 16:23:33.873509 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:33.873366    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c072e0f9-988c-4abf-b108-8e057b284c85" containerName="pull"
Sep 09 16:23:33.873509 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:33.873371    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c072e0f9-988c-4abf-b108-8e057b284c85" containerName="pull"
Sep 09 16:23:33.873509 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:33.873377    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c072e0f9-988c-4abf-b108-8e057b284c85" containerName="extract"
Sep 09 16:23:33.873509 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:33.873383    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c072e0f9-988c-4abf-b108-8e057b284c85" containerName="extract"
Sep 09 16:23:33.873509 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:33.873424    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c072e0f9-988c-4abf-b108-8e057b284c85" containerName="extract"
Sep 09 16:23:33.875331 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:33.875313    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p79mf"
Sep 09 16:23:33.877961 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:33.877935    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 09 16:23:33.878100 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:33.877985    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-9jf5q\""
Sep 09 16:23:33.878100 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:33.877988    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 09 16:23:33.888618 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:33.888589    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p79mf"]
Sep 09 16:23:33.959352 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:33.959305    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d9dbd6d-1944-44ca-9804-985bde2937f9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-p79mf\" (UID: \"7d9dbd6d-1944-44ca-9804-985bde2937f9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p79mf"
Sep 09 16:23:33.959528 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:33.959365    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr92q\" (UniqueName: \"kubernetes.io/projected/7d9dbd6d-1944-44ca-9804-985bde2937f9-kube-api-access-kr92q\") pod \"cert-manager-operator-controller-manager-55c444bf86-p79mf\" (UID: \"7d9dbd6d-1944-44ca-9804-985bde2937f9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p79mf"
Sep 09 16:23:34.059793 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:34.059755    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr92q\" (UniqueName: \"kubernetes.io/projected/7d9dbd6d-1944-44ca-9804-985bde2937f9-kube-api-access-kr92q\") pod \"cert-manager-operator-controller-manager-55c444bf86-p79mf\" (UID: \"7d9dbd6d-1944-44ca-9804-985bde2937f9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p79mf"
Sep 09 16:23:34.059903 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:34.059893    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d9dbd6d-1944-44ca-9804-985bde2937f9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-p79mf\" (UID: \"7d9dbd6d-1944-44ca-9804-985bde2937f9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p79mf"
Sep 09 16:23:34.060225 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:34.060207    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d9dbd6d-1944-44ca-9804-985bde2937f9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-p79mf\" (UID: \"7d9dbd6d-1944-44ca-9804-985bde2937f9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p79mf"
Sep 09 16:23:34.068114 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:34.068075    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr92q\" (UniqueName: \"kubernetes.io/projected/7d9dbd6d-1944-44ca-9804-985bde2937f9-kube-api-access-kr92q\") pod \"cert-manager-operator-controller-manager-55c444bf86-p79mf\" (UID: \"7d9dbd6d-1944-44ca-9804-985bde2937f9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p79mf"
Sep 09 16:23:34.183994 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:34.183900    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p79mf"
Sep 09 16:23:34.460193 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:34.460113    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p79mf"]
Sep 09 16:23:34.464072 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:23:34.464043    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d9dbd6d_1944_44ca_9804_985bde2937f9.slice/crio-cfccdd558bc12b9692757976cab1524d22b31bd881a4dd08e4d867f470030246 WatchSource:0}: Error finding container cfccdd558bc12b9692757976cab1524d22b31bd881a4dd08e4d867f470030246: Status 404 returned error can't find the container with id cfccdd558bc12b9692757976cab1524d22b31bd881a4dd08e4d867f470030246
Sep 09 16:23:34.701055 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:34.701016    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p79mf" event={"ID":"7d9dbd6d-1944-44ca-9804-985bde2937f9","Type":"ContainerStarted","Data":"cfccdd558bc12b9692757976cab1524d22b31bd881a4dd08e4d867f470030246"}
Sep 09 16:23:36.711607 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:36.711563    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p79mf" event={"ID":"7d9dbd6d-1944-44ca-9804-985bde2937f9","Type":"ContainerStarted","Data":"f77135e936b5aee48c73438a4ab3297506b4e81ffb214b0d3b8141873248e44e"}
Sep 09 16:23:36.734208 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:36.734146    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p79mf" podStartSLOduration=1.631417664 podStartE2EDuration="3.73412438s" totalImagesPullingTime="2.102706716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:23:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:23:36.731346926 +0000 UTC m=+582.561554985" watchObservedRunningTime="2026-09-09 16:23:36.73412438 +0000 UTC m=+582.564332504"
Sep 09 16:23:37.611489 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:37.611453    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2"]
Sep 09 16:23:37.613704 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:37.613684    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2"
Sep 09 16:23:37.616370 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:37.616342    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 16:23:37.616511 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:37.616369    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 16:23:37.617397 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:37.617379    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ptw5b\""
Sep 09 16:23:37.623498 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:37.623473    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2"]
Sep 09 16:23:37.794013 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:37.793975    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/123f3915-09ea-4e45-9def-c7c47d076897-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2\" (UID: \"123f3915-09ea-4e45-9def-c7c47d076897\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2"
Sep 09 16:23:37.794448 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:37.794030    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcdvc\" (UniqueName: \"kubernetes.io/projected/123f3915-09ea-4e45-9def-c7c47d076897-kube-api-access-pcdvc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2\" (UID: \"123f3915-09ea-4e45-9def-c7c47d076897\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2"
Sep 09 16:23:37.794448 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:37.794109    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/123f3915-09ea-4e45-9def-c7c47d076897-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2\" (UID: \"123f3915-09ea-4e45-9def-c7c47d076897\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2"
Sep 09 16:23:37.895407 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:37.895281    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/123f3915-09ea-4e45-9def-c7c47d076897-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2\" (UID: \"123f3915-09ea-4e45-9def-c7c47d076897\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2"
Sep 09 16:23:37.895407 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:37.895399    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcdvc\" (UniqueName: \"kubernetes.io/projected/123f3915-09ea-4e45-9def-c7c47d076897-kube-api-access-pcdvc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2\" (UID: \"123f3915-09ea-4e45-9def-c7c47d076897\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2"
Sep 09 16:23:37.895622 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:37.895467    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/123f3915-09ea-4e45-9def-c7c47d076897-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2\" (UID: \"123f3915-09ea-4e45-9def-c7c47d076897\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2"
Sep 09 16:23:37.895701 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:37.895678    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/123f3915-09ea-4e45-9def-c7c47d076897-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2\" (UID: \"123f3915-09ea-4e45-9def-c7c47d076897\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2"
Sep 09 16:23:37.895805 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:37.895785    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/123f3915-09ea-4e45-9def-c7c47d076897-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2\" (UID: \"123f3915-09ea-4e45-9def-c7c47d076897\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2"
Sep 09 16:23:37.904850 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:37.904826    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcdvc\" (UniqueName: \"kubernetes.io/projected/123f3915-09ea-4e45-9def-c7c47d076897-kube-api-access-pcdvc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2\" (UID: \"123f3915-09ea-4e45-9def-c7c47d076897\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2"
Sep 09 16:23:37.922420 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:37.922387    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2"
Sep 09 16:23:38.218414 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:38.218323    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2"]
Sep 09 16:23:38.221354 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:23:38.221323    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123f3915_09ea_4e45_9def_c7c47d076897.slice/crio-654f69e48012924e75ac728b09858f130268417ad3003da1104553c4cba9fa86 WatchSource:0}: Error finding container 654f69e48012924e75ac728b09858f130268417ad3003da1104553c4cba9fa86: Status 404 returned error can't find the container with id 654f69e48012924e75ac728b09858f130268417ad3003da1104553c4cba9fa86
Sep 09 16:23:38.722457 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:38.722425    2555 generic.go:356] "Generic (PLEG): container finished" podID="123f3915-09ea-4e45-9def-c7c47d076897" containerID="14013f733410357494284790e042ebb02ec23d3717a22d2326964cc7bd9e0a29" exitCode=0
Sep 09 16:23:38.722638 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:38.722464    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2" event={"ID":"123f3915-09ea-4e45-9def-c7c47d076897","Type":"ContainerDied","Data":"14013f733410357494284790e042ebb02ec23d3717a22d2326964cc7bd9e0a29"}
Sep 09 16:23:38.722638 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:38.722502    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2" event={"ID":"123f3915-09ea-4e45-9def-c7c47d076897","Type":"ContainerStarted","Data":"654f69e48012924e75ac728b09858f130268417ad3003da1104553c4cba9fa86"}
Sep 09 16:23:40.859177 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:40.859135    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xzf8x"]
Sep 09 16:23:40.866829 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:40.866802    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xzf8x"
Sep 09 16:23:40.870331 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:40.870080    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 09 16:23:40.870331 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:40.870133    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 09 16:23:40.871657 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:40.871394    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-58924\""
Sep 09 16:23:40.871961 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:40.871935    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xzf8x"]
Sep 09 16:23:41.023909 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:41.023874    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab125eee-419c-456d-ba87-be564a51d38d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xzf8x\" (UID: \"ab125eee-419c-456d-ba87-be564a51d38d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xzf8x"
Sep 09 16:23:41.024069 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:41.023947    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrkrf\" (UniqueName: \"kubernetes.io/projected/ab125eee-419c-456d-ba87-be564a51d38d-kube-api-access-jrkrf\") pod \"cert-manager-cainjector-559cdd8c87-xzf8x\" (UID: \"ab125eee-419c-456d-ba87-be564a51d38d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xzf8x"
Sep 09 16:23:41.125176 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:41.125075    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab125eee-419c-456d-ba87-be564a51d38d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xzf8x\" (UID: \"ab125eee-419c-456d-ba87-be564a51d38d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xzf8x"
Sep 09 16:23:41.125176 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:41.125172    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrkrf\" (UniqueName: \"kubernetes.io/projected/ab125eee-419c-456d-ba87-be564a51d38d-kube-api-access-jrkrf\") pod \"cert-manager-cainjector-559cdd8c87-xzf8x\" (UID: \"ab125eee-419c-456d-ba87-be564a51d38d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xzf8x"
Sep 09 16:23:41.133403 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:41.133369    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrkrf\" (UniqueName: \"kubernetes.io/projected/ab125eee-419c-456d-ba87-be564a51d38d-kube-api-access-jrkrf\") pod \"cert-manager-cainjector-559cdd8c87-xzf8x\" (UID: \"ab125eee-419c-456d-ba87-be564a51d38d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xzf8x"
Sep 09 16:23:41.133533 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:41.133429    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab125eee-419c-456d-ba87-be564a51d38d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xzf8x\" (UID: \"ab125eee-419c-456d-ba87-be564a51d38d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xzf8x"
Sep 09 16:23:41.179195 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:41.179164    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xzf8x"
Sep 09 16:23:41.447244 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:41.447167    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xzf8x"]
Sep 09 16:23:41.450339 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:23:41.450287    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab125eee_419c_456d_ba87_be564a51d38d.slice/crio-9021c39958c36911f57950fe403a988fb7a36ec0997cc8fc9b904aa1718ec5cc WatchSource:0}: Error finding container 9021c39958c36911f57950fe403a988fb7a36ec0997cc8fc9b904aa1718ec5cc: Status 404 returned error can't find the container with id 9021c39958c36911f57950fe403a988fb7a36ec0997cc8fc9b904aa1718ec5cc
Sep 09 16:23:41.736667 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:41.736566    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xzf8x" event={"ID":"ab125eee-419c-456d-ba87-be564a51d38d","Type":"ContainerStarted","Data":"9021c39958c36911f57950fe403a988fb7a36ec0997cc8fc9b904aa1718ec5cc"}
Sep 09 16:23:41.738426 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:41.738395    2555 generic.go:356] "Generic (PLEG): container finished" podID="123f3915-09ea-4e45-9def-c7c47d076897" containerID="c334d50e010c281979683c175fd2babde2882226e1a98c84188389ffd7bd54f2" exitCode=0
Sep 09 16:23:41.738566 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:41.738440    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2" event={"ID":"123f3915-09ea-4e45-9def-c7c47d076897","Type":"ContainerDied","Data":"c334d50e010c281979683c175fd2babde2882226e1a98c84188389ffd7bd54f2"}
Sep 09 16:23:42.745512 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:42.745473    2555 generic.go:356] "Generic (PLEG): container finished" podID="123f3915-09ea-4e45-9def-c7c47d076897" containerID="5110dc9f3677370ddd7e660ac2419ee9c8f694ade1c1fc8696323b08a0e867f9" exitCode=0
Sep 09 16:23:42.745938 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:42.745556    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2" event={"ID":"123f3915-09ea-4e45-9def-c7c47d076897","Type":"ContainerDied","Data":"5110dc9f3677370ddd7e660ac2419ee9c8f694ade1c1fc8696323b08a0e867f9"}
Sep 09 16:23:44.238652 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:44.238629    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2"
Sep 09 16:23:44.248848 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:44.248826    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/123f3915-09ea-4e45-9def-c7c47d076897-bundle\") pod \"123f3915-09ea-4e45-9def-c7c47d076897\" (UID: \"123f3915-09ea-4e45-9def-c7c47d076897\") "
Sep 09 16:23:44.248953 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:44.248856    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/123f3915-09ea-4e45-9def-c7c47d076897-util\") pod \"123f3915-09ea-4e45-9def-c7c47d076897\" (UID: \"123f3915-09ea-4e45-9def-c7c47d076897\") "
Sep 09 16:23:44.248953 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:44.248875    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcdvc\" (UniqueName: \"kubernetes.io/projected/123f3915-09ea-4e45-9def-c7c47d076897-kube-api-access-pcdvc\") pod \"123f3915-09ea-4e45-9def-c7c47d076897\" (UID: \"123f3915-09ea-4e45-9def-c7c47d076897\") "
Sep 09 16:23:44.249152 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:44.249132    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123f3915-09ea-4e45-9def-c7c47d076897-bundle" (OuterVolumeSpecName: "bundle") pod "123f3915-09ea-4e45-9def-c7c47d076897" (UID: "123f3915-09ea-4e45-9def-c7c47d076897"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:23:44.250962 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:44.250943    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123f3915-09ea-4e45-9def-c7c47d076897-kube-api-access-pcdvc" (OuterVolumeSpecName: "kube-api-access-pcdvc") pod "123f3915-09ea-4e45-9def-c7c47d076897" (UID: "123f3915-09ea-4e45-9def-c7c47d076897"). InnerVolumeSpecName "kube-api-access-pcdvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:23:44.253487 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:44.253454    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123f3915-09ea-4e45-9def-c7c47d076897-util" (OuterVolumeSpecName: "util") pod "123f3915-09ea-4e45-9def-c7c47d076897" (UID: "123f3915-09ea-4e45-9def-c7c47d076897"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:23:44.350374 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:44.350344    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/123f3915-09ea-4e45-9def-c7c47d076897-bundle\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:23:44.350374 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:44.350375    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/123f3915-09ea-4e45-9def-c7c47d076897-util\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:23:44.350523 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:44.350385    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcdvc\" (UniqueName: \"kubernetes.io/projected/123f3915-09ea-4e45-9def-c7c47d076897-kube-api-access-pcdvc\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:23:44.758075 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:44.758042    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2"
Sep 09 16:23:44.758075 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:44.758056    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkxrl2" event={"ID":"123f3915-09ea-4e45-9def-c7c47d076897","Type":"ContainerDied","Data":"654f69e48012924e75ac728b09858f130268417ad3003da1104553c4cba9fa86"}
Sep 09 16:23:44.758432 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:44.758102    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="654f69e48012924e75ac728b09858f130268417ad3003da1104553c4cba9fa86"
Sep 09 16:23:44.759778 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:44.759753    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xzf8x" event={"ID":"ab125eee-419c-456d-ba87-be564a51d38d","Type":"ContainerStarted","Data":"dc0319190efb8a7d6c777c35b55baf8499a3f9b678d1f989cacdb793a8f6487c"}
Sep 09 16:23:44.774930 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:44.774879    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xzf8x" podStartSLOduration=1.93244543 podStartE2EDuration="4.774863824s" totalImagesPullingTime="2.842418394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:23:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:23:44.774117612 +0000 UTC m=+590.604325697" watchObservedRunningTime="2026-09-09 16:23:44.774863824 +0000 UTC m=+590.605071883"
Sep 09 16:23:48.051790 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:48.051750    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6xnxp"]
Sep 09 16:23:48.052180 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:48.051999    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="123f3915-09ea-4e45-9def-c7c47d076897" containerName="util"
Sep 09 16:23:48.052180 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:48.052010    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="123f3915-09ea-4e45-9def-c7c47d076897" containerName="util"
Sep 09 16:23:48.052180 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:48.052021    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="123f3915-09ea-4e45-9def-c7c47d076897" containerName="extract"
Sep 09 16:23:48.052180 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:48.052026    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="123f3915-09ea-4e45-9def-c7c47d076897" containerName="extract"
Sep 09 16:23:48.052180 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:48.052036    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="123f3915-09ea-4e45-9def-c7c47d076897" containerName="pull"
Sep 09 16:23:48.052180 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:48.052042    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="123f3915-09ea-4e45-9def-c7c47d076897" containerName="pull"
Sep 09 16:23:48.052180 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:48.052091    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="123f3915-09ea-4e45-9def-c7c47d076897" containerName="extract"
Sep 09 16:23:48.054947 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:48.054927    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-6xnxp"
Sep 09 16:23:48.057921 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:48.057898    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-mxw5p\""
Sep 09 16:23:48.062474 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:48.062450    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6xnxp"]
Sep 09 16:23:48.079059 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:48.079025    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afbce63c-7438-458d-a45f-8fb49564349e-bound-sa-token\") pod \"cert-manager-55b4d954d5-6xnxp\" (UID: \"afbce63c-7438-458d-a45f-8fb49564349e\") " pod="cert-manager/cert-manager-55b4d954d5-6xnxp"
Sep 09 16:23:48.079233 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:48.079088    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j92q2\" (UniqueName: \"kubernetes.io/projected/afbce63c-7438-458d-a45f-8fb49564349e-kube-api-access-j92q2\") pod \"cert-manager-55b4d954d5-6xnxp\" (UID: \"afbce63c-7438-458d-a45f-8fb49564349e\") " pod="cert-manager/cert-manager-55b4d954d5-6xnxp"
Sep 09 16:23:48.180449 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:48.180410    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afbce63c-7438-458d-a45f-8fb49564349e-bound-sa-token\") pod \"cert-manager-55b4d954d5-6xnxp\" (UID: \"afbce63c-7438-458d-a45f-8fb49564349e\") " pod="cert-manager/cert-manager-55b4d954d5-6xnxp"
Sep 09 16:23:48.180619 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:48.180476    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j92q2\" (UniqueName: \"kubernetes.io/projected/afbce63c-7438-458d-a45f-8fb49564349e-kube-api-access-j92q2\") pod \"cert-manager-55b4d954d5-6xnxp\" (UID: \"afbce63c-7438-458d-a45f-8fb49564349e\") " pod="cert-manager/cert-manager-55b4d954d5-6xnxp"
Sep 09 16:23:48.188882 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:48.188853    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/afbce63c-7438-458d-a45f-8fb49564349e-bound-sa-token\") pod \"cert-manager-55b4d954d5-6xnxp\" (UID: \"afbce63c-7438-458d-a45f-8fb49564349e\") " pod="cert-manager/cert-manager-55b4d954d5-6xnxp"
Sep 09 16:23:48.189009 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:48.188988    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j92q2\" (UniqueName: \"kubernetes.io/projected/afbce63c-7438-458d-a45f-8fb49564349e-kube-api-access-j92q2\") pod \"cert-manager-55b4d954d5-6xnxp\" (UID: \"afbce63c-7438-458d-a45f-8fb49564349e\") " pod="cert-manager/cert-manager-55b4d954d5-6xnxp"
Sep 09 16:23:48.364166 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:48.364068    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-6xnxp"
Sep 09 16:23:48.632740 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:48.632636    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6xnxp"]
Sep 09 16:23:48.636394 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:23:48.636346    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafbce63c_7438_458d_a45f_8fb49564349e.slice/crio-59bfe001009afb17b0ca73ee5fc160dc5ca2386b0a5c5aadc3039c6fa5ea413e WatchSource:0}: Error finding container 59bfe001009afb17b0ca73ee5fc160dc5ca2386b0a5c5aadc3039c6fa5ea413e: Status 404 returned error can't find the container with id 59bfe001009afb17b0ca73ee5fc160dc5ca2386b0a5c5aadc3039c6fa5ea413e
Sep 09 16:23:48.777850 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:48.777815    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-6xnxp" event={"ID":"afbce63c-7438-458d-a45f-8fb49564349e","Type":"ContainerStarted","Data":"802aed804932d8f11e0bd88b70c5fbee1ed0346e515abb99a9c8c6d6874c6ace"}
Sep 09 16:23:48.777850 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:48.777856    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-6xnxp" event={"ID":"afbce63c-7438-458d-a45f-8fb49564349e","Type":"ContainerStarted","Data":"59bfe001009afb17b0ca73ee5fc160dc5ca2386b0a5c5aadc3039c6fa5ea413e"}
Sep 09 16:23:48.798100 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:48.798043    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-6xnxp" podStartSLOduration=0.798028415 podStartE2EDuration="798.028415ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:23:48.797005423 +0000 UTC m=+594.627213482" watchObservedRunningTime="2026-09-09 16:23:48.798028415 +0000 UTC m=+594.628236474"
Sep 09 16:23:50.588682 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:50.588645    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x62wl"]
Sep 09 16:23:50.634714 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:50.634679    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x62wl"]
Sep 09 16:23:50.634921 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:50.634831    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x62wl"
Sep 09 16:23:50.638106 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:50.638079    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 16:23:50.638250 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:50.638135    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 16:23:50.639363 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:50.639337    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-z7z42\""
Sep 09 16:23:50.702492 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:50.702445    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmd7m\" (UniqueName: \"kubernetes.io/projected/d28b0a55-6832-41d7-a55e-a92b3d921803-kube-api-access-bmd7m\") pod \"openshift-lws-operator-fd8ccff4c-x62wl\" (UID: \"d28b0a55-6832-41d7-a55e-a92b3d921803\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x62wl"
Sep 09 16:23:50.702674 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:50.702501    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d28b0a55-6832-41d7-a55e-a92b3d921803-tmp\") pod \"openshift-lws-operator-fd8ccff4c-x62wl\" (UID: \"d28b0a55-6832-41d7-a55e-a92b3d921803\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x62wl"
Sep 09 16:23:50.803949 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:50.803895    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmd7m\" (UniqueName: \"kubernetes.io/projected/d28b0a55-6832-41d7-a55e-a92b3d921803-kube-api-access-bmd7m\") pod \"openshift-lws-operator-fd8ccff4c-x62wl\" (UID: \"d28b0a55-6832-41d7-a55e-a92b3d921803\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x62wl"
Sep 09 16:23:50.803949 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:50.803954    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d28b0a55-6832-41d7-a55e-a92b3d921803-tmp\") pod \"openshift-lws-operator-fd8ccff4c-x62wl\" (UID: \"d28b0a55-6832-41d7-a55e-a92b3d921803\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x62wl"
Sep 09 16:23:50.804457 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:50.804437    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d28b0a55-6832-41d7-a55e-a92b3d921803-tmp\") pod \"openshift-lws-operator-fd8ccff4c-x62wl\" (UID: \"d28b0a55-6832-41d7-a55e-a92b3d921803\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x62wl"
Sep 09 16:23:50.812386 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:50.812354    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmd7m\" (UniqueName: \"kubernetes.io/projected/d28b0a55-6832-41d7-a55e-a92b3d921803-kube-api-access-bmd7m\") pod \"openshift-lws-operator-fd8ccff4c-x62wl\" (UID: \"d28b0a55-6832-41d7-a55e-a92b3d921803\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x62wl"
Sep 09 16:23:50.944435 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:50.944333    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x62wl"
Sep 09 16:23:51.234407 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.234343    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x62wl"]
Sep 09 16:23:51.237132 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:23:51.237104    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd28b0a55_6832_41d7_a55e_a92b3d921803.slice/crio-33ea09a256ddbb762473432c65301a525dd73e0066cd7e9ab358a6d13a8be223 WatchSource:0}: Error finding container 33ea09a256ddbb762473432c65301a525dd73e0066cd7e9ab358a6d13a8be223: Status 404 returned error can't find the container with id 33ea09a256ddbb762473432c65301a525dd73e0066cd7e9ab358a6d13a8be223
Sep 09 16:23:51.536226 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.536182    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b6fdc5d75-lrmst"]
Sep 09 16:23:51.558764 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.558725    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b6fdc5d75-lrmst"]
Sep 09 16:23:51.558933 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.558874    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:23:51.611409 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.611370    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzvdq\" (UniqueName: \"kubernetes.io/projected/784b5b32-6777-4bf9-83af-73c3c5f9f69a-kube-api-access-jzvdq\") pod \"console-5b6fdc5d75-lrmst\" (UID: \"784b5b32-6777-4bf9-83af-73c3c5f9f69a\") " pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:23:51.611409 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.611414    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/784b5b32-6777-4bf9-83af-73c3c5f9f69a-console-serving-cert\") pod \"console-5b6fdc5d75-lrmst\" (UID: \"784b5b32-6777-4bf9-83af-73c3c5f9f69a\") " pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:23:51.611895 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.611432    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/784b5b32-6777-4bf9-83af-73c3c5f9f69a-oauth-serving-cert\") pod \"console-5b6fdc5d75-lrmst\" (UID: \"784b5b32-6777-4bf9-83af-73c3c5f9f69a\") " pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:23:51.611895 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.611482    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/784b5b32-6777-4bf9-83af-73c3c5f9f69a-service-ca\") pod \"console-5b6fdc5d75-lrmst\" (UID: \"784b5b32-6777-4bf9-83af-73c3c5f9f69a\") " pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:23:51.611895 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.611552    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/784b5b32-6777-4bf9-83af-73c3c5f9f69a-trusted-ca-bundle\") pod \"console-5b6fdc5d75-lrmst\" (UID: \"784b5b32-6777-4bf9-83af-73c3c5f9f69a\") " pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:23:51.611895 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.611597    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/784b5b32-6777-4bf9-83af-73c3c5f9f69a-console-oauth-config\") pod \"console-5b6fdc5d75-lrmst\" (UID: \"784b5b32-6777-4bf9-83af-73c3c5f9f69a\") " pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:23:51.611895 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.611688    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/784b5b32-6777-4bf9-83af-73c3c5f9f69a-console-config\") pod \"console-5b6fdc5d75-lrmst\" (UID: \"784b5b32-6777-4bf9-83af-73c3c5f9f69a\") " pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:23:51.712779 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.712728    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzvdq\" (UniqueName: \"kubernetes.io/projected/784b5b32-6777-4bf9-83af-73c3c5f9f69a-kube-api-access-jzvdq\") pod \"console-5b6fdc5d75-lrmst\" (UID: \"784b5b32-6777-4bf9-83af-73c3c5f9f69a\") " pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:23:51.712779 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.712776    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/784b5b32-6777-4bf9-83af-73c3c5f9f69a-console-serving-cert\") pod \"console-5b6fdc5d75-lrmst\" (UID: \"784b5b32-6777-4bf9-83af-73c3c5f9f69a\") " pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:23:51.713063 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.712798    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/784b5b32-6777-4bf9-83af-73c3c5f9f69a-oauth-serving-cert\") pod \"console-5b6fdc5d75-lrmst\" (UID: \"784b5b32-6777-4bf9-83af-73c3c5f9f69a\") " pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:23:51.713063 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.712827    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/784b5b32-6777-4bf9-83af-73c3c5f9f69a-service-ca\") pod \"console-5b6fdc5d75-lrmst\" (UID: \"784b5b32-6777-4bf9-83af-73c3c5f9f69a\") " pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:23:51.713063 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.712865    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/784b5b32-6777-4bf9-83af-73c3c5f9f69a-trusted-ca-bundle\") pod \"console-5b6fdc5d75-lrmst\" (UID: \"784b5b32-6777-4bf9-83af-73c3c5f9f69a\") " pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:23:51.713063 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.712932    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/784b5b32-6777-4bf9-83af-73c3c5f9f69a-console-oauth-config\") pod \"console-5b6fdc5d75-lrmst\" (UID: \"784b5b32-6777-4bf9-83af-73c3c5f9f69a\") " pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:23:51.713063 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.712966    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/784b5b32-6777-4bf9-83af-73c3c5f9f69a-console-config\") pod \"console-5b6fdc5d75-lrmst\" (UID: \"784b5b32-6777-4bf9-83af-73c3c5f9f69a\") " pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:23:51.713758 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.713723    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/784b5b32-6777-4bf9-83af-73c3c5f9f69a-service-ca\") pod \"console-5b6fdc5d75-lrmst\" (UID: \"784b5b32-6777-4bf9-83af-73c3c5f9f69a\") " pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:23:51.713758 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.713747    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/784b5b32-6777-4bf9-83af-73c3c5f9f69a-console-config\") pod \"console-5b6fdc5d75-lrmst\" (UID: \"784b5b32-6777-4bf9-83af-73c3c5f9f69a\") " pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:23:51.713906 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.713723    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/784b5b32-6777-4bf9-83af-73c3c5f9f69a-oauth-serving-cert\") pod \"console-5b6fdc5d75-lrmst\" (UID: \"784b5b32-6777-4bf9-83af-73c3c5f9f69a\") " pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:23:51.714327 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.714283    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/784b5b32-6777-4bf9-83af-73c3c5f9f69a-trusted-ca-bundle\") pod \"console-5b6fdc5d75-lrmst\" (UID: \"784b5b32-6777-4bf9-83af-73c3c5f9f69a\") " pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:23:51.715373 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.715352    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/784b5b32-6777-4bf9-83af-73c3c5f9f69a-console-oauth-config\") pod \"console-5b6fdc5d75-lrmst\" (UID: \"784b5b32-6777-4bf9-83af-73c3c5f9f69a\") " pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:23:51.715640 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.715621    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/784b5b32-6777-4bf9-83af-73c3c5f9f69a-console-serving-cert\") pod \"console-5b6fdc5d75-lrmst\" (UID: \"784b5b32-6777-4bf9-83af-73c3c5f9f69a\") " pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:23:51.720529 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.720506    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzvdq\" (UniqueName: \"kubernetes.io/projected/784b5b32-6777-4bf9-83af-73c3c5f9f69a-kube-api-access-jzvdq\") pod \"console-5b6fdc5d75-lrmst\" (UID: \"784b5b32-6777-4bf9-83af-73c3c5f9f69a\") " pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:23:51.790588 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.790493    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x62wl" event={"ID":"d28b0a55-6832-41d7-a55e-a92b3d921803","Type":"ContainerStarted","Data":"33ea09a256ddbb762473432c65301a525dd73e0066cd7e9ab358a6d13a8be223"}
Sep 09 16:23:51.868428 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:51.868392    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:23:52.176473 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:52.176378    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b6fdc5d75-lrmst"]
Sep 09 16:23:52.180473 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:23:52.180444    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784b5b32_6777_4bf9_83af_73c3c5f9f69a.slice/crio-fe26bef3bc65d88659495fb8564fe19c0f46d676ab9e34b801d3e3f390268416 WatchSource:0}: Error finding container fe26bef3bc65d88659495fb8564fe19c0f46d676ab9e34b801d3e3f390268416: Status 404 returned error can't find the container with id fe26bef3bc65d88659495fb8564fe19c0f46d676ab9e34b801d3e3f390268416
Sep 09 16:23:52.797573 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:52.797527    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b6fdc5d75-lrmst" event={"ID":"784b5b32-6777-4bf9-83af-73c3c5f9f69a","Type":"ContainerStarted","Data":"3c8e563b0fdbd09fe3c6a9b5e0b34c3d82c89aaae30e9b8144722ae01483bc94"}
Sep 09 16:23:52.798029 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:52.797595    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b6fdc5d75-lrmst" event={"ID":"784b5b32-6777-4bf9-83af-73c3c5f9f69a","Type":"ContainerStarted","Data":"fe26bef3bc65d88659495fb8564fe19c0f46d676ab9e34b801d3e3f390268416"}
Sep 09 16:23:52.815278 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:52.815224    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b6fdc5d75-lrmst" podStartSLOduration=1.8152064129999999 podStartE2EDuration="1.815206413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:23:52.814362193 +0000 UTC m=+598.644570272" watchObservedRunningTime="2026-09-09 16:23:52.815206413 +0000 UTC m=+598.645414473"
Sep 09 16:23:53.803897 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:53.803796    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x62wl" event={"ID":"d28b0a55-6832-41d7-a55e-a92b3d921803","Type":"ContainerStarted","Data":"d490dbd0825a521ee7dc5bd9bc6ce9ebbfb06411468d33828919bf9da73927a1"}
Sep 09 16:23:53.819832 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:53.819770    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x62wl" podStartSLOduration=1.5476275130000001 podStartE2EDuration="3.819753357s" totalImagesPullingTime="2.272125844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:23:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:23:53.818956094 +0000 UTC m=+599.649164155" watchObservedRunningTime="2026-09-09 16:23:53.819753357 +0000 UTC m=+599.649961416"
Sep 09 16:23:54.546722 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:54.546687    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgcjl_561d9dd7-04c3-42dc-bcf6-2c92fd61706b/console-operator/1.log"
Sep 09 16:23:54.546964 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:54.546945    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgcjl_561d9dd7-04c3-42dc-bcf6-2c92fd61706b/console-operator/1.log"
Sep 09 16:23:58.364022 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:58.363984    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx"]
Sep 09 16:23:58.365977 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:58.365948    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx"
Sep 09 16:23:58.368818 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:58.368791    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ptw5b\""
Sep 09 16:23:58.368958 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:58.368855    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 16:23:58.368958 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:58.368855    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 16:23:58.378457 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:58.378428    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx"]
Sep 09 16:23:58.471748 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:58.471702    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkkn7\" (UniqueName: \"kubernetes.io/projected/ea60e937-8010-453c-9414-bb8a6fb7fbb1-kube-api-access-hkkn7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx\" (UID: \"ea60e937-8010-453c-9414-bb8a6fb7fbb1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx"
Sep 09 16:23:58.471748 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:58.471757    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea60e937-8010-453c-9414-bb8a6fb7fbb1-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx\" (UID: \"ea60e937-8010-453c-9414-bb8a6fb7fbb1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx"
Sep 09 16:23:58.472008 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:58.471875    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea60e937-8010-453c-9414-bb8a6fb7fbb1-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx\" (UID: \"ea60e937-8010-453c-9414-bb8a6fb7fbb1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx"
Sep 09 16:23:58.577534 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:58.577414    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkkn7\" (UniqueName: \"kubernetes.io/projected/ea60e937-8010-453c-9414-bb8a6fb7fbb1-kube-api-access-hkkn7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx\" (UID: \"ea60e937-8010-453c-9414-bb8a6fb7fbb1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx"
Sep 09 16:23:58.577931 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:58.577747    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea60e937-8010-453c-9414-bb8a6fb7fbb1-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx\" (UID: \"ea60e937-8010-453c-9414-bb8a6fb7fbb1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx"
Sep 09 16:23:58.578053 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:58.578035    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea60e937-8010-453c-9414-bb8a6fb7fbb1-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx\" (UID: \"ea60e937-8010-453c-9414-bb8a6fb7fbb1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx"
Sep 09 16:23:58.578344 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:58.578283    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea60e937-8010-453c-9414-bb8a6fb7fbb1-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx\" (UID: \"ea60e937-8010-453c-9414-bb8a6fb7fbb1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx"
Sep 09 16:23:58.578344 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:58.578337    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea60e937-8010-453c-9414-bb8a6fb7fbb1-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx\" (UID: \"ea60e937-8010-453c-9414-bb8a6fb7fbb1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx"
Sep 09 16:23:58.590487 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:58.590448    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkkn7\" (UniqueName: \"kubernetes.io/projected/ea60e937-8010-453c-9414-bb8a6fb7fbb1-kube-api-access-hkkn7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx\" (UID: \"ea60e937-8010-453c-9414-bb8a6fb7fbb1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx"
Sep 09 16:23:58.675393 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:58.675264    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx"
Sep 09 16:23:58.952093 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:58.952031    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx"]
Sep 09 16:23:58.954869 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:23:58.954835    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea60e937_8010_453c_9414_bb8a6fb7fbb1.slice/crio-623125e8061131f50403eb5bc08051800184403e3555e98c68dc93ba9c00a825 WatchSource:0}: Error finding container 623125e8061131f50403eb5bc08051800184403e3555e98c68dc93ba9c00a825: Status 404 returned error can't find the container with id 623125e8061131f50403eb5bc08051800184403e3555e98c68dc93ba9c00a825
Sep 09 16:23:59.830897 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:59.830858    2555 generic.go:356] "Generic (PLEG): container finished" podID="ea60e937-8010-453c-9414-bb8a6fb7fbb1" containerID="6135c5727d8cba697775dfbfd36bf27ffecd813a44caeae3337694b8e4cdf5e7" exitCode=0
Sep 09 16:23:59.831330 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:59.830941    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx" event={"ID":"ea60e937-8010-453c-9414-bb8a6fb7fbb1","Type":"ContainerDied","Data":"6135c5727d8cba697775dfbfd36bf27ffecd813a44caeae3337694b8e4cdf5e7"}
Sep 09 16:23:59.831330 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:59.830988    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx" event={"ID":"ea60e937-8010-453c-9414-bb8a6fb7fbb1","Type":"ContainerStarted","Data":"623125e8061131f50403eb5bc08051800184403e3555e98c68dc93ba9c00a825"}
Sep 09 16:23:59.831463 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:23:59.831448    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 16:24:00.838086 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:00.838048    2555 generic.go:356] "Generic (PLEG): container finished" podID="ea60e937-8010-453c-9414-bb8a6fb7fbb1" containerID="0e2504dc5e8bd90dc1da23f7bed60ace27ed3b6812e34ca82594613f29688d37" exitCode=0
Sep 09 16:24:00.838488 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:00.838111    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx" event={"ID":"ea60e937-8010-453c-9414-bb8a6fb7fbb1","Type":"ContainerDied","Data":"0e2504dc5e8bd90dc1da23f7bed60ace27ed3b6812e34ca82594613f29688d37"}
Sep 09 16:24:01.845134 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:01.845091    2555 generic.go:356] "Generic (PLEG): container finished" podID="ea60e937-8010-453c-9414-bb8a6fb7fbb1" containerID="9acf42c452a638ea56ba3ec69e71035efcaaf165ae3e6c3a069e487a4cd9ea47" exitCode=0
Sep 09 16:24:01.845555 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:01.845167    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx" event={"ID":"ea60e937-8010-453c-9414-bb8a6fb7fbb1","Type":"ContainerDied","Data":"9acf42c452a638ea56ba3ec69e71035efcaaf165ae3e6c3a069e487a4cd9ea47"}
Sep 09 16:24:01.868714 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:01.868682    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:24:01.868901 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:01.868744    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:24:01.875398 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:01.875360    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:24:02.853484 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:02.853455    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b6fdc5d75-lrmst"
Sep 09 16:24:02.898067 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:02.898025    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cd7c6fcc8-64bpg"]
Sep 09 16:24:02.992629 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:02.992596    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx"
Sep 09 16:24:03.118722 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:03.118629    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkkn7\" (UniqueName: \"kubernetes.io/projected/ea60e937-8010-453c-9414-bb8a6fb7fbb1-kube-api-access-hkkn7\") pod \"ea60e937-8010-453c-9414-bb8a6fb7fbb1\" (UID: \"ea60e937-8010-453c-9414-bb8a6fb7fbb1\") "
Sep 09 16:24:03.118722 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:03.118696    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea60e937-8010-453c-9414-bb8a6fb7fbb1-bundle\") pod \"ea60e937-8010-453c-9414-bb8a6fb7fbb1\" (UID: \"ea60e937-8010-453c-9414-bb8a6fb7fbb1\") "
Sep 09 16:24:03.118722 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:03.118720    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea60e937-8010-453c-9414-bb8a6fb7fbb1-util\") pod \"ea60e937-8010-453c-9414-bb8a6fb7fbb1\" (UID: \"ea60e937-8010-453c-9414-bb8a6fb7fbb1\") "
Sep 09 16:24:03.119536 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:03.119507    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea60e937-8010-453c-9414-bb8a6fb7fbb1-bundle" (OuterVolumeSpecName: "bundle") pod "ea60e937-8010-453c-9414-bb8a6fb7fbb1" (UID: "ea60e937-8010-453c-9414-bb8a6fb7fbb1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:24:03.120830 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:03.120809    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea60e937-8010-453c-9414-bb8a6fb7fbb1-kube-api-access-hkkn7" (OuterVolumeSpecName: "kube-api-access-hkkn7") pod "ea60e937-8010-453c-9414-bb8a6fb7fbb1" (UID: "ea60e937-8010-453c-9414-bb8a6fb7fbb1"). InnerVolumeSpecName "kube-api-access-hkkn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:24:03.126654 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:03.126614    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea60e937-8010-453c-9414-bb8a6fb7fbb1-util" (OuterVolumeSpecName: "util") pod "ea60e937-8010-453c-9414-bb8a6fb7fbb1" (UID: "ea60e937-8010-453c-9414-bb8a6fb7fbb1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:24:03.219441 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:03.219397    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea60e937-8010-453c-9414-bb8a6fb7fbb1-util\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:24:03.219441 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:03.219435    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkkn7\" (UniqueName: \"kubernetes.io/projected/ea60e937-8010-453c-9414-bb8a6fb7fbb1-kube-api-access-hkkn7\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:24:03.219441 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:03.219446    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea60e937-8010-453c-9414-bb8a6fb7fbb1-bundle\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:24:03.855976 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:03.855933    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx" event={"ID":"ea60e937-8010-453c-9414-bb8a6fb7fbb1","Type":"ContainerDied","Data":"623125e8061131f50403eb5bc08051800184403e3555e98c68dc93ba9c00a825"}
Sep 09 16:24:03.855976 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:03.855976    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwf7dx"
Sep 09 16:24:03.856442 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:03.855990    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="623125e8061131f50403eb5bc08051800184403e3555e98c68dc93ba9c00a825"
Sep 09 16:24:07.769999 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:07.769960    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r"]
Sep 09 16:24:07.770429 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:07.770228    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea60e937-8010-453c-9414-bb8a6fb7fbb1" containerName="pull"
Sep 09 16:24:07.770429 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:07.770240    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea60e937-8010-453c-9414-bb8a6fb7fbb1" containerName="pull"
Sep 09 16:24:07.770429 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:07.770260    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea60e937-8010-453c-9414-bb8a6fb7fbb1" containerName="util"
Sep 09 16:24:07.770429 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:07.770265    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea60e937-8010-453c-9414-bb8a6fb7fbb1" containerName="util"
Sep 09 16:24:07.770429 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:07.770272    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea60e937-8010-453c-9414-bb8a6fb7fbb1" containerName="extract"
Sep 09 16:24:07.770429 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:07.770277    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea60e937-8010-453c-9414-bb8a6fb7fbb1" containerName="extract"
Sep 09 16:24:07.770429 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:07.770350    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea60e937-8010-453c-9414-bb8a6fb7fbb1" containerName="extract"
Sep 09 16:24:07.772629 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:07.772611    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r"
Sep 09 16:24:07.775246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:07.775222    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 16:24:07.776526 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:07.776502    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 16:24:07.776640 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:07.776490    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ptw5b\""
Sep 09 16:24:07.781556 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:07.781522    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r"]
Sep 09 16:24:07.860857 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:07.860815    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d66ceea7-486c-4f66-b01f-b081d7a8ad83-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r\" (UID: \"d66ceea7-486c-4f66-b01f-b081d7a8ad83\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r"
Sep 09 16:24:07.861046 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:07.860905    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jllhx\" (UniqueName: \"kubernetes.io/projected/d66ceea7-486c-4f66-b01f-b081d7a8ad83-kube-api-access-jllhx\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r\" (UID: \"d66ceea7-486c-4f66-b01f-b081d7a8ad83\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r"
Sep 09 16:24:07.861046 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:07.860930    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d66ceea7-486c-4f66-b01f-b081d7a8ad83-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r\" (UID: \"d66ceea7-486c-4f66-b01f-b081d7a8ad83\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r"
Sep 09 16:24:07.961521 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:07.961473    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d66ceea7-486c-4f66-b01f-b081d7a8ad83-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r\" (UID: \"d66ceea7-486c-4f66-b01f-b081d7a8ad83\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r"
Sep 09 16:24:07.961725 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:07.961588    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jllhx\" (UniqueName: \"kubernetes.io/projected/d66ceea7-486c-4f66-b01f-b081d7a8ad83-kube-api-access-jllhx\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r\" (UID: \"d66ceea7-486c-4f66-b01f-b081d7a8ad83\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r"
Sep 09 16:24:07.961725 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:07.961625    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d66ceea7-486c-4f66-b01f-b081d7a8ad83-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r\" (UID: \"d66ceea7-486c-4f66-b01f-b081d7a8ad83\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r"
Sep 09 16:24:07.961962 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:07.961936    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d66ceea7-486c-4f66-b01f-b081d7a8ad83-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r\" (UID: \"d66ceea7-486c-4f66-b01f-b081d7a8ad83\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r"
Sep 09 16:24:07.962028 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:07.961949    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d66ceea7-486c-4f66-b01f-b081d7a8ad83-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r\" (UID: \"d66ceea7-486c-4f66-b01f-b081d7a8ad83\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r"
Sep 09 16:24:07.969863 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:07.969824    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jllhx\" (UniqueName: \"kubernetes.io/projected/d66ceea7-486c-4f66-b01f-b081d7a8ad83-kube-api-access-jllhx\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r\" (UID: \"d66ceea7-486c-4f66-b01f-b081d7a8ad83\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r"
Sep 09 16:24:08.081741 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:08.081632    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r"
Sep 09 16:24:08.352840 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:08.352732    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r"]
Sep 09 16:24:08.355537 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:24:08.355493    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66ceea7_486c_4f66_b01f_b081d7a8ad83.slice/crio-0cc40e2724a4411cb6fbe0872d868189edcdd48f7e195cc36a0993874756626a WatchSource:0}: Error finding container 0cc40e2724a4411cb6fbe0872d868189edcdd48f7e195cc36a0993874756626a: Status 404 returned error can't find the container with id 0cc40e2724a4411cb6fbe0872d868189edcdd48f7e195cc36a0993874756626a
Sep 09 16:24:08.885263 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:08.885223    2555 generic.go:356] "Generic (PLEG): container finished" podID="d66ceea7-486c-4f66-b01f-b081d7a8ad83" containerID="76cde543bee161b620f8c76741bd77bb2353559facb8ee52790de1ed61b04c59" exitCode=0
Sep 09 16:24:08.885669 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:08.885279    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r" event={"ID":"d66ceea7-486c-4f66-b01f-b081d7a8ad83","Type":"ContainerDied","Data":"76cde543bee161b620f8c76741bd77bb2353559facb8ee52790de1ed61b04c59"}
Sep 09 16:24:08.885669 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:08.885352    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r" event={"ID":"d66ceea7-486c-4f66-b01f-b081d7a8ad83","Type":"ContainerStarted","Data":"0cc40e2724a4411cb6fbe0872d868189edcdd48f7e195cc36a0993874756626a"}
Sep 09 16:24:09.891754 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:09.891664    2555 generic.go:356] "Generic (PLEG): container finished" podID="d66ceea7-486c-4f66-b01f-b081d7a8ad83" containerID="defcb6e7716f89c03b77d8eb61acf2b8e3fb7a33eb4cfd0cdcd1c07105d2e43a" exitCode=0
Sep 09 16:24:09.892180 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:09.891747    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r" event={"ID":"d66ceea7-486c-4f66-b01f-b081d7a8ad83","Type":"ContainerDied","Data":"defcb6e7716f89c03b77d8eb61acf2b8e3fb7a33eb4cfd0cdcd1c07105d2e43a"}
Sep 09 16:24:10.303669 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.303628    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds"]
Sep 09 16:24:10.305865 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.305844    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds"
Sep 09 16:24:10.308657 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.308631    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 16:24:10.308916 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.308889    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 16:24:10.309044 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.308891    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-q77nk\""
Sep 09 16:24:10.309044 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.308988    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 16:24:10.309044 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.309027    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 16:24:10.328903 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.328865    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds"]
Sep 09 16:24:10.382840 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.382797    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/39e57a13-2360-4e1f-b466-f0c6749db833-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b9b77fc68-4qwds\" (UID: \"39e57a13-2360-4e1f-b466-f0c6749db833\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds"
Sep 09 16:24:10.382840 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.382842    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39e57a13-2360-4e1f-b466-f0c6749db833-tmp\") pod \"opendatahub-operator-controller-manager-b9b77fc68-4qwds\" (UID: \"39e57a13-2360-4e1f-b466-f0c6749db833\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds"
Sep 09 16:24:10.383048 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.382864    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/39e57a13-2360-4e1f-b466-f0c6749db833-manifests\") pod \"opendatahub-operator-controller-manager-b9b77fc68-4qwds\" (UID: \"39e57a13-2360-4e1f-b466-f0c6749db833\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds"
Sep 09 16:24:10.383048 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.382885    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39e57a13-2360-4e1f-b466-f0c6749db833-webhook-cert\") pod \"opendatahub-operator-controller-manager-b9b77fc68-4qwds\" (UID: \"39e57a13-2360-4e1f-b466-f0c6749db833\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds"
Sep 09 16:24:10.383048 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.382920    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p48q2\" (UniqueName: \"kubernetes.io/projected/39e57a13-2360-4e1f-b466-f0c6749db833-kube-api-access-p48q2\") pod \"opendatahub-operator-controller-manager-b9b77fc68-4qwds\" (UID: \"39e57a13-2360-4e1f-b466-f0c6749db833\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds"
Sep 09 16:24:10.484090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.484043    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/39e57a13-2360-4e1f-b466-f0c6749db833-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b9b77fc68-4qwds\" (UID: \"39e57a13-2360-4e1f-b466-f0c6749db833\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds"
Sep 09 16:24:10.484090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.484096    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39e57a13-2360-4e1f-b466-f0c6749db833-tmp\") pod \"opendatahub-operator-controller-manager-b9b77fc68-4qwds\" (UID: \"39e57a13-2360-4e1f-b466-f0c6749db833\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds"
Sep 09 16:24:10.484408 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.484118    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/39e57a13-2360-4e1f-b466-f0c6749db833-manifests\") pod \"opendatahub-operator-controller-manager-b9b77fc68-4qwds\" (UID: \"39e57a13-2360-4e1f-b466-f0c6749db833\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds"
Sep 09 16:24:10.484408 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.484133    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39e57a13-2360-4e1f-b466-f0c6749db833-webhook-cert\") pod \"opendatahub-operator-controller-manager-b9b77fc68-4qwds\" (UID: \"39e57a13-2360-4e1f-b466-f0c6749db833\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds"
Sep 09 16:24:10.484408 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.484160    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48q2\" (UniqueName: \"kubernetes.io/projected/39e57a13-2360-4e1f-b466-f0c6749db833-kube-api-access-p48q2\") pod \"opendatahub-operator-controller-manager-b9b77fc68-4qwds\" (UID: \"39e57a13-2360-4e1f-b466-f0c6749db833\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds"
Sep 09 16:24:10.484656 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.484631    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/39e57a13-2360-4e1f-b466-f0c6749db833-manifests\") pod \"opendatahub-operator-controller-manager-b9b77fc68-4qwds\" (UID: \"39e57a13-2360-4e1f-b466-f0c6749db833\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds"
Sep 09 16:24:10.486434 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.486400    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39e57a13-2360-4e1f-b466-f0c6749db833-tmp\") pod \"opendatahub-operator-controller-manager-b9b77fc68-4qwds\" (UID: \"39e57a13-2360-4e1f-b466-f0c6749db833\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds"
Sep 09 16:24:10.486688 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.486667    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/39e57a13-2360-4e1f-b466-f0c6749db833-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b9b77fc68-4qwds\" (UID: \"39e57a13-2360-4e1f-b466-f0c6749db833\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds"
Sep 09 16:24:10.486726 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.486687    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39e57a13-2360-4e1f-b466-f0c6749db833-webhook-cert\") pod \"opendatahub-operator-controller-manager-b9b77fc68-4qwds\" (UID: \"39e57a13-2360-4e1f-b466-f0c6749db833\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds"
Sep 09 16:24:10.498003 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.497975    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p48q2\" (UniqueName: \"kubernetes.io/projected/39e57a13-2360-4e1f-b466-f0c6749db833-kube-api-access-p48q2\") pod \"opendatahub-operator-controller-manager-b9b77fc68-4qwds\" (UID: \"39e57a13-2360-4e1f-b466-f0c6749db833\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds"
Sep 09 16:24:10.616905 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.616809    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds"
Sep 09 16:24:10.898271 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.898164    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds"]
Sep 09 16:24:10.904887 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.904850    2555 generic.go:356] "Generic (PLEG): container finished" podID="d66ceea7-486c-4f66-b01f-b081d7a8ad83" containerID="9b70951d17d100900396a459fc7bd9b4c6079c28342a0b3f6119b8ca818628f5" exitCode=0
Sep 09 16:24:10.905028 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:10.904957    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r" event={"ID":"d66ceea7-486c-4f66-b01f-b081d7a8ad83","Type":"ContainerDied","Data":"9b70951d17d100900396a459fc7bd9b4c6079c28342a0b3f6119b8ca818628f5"}
Sep 09 16:24:10.905795 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:24:10.905770    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e57a13_2360_4e1f_b466_f0c6749db833.slice/crio-13db419ecdd662d6f070fbe50cf936270d941ea8e42e6691d4be03539818266e WatchSource:0}: Error finding container 13db419ecdd662d6f070fbe50cf936270d941ea8e42e6691d4be03539818266e: Status 404 returned error can't find the container with id 13db419ecdd662d6f070fbe50cf936270d941ea8e42e6691d4be03539818266e
Sep 09 16:24:11.913722 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:11.913679    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds" event={"ID":"39e57a13-2360-4e1f-b466-f0c6749db833","Type":"ContainerStarted","Data":"13db419ecdd662d6f070fbe50cf936270d941ea8e42e6691d4be03539818266e"}
Sep 09 16:24:12.078363 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:12.078333    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r"
Sep 09 16:24:12.202106 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:12.202012    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jllhx\" (UniqueName: \"kubernetes.io/projected/d66ceea7-486c-4f66-b01f-b081d7a8ad83-kube-api-access-jllhx\") pod \"d66ceea7-486c-4f66-b01f-b081d7a8ad83\" (UID: \"d66ceea7-486c-4f66-b01f-b081d7a8ad83\") "
Sep 09 16:24:12.202276 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:12.202131    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d66ceea7-486c-4f66-b01f-b081d7a8ad83-util\") pod \"d66ceea7-486c-4f66-b01f-b081d7a8ad83\" (UID: \"d66ceea7-486c-4f66-b01f-b081d7a8ad83\") "
Sep 09 16:24:12.202276 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:12.202168    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d66ceea7-486c-4f66-b01f-b081d7a8ad83-bundle\") pod \"d66ceea7-486c-4f66-b01f-b081d7a8ad83\" (UID: \"d66ceea7-486c-4f66-b01f-b081d7a8ad83\") "
Sep 09 16:24:12.203247 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:12.203196    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66ceea7-486c-4f66-b01f-b081d7a8ad83-bundle" (OuterVolumeSpecName: "bundle") pod "d66ceea7-486c-4f66-b01f-b081d7a8ad83" (UID: "d66ceea7-486c-4f66-b01f-b081d7a8ad83"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:24:12.205509 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:12.205474    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d66ceea7-486c-4f66-b01f-b081d7a8ad83-kube-api-access-jllhx" (OuterVolumeSpecName: "kube-api-access-jllhx") pod "d66ceea7-486c-4f66-b01f-b081d7a8ad83" (UID: "d66ceea7-486c-4f66-b01f-b081d7a8ad83"). InnerVolumeSpecName "kube-api-access-jllhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:24:12.211008 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:12.210969    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66ceea7-486c-4f66-b01f-b081d7a8ad83-util" (OuterVolumeSpecName: "util") pod "d66ceea7-486c-4f66-b01f-b081d7a8ad83" (UID: "d66ceea7-486c-4f66-b01f-b081d7a8ad83"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:24:12.302901 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:12.302861    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d66ceea7-486c-4f66-b01f-b081d7a8ad83-util\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:24:12.302901 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:12.302901    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d66ceea7-486c-4f66-b01f-b081d7a8ad83-bundle\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:24:12.303137 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:12.302915    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jllhx\" (UniqueName: \"kubernetes.io/projected/d66ceea7-486c-4f66-b01f-b081d7a8ad83-kube-api-access-jllhx\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:24:12.925182 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:12.925085    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r" event={"ID":"d66ceea7-486c-4f66-b01f-b081d7a8ad83","Type":"ContainerDied","Data":"0cc40e2724a4411cb6fbe0872d868189edcdd48f7e195cc36a0993874756626a"}
Sep 09 16:24:12.925182 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:12.925117    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ll26r"
Sep 09 16:24:12.925182 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:12.925147    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cc40e2724a4411cb6fbe0872d868189edcdd48f7e195cc36a0993874756626a"
Sep 09 16:24:14.935892 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:14.935857    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds" event={"ID":"39e57a13-2360-4e1f-b466-f0c6749db833","Type":"ContainerStarted","Data":"81c5c0f5cd7896424770f30b622a132e0076fb5420f0641394d2b0b68e95eee6"}
Sep 09 16:24:15.941574 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:15.941538    2555 generic.go:356] "Generic (PLEG): container finished" podID="39e57a13-2360-4e1f-b466-f0c6749db833" containerID="81c5c0f5cd7896424770f30b622a132e0076fb5420f0641394d2b0b68e95eee6" exitCode=0
Sep 09 16:24:15.941965 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:15.941632    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds" event={"ID":"39e57a13-2360-4e1f-b466-f0c6749db833","Type":"ContainerDied","Data":"81c5c0f5cd7896424770f30b622a132e0076fb5420f0641394d2b0b68e95eee6"}
Sep 09 16:24:16.949094 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:16.949056    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds" event={"ID":"39e57a13-2360-4e1f-b466-f0c6749db833","Type":"ContainerStarted","Data":"a3b376a3d140b67e87c4cc6a03513bf03c779190285f9ae640ee8dc68314f1ef"}
Sep 09 16:24:16.949615 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:16.949148    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds"
Sep 09 16:24:16.973740 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:16.973658    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds" podStartSLOduration=2.4535344820000002 podStartE2EDuration="6.973635063s" totalImagesPullingTime="3.4762964s" totalInitContainerRuntime="1.043804181s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:24:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:24:16.972133492 +0000 UTC m=+622.802341553" watchObservedRunningTime="2026-09-09 16:24:16.973635063 +0000 UTC m=+622.803843123"
Sep 09 16:24:27.921677 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:27.921627    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cd7c6fcc8-64bpg" podUID="88706fbd-166c-4f5e-8524-d7c1b3bfc2ae" containerName="console" containerID="cri-o://f1fd63515339b1d141a24c658073efde10a6e2cb3731942b67d9b7010349a00f" gracePeriod=15
Sep 09 16:24:27.956154 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:27.956125    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-4qwds"
Sep 09 16:24:28.189544 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:28.189468    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:24:28.347234 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:28.347194    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8dk6\" (UniqueName: \"kubernetes.io/projected/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-kube-api-access-w8dk6\") pod \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") "
Sep 09 16:24:28.347234 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:28.347237    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-console-oauth-config\") pod \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") "
Sep 09 16:24:28.347483 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:28.347273    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-trusted-ca-bundle\") pod \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") "
Sep 09 16:24:28.347483 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:28.347307    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-service-ca\") pod \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") "
Sep 09 16:24:28.347483 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:28.347324    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-oauth-serving-cert\") pod \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") "
Sep 09 16:24:28.347483 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:28.347360    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-console-config\") pod \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") "
Sep 09 16:24:28.347483 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:28.347402    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-console-serving-cert\") pod \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\" (UID: \"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae\") "
Sep 09 16:24:28.347864 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:28.347832    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "88706fbd-166c-4f5e-8524-d7c1b3bfc2ae" (UID: "88706fbd-166c-4f5e-8524-d7c1b3bfc2ae"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:24:28.347997 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:28.347857    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-service-ca" (OuterVolumeSpecName: "service-ca") pod "88706fbd-166c-4f5e-8524-d7c1b3bfc2ae" (UID: "88706fbd-166c-4f5e-8524-d7c1b3bfc2ae"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:24:28.347997 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:28.347867    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "88706fbd-166c-4f5e-8524-d7c1b3bfc2ae" (UID: "88706fbd-166c-4f5e-8524-d7c1b3bfc2ae"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:24:28.347997 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:28.347965    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-console-config" (OuterVolumeSpecName: "console-config") pod "88706fbd-166c-4f5e-8524-d7c1b3bfc2ae" (UID: "88706fbd-166c-4f5e-8524-d7c1b3bfc2ae"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:24:28.349554 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:28.349526    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "88706fbd-166c-4f5e-8524-d7c1b3bfc2ae" (UID: "88706fbd-166c-4f5e-8524-d7c1b3bfc2ae"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:24:28.349925 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:28.349901    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "88706fbd-166c-4f5e-8524-d7c1b3bfc2ae" (UID: "88706fbd-166c-4f5e-8524-d7c1b3bfc2ae"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:24:28.349992 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:28.349905    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-kube-api-access-w8dk6" (OuterVolumeSpecName: "kube-api-access-w8dk6") pod "88706fbd-166c-4f5e-8524-d7c1b3bfc2ae" (UID: "88706fbd-166c-4f5e-8524-d7c1b3bfc2ae"). InnerVolumeSpecName "kube-api-access-w8dk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:24:28.448410 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:28.448288    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-console-config\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:24:28.448410 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:28.448351    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-console-serving-cert\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:24:28.448410 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:28.448363    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8dk6\" (UniqueName: \"kubernetes.io/projected/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-kube-api-access-w8dk6\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:24:28.448410 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:28.448396    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-console-oauth-config\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:24:28.448410 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:28.448405    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-trusted-ca-bundle\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:24:28.448410 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:28.448416    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-service-ca\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:24:28.448735 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:28.448425    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae-oauth-serving-cert\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:24:29.007789 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:29.007748    2555 generic.go:356] "Generic (PLEG): container finished" podID="88706fbd-166c-4f5e-8524-d7c1b3bfc2ae" containerID="f1fd63515339b1d141a24c658073efde10a6e2cb3731942b67d9b7010349a00f" exitCode=0
Sep 09 16:24:29.008194 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:29.007815    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd7c6fcc8-64bpg"
Sep 09 16:24:29.008194 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:29.007832    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd7c6fcc8-64bpg" event={"ID":"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae","Type":"ContainerDied","Data":"f1fd63515339b1d141a24c658073efde10a6e2cb3731942b67d9b7010349a00f"}
Sep 09 16:24:29.008194 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:29.007895    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd7c6fcc8-64bpg" event={"ID":"88706fbd-166c-4f5e-8524-d7c1b3bfc2ae","Type":"ContainerDied","Data":"9e5d0c4104b229e40f1689e4016143ecb5939f002e23a4f761be21dd4c163cdb"}
Sep 09 16:24:29.008194 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:29.007915    2555 scope.go:117] "RemoveContainer" containerID="f1fd63515339b1d141a24c658073efde10a6e2cb3731942b67d9b7010349a00f"
Sep 09 16:24:29.016236 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:29.016218    2555 scope.go:117] "RemoveContainer" containerID="f1fd63515339b1d141a24c658073efde10a6e2cb3731942b67d9b7010349a00f"
Sep 09 16:24:29.016542 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:24:29.016523    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1fd63515339b1d141a24c658073efde10a6e2cb3731942b67d9b7010349a00f\": container with ID starting with f1fd63515339b1d141a24c658073efde10a6e2cb3731942b67d9b7010349a00f not found: ID does not exist" containerID="f1fd63515339b1d141a24c658073efde10a6e2cb3731942b67d9b7010349a00f"
Sep 09 16:24:29.016607 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:29.016549    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1fd63515339b1d141a24c658073efde10a6e2cb3731942b67d9b7010349a00f"} err="failed to get container status \"f1fd63515339b1d141a24c658073efde10a6e2cb3731942b67d9b7010349a00f\": rpc error: code = NotFound desc = could not find container \"f1fd63515339b1d141a24c658073efde10a6e2cb3731942b67d9b7010349a00f\": container with ID starting with f1fd63515339b1d141a24c658073efde10a6e2cb3731942b67d9b7010349a00f not found: ID does not exist"
Sep 09 16:24:29.024234 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:29.024204    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cd7c6fcc8-64bpg"]
Sep 09 16:24:29.027580 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:29.027550    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cd7c6fcc8-64bpg"]
Sep 09 16:24:30.608021 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:30.607983    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88706fbd-166c-4f5e-8524-d7c1b3bfc2ae" path="/var/lib/kubelet/pods/88706fbd-166c-4f5e-8524-d7c1b3bfc2ae/volumes"
Sep 09 16:24:31.595172 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.595128    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-769d969b5b-crxch"]
Sep 09 16:24:31.595459 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.595444    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88706fbd-166c-4f5e-8524-d7c1b3bfc2ae" containerName="console"
Sep 09 16:24:31.595528 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.595461    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="88706fbd-166c-4f5e-8524-d7c1b3bfc2ae" containerName="console"
Sep 09 16:24:31.595528 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.595483    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66ceea7-486c-4f66-b01f-b081d7a8ad83" containerName="extract"
Sep 09 16:24:31.595528 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.595489    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66ceea7-486c-4f66-b01f-b081d7a8ad83" containerName="extract"
Sep 09 16:24:31.595528 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.595502    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66ceea7-486c-4f66-b01f-b081d7a8ad83" containerName="util"
Sep 09 16:24:31.595528 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.595508    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66ceea7-486c-4f66-b01f-b081d7a8ad83" containerName="util"
Sep 09 16:24:31.595528 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.595514    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66ceea7-486c-4f66-b01f-b081d7a8ad83" containerName="pull"
Sep 09 16:24:31.595528 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.595519    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66ceea7-486c-4f66-b01f-b081d7a8ad83" containerName="pull"
Sep 09 16:24:31.595860 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.595569    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d66ceea7-486c-4f66-b01f-b081d7a8ad83" containerName="extract"
Sep 09 16:24:31.595860 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.595579    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="88706fbd-166c-4f5e-8524-d7c1b3bfc2ae" containerName="console"
Sep 09 16:24:31.598265 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.598246    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-769d969b5b-crxch"
Sep 09 16:24:31.601773 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.601735    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 16:24:31.601773 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.601766    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 16:24:31.602005 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.601982    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 16:24:31.603164 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.603145    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 16:24:31.614532 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.614506    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-769d969b5b-crxch"]
Sep 09 16:24:31.672193 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.672157    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d13928a1-4968-4e90-89a9-fee8bb59477e-tmp\") pod \"kube-auth-proxy-769d969b5b-crxch\" (UID: \"d13928a1-4968-4e90-89a9-fee8bb59477e\") " pod="openshift-ingress/kube-auth-proxy-769d969b5b-crxch"
Sep 09 16:24:31.672421 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.672334    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd6ld\" (UniqueName: \"kubernetes.io/projected/d13928a1-4968-4e90-89a9-fee8bb59477e-kube-api-access-nd6ld\") pod \"kube-auth-proxy-769d969b5b-crxch\" (UID: \"d13928a1-4968-4e90-89a9-fee8bb59477e\") " pod="openshift-ingress/kube-auth-proxy-769d969b5b-crxch"
Sep 09 16:24:31.672498 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.672434    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d13928a1-4968-4e90-89a9-fee8bb59477e-tls-certs\") pod \"kube-auth-proxy-769d969b5b-crxch\" (UID: \"d13928a1-4968-4e90-89a9-fee8bb59477e\") " pod="openshift-ingress/kube-auth-proxy-769d969b5b-crxch"
Sep 09 16:24:31.773261 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.773222    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd6ld\" (UniqueName: \"kubernetes.io/projected/d13928a1-4968-4e90-89a9-fee8bb59477e-kube-api-access-nd6ld\") pod \"kube-auth-proxy-769d969b5b-crxch\" (UID: \"d13928a1-4968-4e90-89a9-fee8bb59477e\") " pod="openshift-ingress/kube-auth-proxy-769d969b5b-crxch"
Sep 09 16:24:31.773485 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.773268    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d13928a1-4968-4e90-89a9-fee8bb59477e-tls-certs\") pod \"kube-auth-proxy-769d969b5b-crxch\" (UID: \"d13928a1-4968-4e90-89a9-fee8bb59477e\") " pod="openshift-ingress/kube-auth-proxy-769d969b5b-crxch"
Sep 09 16:24:31.773485 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.773330    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d13928a1-4968-4e90-89a9-fee8bb59477e-tmp\") pod \"kube-auth-proxy-769d969b5b-crxch\" (UID: \"d13928a1-4968-4e90-89a9-fee8bb59477e\") " pod="openshift-ingress/kube-auth-proxy-769d969b5b-crxch"
Sep 09 16:24:31.773485 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:24:31.773469    2555 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 09 16:24:31.773647 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:24:31.773578    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d13928a1-4968-4e90-89a9-fee8bb59477e-tls-certs podName:d13928a1-4968-4e90-89a9-fee8bb59477e nodeName:}" failed. No retries permitted until 2026-09-09 16:24:32.273541676 +0000 UTC m=+638.103749728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d13928a1-4968-4e90-89a9-fee8bb59477e-tls-certs") pod "kube-auth-proxy-769d969b5b-crxch" (UID: "d13928a1-4968-4e90-89a9-fee8bb59477e") : secret "kube-auth-proxy-tls" not found
Sep 09 16:24:31.775703 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.775670    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d13928a1-4968-4e90-89a9-fee8bb59477e-tmp\") pod \"kube-auth-proxy-769d969b5b-crxch\" (UID: \"d13928a1-4968-4e90-89a9-fee8bb59477e\") " pod="openshift-ingress/kube-auth-proxy-769d969b5b-crxch"
Sep 09 16:24:31.782737 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:31.782713    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd6ld\" (UniqueName: \"kubernetes.io/projected/d13928a1-4968-4e90-89a9-fee8bb59477e-kube-api-access-nd6ld\") pod \"kube-auth-proxy-769d969b5b-crxch\" (UID: \"d13928a1-4968-4e90-89a9-fee8bb59477e\") " pod="openshift-ingress/kube-auth-proxy-769d969b5b-crxch"
Sep 09 16:24:32.277848 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.277814    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d13928a1-4968-4e90-89a9-fee8bb59477e-tls-certs\") pod \"kube-auth-proxy-769d969b5b-crxch\" (UID: \"d13928a1-4968-4e90-89a9-fee8bb59477e\") " pod="openshift-ingress/kube-auth-proxy-769d969b5b-crxch"
Sep 09 16:24:32.280570 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.280538    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d13928a1-4968-4e90-89a9-fee8bb59477e-tls-certs\") pod \"kube-auth-proxy-769d969b5b-crxch\" (UID: \"d13928a1-4968-4e90-89a9-fee8bb59477e\") " pod="openshift-ingress/kube-auth-proxy-769d969b5b-crxch"
Sep 09 16:24:32.509205 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.509149    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-769d969b5b-crxch"
Sep 09 16:24:32.780996 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.780852    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-769d969b5b-crxch"]
Sep 09 16:24:32.783529 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:24:32.783491    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13928a1_4968_4e90_89a9_fee8bb59477e.slice/crio-91597c434f808392496ffcec773b45c5dfae063522f853433b2a1940cffeece0 WatchSource:0}: Error finding container 91597c434f808392496ffcec773b45c5dfae063522f853433b2a1940cffeece0: Status 404 returned error can't find the container with id 91597c434f808392496ffcec773b45c5dfae063522f853433b2a1940cffeece0
Sep 09 16:24:32.827142 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.827055    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"]
Sep 09 16:24:32.829934 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.829909    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:32.832699 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.832670    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 09 16:24:32.832926 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.832908    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 09 16:24:32.833016 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.832938    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 16:24:32.833016 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.832966    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qbgwk\""
Sep 09 16:24:32.833016 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.832944    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 09 16:24:32.849758 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.849725    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"]
Sep 09 16:24:32.883667 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.883620    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/560cbd82-926d-496b-8f41-578a5589e6c8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-f6ptp\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:32.883839 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.883680    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/560cbd82-926d-496b-8f41-578a5589e6c8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-f6ptp\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:32.883839 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.883713    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/560cbd82-926d-496b-8f41-578a5589e6c8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-f6ptp\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:32.883839 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.883750    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/560cbd82-926d-496b-8f41-578a5589e6c8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-f6ptp\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:32.883839 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.883782    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/560cbd82-926d-496b-8f41-578a5589e6c8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-f6ptp\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:32.883839 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.883818    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98j6k\" (UniqueName: \"kubernetes.io/projected/560cbd82-926d-496b-8f41-578a5589e6c8-kube-api-access-98j6k\") pod \"istiod-openshift-gateway-7b7995d5cd-f6ptp\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:32.884061 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.883877    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/560cbd82-926d-496b-8f41-578a5589e6c8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-f6ptp\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:32.984477 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.984435    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/560cbd82-926d-496b-8f41-578a5589e6c8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-f6ptp\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:32.984674 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.984497    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/560cbd82-926d-496b-8f41-578a5589e6c8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-f6ptp\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:32.984674 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.984527    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/560cbd82-926d-496b-8f41-578a5589e6c8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-f6ptp\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:32.984674 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.984551    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/560cbd82-926d-496b-8f41-578a5589e6c8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-f6ptp\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:32.984674 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.984580    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/560cbd82-926d-496b-8f41-578a5589e6c8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-f6ptp\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:32.984674 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.984597    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/560cbd82-926d-496b-8f41-578a5589e6c8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-f6ptp\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:32.984674 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.984611    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98j6k\" (UniqueName: \"kubernetes.io/projected/560cbd82-926d-496b-8f41-578a5589e6c8-kube-api-access-98j6k\") pod \"istiod-openshift-gateway-7b7995d5cd-f6ptp\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:32.985537 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.985507    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/560cbd82-926d-496b-8f41-578a5589e6c8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-f6ptp\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:32.987133 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.987103    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/560cbd82-926d-496b-8f41-578a5589e6c8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-f6ptp\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:32.987322 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.987272    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/560cbd82-926d-496b-8f41-578a5589e6c8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-f6ptp\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:32.987406 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.987376    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/560cbd82-926d-496b-8f41-578a5589e6c8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-f6ptp\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:32.987468 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.987406    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/560cbd82-926d-496b-8f41-578a5589e6c8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-f6ptp\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:32.997811 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.997780    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/560cbd82-926d-496b-8f41-578a5589e6c8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-f6ptp\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:32.997934 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:32.997822    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98j6k\" (UniqueName: \"kubernetes.io/projected/560cbd82-926d-496b-8f41-578a5589e6c8-kube-api-access-98j6k\") pod \"istiod-openshift-gateway-7b7995d5cd-f6ptp\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:33.027129 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:33.027089    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-769d969b5b-crxch" event={"ID":"d13928a1-4968-4e90-89a9-fee8bb59477e","Type":"ContainerStarted","Data":"91597c434f808392496ffcec773b45c5dfae063522f853433b2a1940cffeece0"}
Sep 09 16:24:33.140709 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:33.140604    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:33.427388 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:33.427034    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"]
Sep 09 16:24:33.430331 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:24:33.430271    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560cbd82_926d_496b_8f41_578a5589e6c8.slice/crio-6391b51c8e7ac9b69c50c6a1b62c1d09818180ef16467cf37bc0b58b6753ab82 WatchSource:0}: Error finding container 6391b51c8e7ac9b69c50c6a1b62c1d09818180ef16467cf37bc0b58b6753ab82: Status 404 returned error can't find the container with id 6391b51c8e7ac9b69c50c6a1b62c1d09818180ef16467cf37bc0b58b6753ab82
Sep 09 16:24:34.036238 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:34.036195    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp" event={"ID":"560cbd82-926d-496b-8f41-578a5589e6c8","Type":"ContainerStarted","Data":"6391b51c8e7ac9b69c50c6a1b62c1d09818180ef16467cf37bc0b58b6753ab82"}
Sep 09 16:24:36.049130 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:36.049082    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-769d969b5b-crxch" event={"ID":"d13928a1-4968-4e90-89a9-fee8bb59477e","Type":"ContainerStarted","Data":"a7cdad00c397d1d2344f0cf616b7c9b6091a604a7db661518ab05df7d046846a"}
Sep 09 16:24:36.065765 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:36.065709    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-769d969b5b-crxch" podStartSLOduration=2.599875838 podStartE2EDuration="5.065692258s" totalImagesPullingTime="2.46581642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:24:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:24:36.06390384 +0000 UTC m=+641.894111900" watchObservedRunningTime="2026-09-09 16:24:36.065692258 +0000 UTC m=+641.895900316"
Sep 09 16:24:36.990749 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:36.990703    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:24:36.990828 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:36.990812    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:24:37.059966 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:37.059928    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp" event={"ID":"560cbd82-926d-496b-8f41-578a5589e6c8","Type":"ContainerStarted","Data":"59cf613dbbe978d2878f518f4a9f9a258d6dd0de4f6747d37044a52309958cbd"}
Sep 09 16:24:38.065077 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:38.065037    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:38.066744 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:38.066716    2555 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-f6ptp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 16:24:38.066862 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:38.066785    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp" podUID="560cbd82-926d-496b-8f41-578a5589e6c8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 16:24:38.091357 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:38.091277    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp" podStartSLOduration=2.53400581 podStartE2EDuration="6.091258834s" totalImagesPullingTime="3.557253024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:24:38.087095675 +0000 UTC m=+643.917303733" watchObservedRunningTime="2026-09-09 16:24:38.091258834 +0000 UTC m=+643.921466893"
Sep 09 16:24:39.070382 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:39.070347    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:51.758819 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:51.758778    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-567dd"]
Sep 09 16:24:51.762420 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:51.762391    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-567dd"
Sep 09 16:24:51.767538 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:51.767514    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 09 16:24:51.767685 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:51.767515    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-rxhfs\""
Sep 09 16:24:51.774753 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:51.774720    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-596jg"]
Sep 09 16:24:51.777691 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:51.777664    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-567dd"]
Sep 09 16:24:51.777806 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:51.777775    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-596jg"
Sep 09 16:24:51.780398 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:51.780362    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 09 16:24:51.780514 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:51.780484    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-j9lhh\""
Sep 09 16:24:51.800053 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:51.800015    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-596jg"]
Sep 09 16:24:51.832226 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:51.832189    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b92f3d8-9f59-4f7f-8e59-e82b115c2426-cert\") pod \"kserve-controller-manager-d6f4b65b8-567dd\" (UID: \"9b92f3d8-9f59-4f7f-8e59-e82b115c2426\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-567dd"
Sep 09 16:24:51.832413 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:51.832253    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/613f5bc6-e044-40f3-a80f-1d65e6f59d6c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-596jg\" (UID: \"613f5bc6-e044-40f3-a80f-1d65e6f59d6c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-596jg"
Sep 09 16:24:51.832413 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:51.832304    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5frvk\" (UniqueName: \"kubernetes.io/projected/9b92f3d8-9f59-4f7f-8e59-e82b115c2426-kube-api-access-5frvk\") pod \"kserve-controller-manager-d6f4b65b8-567dd\" (UID: \"9b92f3d8-9f59-4f7f-8e59-e82b115c2426\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-567dd"
Sep 09 16:24:51.832413 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:51.832330    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgpw9\" (UniqueName: \"kubernetes.io/projected/613f5bc6-e044-40f3-a80f-1d65e6f59d6c-kube-api-access-dgpw9\") pod \"llmisvc-controller-manager-777c6bd7d5-596jg\" (UID: \"613f5bc6-e044-40f3-a80f-1d65e6f59d6c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-596jg"
Sep 09 16:24:51.933683 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:51.933641    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b92f3d8-9f59-4f7f-8e59-e82b115c2426-cert\") pod \"kserve-controller-manager-d6f4b65b8-567dd\" (UID: \"9b92f3d8-9f59-4f7f-8e59-e82b115c2426\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-567dd"
Sep 09 16:24:51.933888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:51.933701    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/613f5bc6-e044-40f3-a80f-1d65e6f59d6c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-596jg\" (UID: \"613f5bc6-e044-40f3-a80f-1d65e6f59d6c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-596jg"
Sep 09 16:24:51.933888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:51.933731    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5frvk\" (UniqueName: \"kubernetes.io/projected/9b92f3d8-9f59-4f7f-8e59-e82b115c2426-kube-api-access-5frvk\") pod \"kserve-controller-manager-d6f4b65b8-567dd\" (UID: \"9b92f3d8-9f59-4f7f-8e59-e82b115c2426\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-567dd"
Sep 09 16:24:51.933888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:51.933833    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgpw9\" (UniqueName: \"kubernetes.io/projected/613f5bc6-e044-40f3-a80f-1d65e6f59d6c-kube-api-access-dgpw9\") pod \"llmisvc-controller-manager-777c6bd7d5-596jg\" (UID: \"613f5bc6-e044-40f3-a80f-1d65e6f59d6c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-596jg"
Sep 09 16:24:51.936397 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:51.936355    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b92f3d8-9f59-4f7f-8e59-e82b115c2426-cert\") pod \"kserve-controller-manager-d6f4b65b8-567dd\" (UID: \"9b92f3d8-9f59-4f7f-8e59-e82b115c2426\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-567dd"
Sep 09 16:24:51.936515 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:51.936355    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/613f5bc6-e044-40f3-a80f-1d65e6f59d6c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-596jg\" (UID: \"613f5bc6-e044-40f3-a80f-1d65e6f59d6c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-596jg"
Sep 09 16:24:51.950147 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:51.950119    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5frvk\" (UniqueName: \"kubernetes.io/projected/9b92f3d8-9f59-4f7f-8e59-e82b115c2426-kube-api-access-5frvk\") pod \"kserve-controller-manager-d6f4b65b8-567dd\" (UID: \"9b92f3d8-9f59-4f7f-8e59-e82b115c2426\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-567dd"
Sep 09 16:24:51.950269 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:51.950220    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgpw9\" (UniqueName: \"kubernetes.io/projected/613f5bc6-e044-40f3-a80f-1d65e6f59d6c-kube-api-access-dgpw9\") pod \"llmisvc-controller-manager-777c6bd7d5-596jg\" (UID: \"613f5bc6-e044-40f3-a80f-1d65e6f59d6c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-596jg"
Sep 09 16:24:52.074834 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:52.074740    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-567dd"
Sep 09 16:24:52.088337 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:52.088305    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-596jg"
Sep 09 16:24:52.351269 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:52.351237    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-567dd"]
Sep 09 16:24:52.353444 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:24:52.353407    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b92f3d8_9f59_4f7f_8e59_e82b115c2426.slice/crio-f4d9b59ae6969e34acab06282d2bed56b952b04670c4cc196a917a7137920f39 WatchSource:0}: Error finding container f4d9b59ae6969e34acab06282d2bed56b952b04670c4cc196a917a7137920f39: Status 404 returned error can't find the container with id f4d9b59ae6969e34acab06282d2bed56b952b04670c4cc196a917a7137920f39
Sep 09 16:24:52.373487 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:52.373459    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-596jg"]
Sep 09 16:24:52.375464 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:24:52.375430    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod613f5bc6_e044_40f3_a80f_1d65e6f59d6c.slice/crio-0a7590293d61a2b5bfb9c654ebad182012354fbe4a10cd60847582b32480cf97 WatchSource:0}: Error finding container 0a7590293d61a2b5bfb9c654ebad182012354fbe4a10cd60847582b32480cf97: Status 404 returned error can't find the container with id 0a7590293d61a2b5bfb9c654ebad182012354fbe4a10cd60847582b32480cf97
Sep 09 16:24:52.758943 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:52.758903    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"]
Sep 09 16:24:52.759405 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:52.759152    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp" podUID="560cbd82-926d-496b-8f41-578a5589e6c8" containerName="discovery" containerID="cri-o://59cf613dbbe978d2878f518f4a9f9a258d6dd0de4f6747d37044a52309958cbd" gracePeriod=30
Sep 09 16:24:53.010007 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.009928    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:53.043310 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.043265    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/560cbd82-926d-496b-8f41-578a5589e6c8-cacerts\") pod \"560cbd82-926d-496b-8f41-578a5589e6c8\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") "
Sep 09 16:24:53.043495 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.043326    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/560cbd82-926d-496b-8f41-578a5589e6c8-istio-csr-ca-configmap\") pod \"560cbd82-926d-496b-8f41-578a5589e6c8\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") "
Sep 09 16:24:53.043495 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.043358    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/560cbd82-926d-496b-8f41-578a5589e6c8-local-certs\") pod \"560cbd82-926d-496b-8f41-578a5589e6c8\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") "
Sep 09 16:24:53.043495 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.043401    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/560cbd82-926d-496b-8f41-578a5589e6c8-istio-token\") pod \"560cbd82-926d-496b-8f41-578a5589e6c8\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") "
Sep 09 16:24:53.043495 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.043416    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/560cbd82-926d-496b-8f41-578a5589e6c8-istio-kubeconfig\") pod \"560cbd82-926d-496b-8f41-578a5589e6c8\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") "
Sep 09 16:24:53.043495 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.043434    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98j6k\" (UniqueName: \"kubernetes.io/projected/560cbd82-926d-496b-8f41-578a5589e6c8-kube-api-access-98j6k\") pod \"560cbd82-926d-496b-8f41-578a5589e6c8\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") "
Sep 09 16:24:53.043761 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.043506    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/560cbd82-926d-496b-8f41-578a5589e6c8-istio-csr-dns-cert\") pod \"560cbd82-926d-496b-8f41-578a5589e6c8\" (UID: \"560cbd82-926d-496b-8f41-578a5589e6c8\") "
Sep 09 16:24:53.044108 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.044075    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/560cbd82-926d-496b-8f41-578a5589e6c8-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "560cbd82-926d-496b-8f41-578a5589e6c8" (UID: "560cbd82-926d-496b-8f41-578a5589e6c8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:24:53.046410 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.046358    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/560cbd82-926d-496b-8f41-578a5589e6c8-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "560cbd82-926d-496b-8f41-578a5589e6c8" (UID: "560cbd82-926d-496b-8f41-578a5589e6c8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:24:53.046410 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.046381    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/560cbd82-926d-496b-8f41-578a5589e6c8-cacerts" (OuterVolumeSpecName: "cacerts") pod "560cbd82-926d-496b-8f41-578a5589e6c8" (UID: "560cbd82-926d-496b-8f41-578a5589e6c8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:24:53.046410 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.046401    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/560cbd82-926d-496b-8f41-578a5589e6c8-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "560cbd82-926d-496b-8f41-578a5589e6c8" (UID: "560cbd82-926d-496b-8f41-578a5589e6c8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:24:53.046641 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.046490    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/560cbd82-926d-496b-8f41-578a5589e6c8-istio-token" (OuterVolumeSpecName: "istio-token") pod "560cbd82-926d-496b-8f41-578a5589e6c8" (UID: "560cbd82-926d-496b-8f41-578a5589e6c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:24:53.046857 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.046832    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/560cbd82-926d-496b-8f41-578a5589e6c8-kube-api-access-98j6k" (OuterVolumeSpecName: "kube-api-access-98j6k") pod "560cbd82-926d-496b-8f41-578a5589e6c8" (UID: "560cbd82-926d-496b-8f41-578a5589e6c8"). InnerVolumeSpecName "kube-api-access-98j6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:24:53.047000 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.046985    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/560cbd82-926d-496b-8f41-578a5589e6c8-local-certs" (OuterVolumeSpecName: "local-certs") pod "560cbd82-926d-496b-8f41-578a5589e6c8" (UID: "560cbd82-926d-496b-8f41-578a5589e6c8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:24:53.144472 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.144439    2555 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/560cbd82-926d-496b-8f41-578a5589e6c8-istio-csr-dns-cert\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:24:53.144472 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.144471    2555 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/560cbd82-926d-496b-8f41-578a5589e6c8-cacerts\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:24:53.144701 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.144485    2555 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/560cbd82-926d-496b-8f41-578a5589e6c8-istio-csr-ca-configmap\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:24:53.144701 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.144497    2555 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/560cbd82-926d-496b-8f41-578a5589e6c8-local-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:24:53.144701 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.144510    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/560cbd82-926d-496b-8f41-578a5589e6c8-istio-token\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:24:53.144701 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.144522    2555 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/560cbd82-926d-496b-8f41-578a5589e6c8-istio-kubeconfig\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:24:53.144701 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.144535    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98j6k\" (UniqueName: \"kubernetes.io/projected/560cbd82-926d-496b-8f41-578a5589e6c8-kube-api-access-98j6k\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:24:53.145470 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.145408    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-567dd" event={"ID":"9b92f3d8-9f59-4f7f-8e59-e82b115c2426","Type":"ContainerStarted","Data":"f4d9b59ae6969e34acab06282d2bed56b952b04670c4cc196a917a7137920f39"}
Sep 09 16:24:53.147351 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.147284    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-596jg" event={"ID":"613f5bc6-e044-40f3-a80f-1d65e6f59d6c","Type":"ContainerStarted","Data":"0a7590293d61a2b5bfb9c654ebad182012354fbe4a10cd60847582b32480cf97"}
Sep 09 16:24:53.149049 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.149017    2555 generic.go:356] "Generic (PLEG): container finished" podID="560cbd82-926d-496b-8f41-578a5589e6c8" containerID="59cf613dbbe978d2878f518f4a9f9a258d6dd0de4f6747d37044a52309958cbd" exitCode=0
Sep 09 16:24:53.149171 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.149068    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"
Sep 09 16:24:53.149171 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.149083    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp" event={"ID":"560cbd82-926d-496b-8f41-578a5589e6c8","Type":"ContainerDied","Data":"59cf613dbbe978d2878f518f4a9f9a258d6dd0de4f6747d37044a52309958cbd"}
Sep 09 16:24:53.149171 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.149134    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp" event={"ID":"560cbd82-926d-496b-8f41-578a5589e6c8","Type":"ContainerDied","Data":"6391b51c8e7ac9b69c50c6a1b62c1d09818180ef16467cf37bc0b58b6753ab82"}
Sep 09 16:24:53.149171 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.149161    2555 scope.go:117] "RemoveContainer" containerID="59cf613dbbe978d2878f518f4a9f9a258d6dd0de4f6747d37044a52309958cbd"
Sep 09 16:24:53.161140 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.160732    2555 scope.go:117] "RemoveContainer" containerID="59cf613dbbe978d2878f518f4a9f9a258d6dd0de4f6747d37044a52309958cbd"
Sep 09 16:24:53.161140 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:24:53.161129    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59cf613dbbe978d2878f518f4a9f9a258d6dd0de4f6747d37044a52309958cbd\": container with ID starting with 59cf613dbbe978d2878f518f4a9f9a258d6dd0de4f6747d37044a52309958cbd not found: ID does not exist" containerID="59cf613dbbe978d2878f518f4a9f9a258d6dd0de4f6747d37044a52309958cbd"
Sep 09 16:24:53.161285 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.161160    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59cf613dbbe978d2878f518f4a9f9a258d6dd0de4f6747d37044a52309958cbd"} err="failed to get container status \"59cf613dbbe978d2878f518f4a9f9a258d6dd0de4f6747d37044a52309958cbd\": rpc error: code = NotFound desc = could not find container \"59cf613dbbe978d2878f518f4a9f9a258d6dd0de4f6747d37044a52309958cbd\": container with ID starting with 59cf613dbbe978d2878f518f4a9f9a258d6dd0de4f6747d37044a52309958cbd not found: ID does not exist"
Sep 09 16:24:53.185441 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.185407    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"]
Sep 09 16:24:53.198534 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:53.194996    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-f6ptp"]
Sep 09 16:24:54.609079 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:54.609036    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="560cbd82-926d-496b-8f41-578a5589e6c8" path="/var/lib/kubelet/pods/560cbd82-926d-496b-8f41-578a5589e6c8/volumes"
Sep 09 16:24:56.169711 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:56.169668    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-567dd" event={"ID":"9b92f3d8-9f59-4f7f-8e59-e82b115c2426","Type":"ContainerStarted","Data":"39c853fddefcce232568d3cfa6513dc3e4fc3807b0a3251476178725abc6ee5d"}
Sep 09 16:24:56.170118 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:56.169847    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-567dd"
Sep 09 16:24:56.171154 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:56.171132    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-596jg" event={"ID":"613f5bc6-e044-40f3-a80f-1d65e6f59d6c","Type":"ContainerStarted","Data":"7de2719198986edb0eab923a1e10bcbd10166e95de33c6894e1f91c2ef469879"}
Sep 09 16:24:56.171313 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:56.171177    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-596jg"
Sep 09 16:24:56.186502 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:56.186448    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-567dd" podStartSLOduration=2.809331122 podStartE2EDuration="5.186426991s" totalImagesPullingTime="2.377095869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:24:56.184373877 +0000 UTC m=+662.014581913" watchObservedRunningTime="2026-09-09 16:24:56.186426991 +0000 UTC m=+662.016635050"
Sep 09 16:24:56.201070 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:24:56.201015    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-596jg" podStartSLOduration=2.235483282 podStartE2EDuration="5.200996402s" totalImagesPullingTime="2.96551312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:24:56.199272695 +0000 UTC m=+662.029480758" watchObservedRunningTime="2026-09-09 16:24:56.200996402 +0000 UTC m=+662.031204510"
Sep 09 16:25:27.178981 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:27.178939    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-596jg"
Sep 09 16:25:27.182126 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:27.182101    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-567dd"
Sep 09 16:25:47.021554 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.021467    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65"]
Sep 09 16:25:47.022069 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.021785    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="560cbd82-926d-496b-8f41-578a5589e6c8" containerName="discovery"
Sep 09 16:25:47.022069 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.021797    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="560cbd82-926d-496b-8f41-578a5589e6c8" containerName="discovery"
Sep 09 16:25:47.022069 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.021847    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="560cbd82-926d-496b-8f41-578a5589e6c8" containerName="discovery"
Sep 09 16:25:47.026354 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.026333    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65"
Sep 09 16:25:47.029480 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.029455    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ptw5b\""
Sep 09 16:25:47.029816 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.029797    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 16:25:47.030730 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.030710    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 16:25:47.035121 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.035095    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65"]
Sep 09 16:25:47.118487 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.118451    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9"]
Sep 09 16:25:47.121038 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.121016    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9"
Sep 09 16:25:47.127144 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.127118    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kv7q\" (UniqueName: \"kubernetes.io/projected/50b79790-9bed-46ba-8f67-457d26c52bbc-kube-api-access-4kv7q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65\" (UID: \"50b79790-9bed-46ba-8f67-457d26c52bbc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65"
Sep 09 16:25:47.127362 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.127157    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50b79790-9bed-46ba-8f67-457d26c52bbc-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65\" (UID: \"50b79790-9bed-46ba-8f67-457d26c52bbc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65"
Sep 09 16:25:47.127362 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.127233    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50b79790-9bed-46ba-8f67-457d26c52bbc-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65\" (UID: \"50b79790-9bed-46ba-8f67-457d26c52bbc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65"
Sep 09 16:25:47.131984 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.131950    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9"]
Sep 09 16:25:47.219820 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.219775    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg"]
Sep 09 16:25:47.222496 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.222473    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg"
Sep 09 16:25:47.227932 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.227906    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50b79790-9bed-46ba-8f67-457d26c52bbc-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65\" (UID: \"50b79790-9bed-46ba-8f67-457d26c52bbc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65"
Sep 09 16:25:47.228071 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.227969    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8882e519-41bf-4d40-b740-f5d93b85f9bb-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9\" (UID: \"8882e519-41bf-4d40-b740-f5d93b85f9bb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9"
Sep 09 16:25:47.228071 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.227994    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vslcb\" (UniqueName: \"kubernetes.io/projected/8882e519-41bf-4d40-b740-f5d93b85f9bb-kube-api-access-vslcb\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9\" (UID: \"8882e519-41bf-4d40-b740-f5d93b85f9bb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9"
Sep 09 16:25:47.228071 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.228021    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kv7q\" (UniqueName: \"kubernetes.io/projected/50b79790-9bed-46ba-8f67-457d26c52bbc-kube-api-access-4kv7q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65\" (UID: \"50b79790-9bed-46ba-8f67-457d26c52bbc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65"
Sep 09 16:25:47.228071 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.228040    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8882e519-41bf-4d40-b740-f5d93b85f9bb-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9\" (UID: \"8882e519-41bf-4d40-b740-f5d93b85f9bb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9"
Sep 09 16:25:47.228253 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.228080    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50b79790-9bed-46ba-8f67-457d26c52bbc-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65\" (UID: \"50b79790-9bed-46ba-8f67-457d26c52bbc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65"
Sep 09 16:25:47.228360 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.228338    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50b79790-9bed-46ba-8f67-457d26c52bbc-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65\" (UID: \"50b79790-9bed-46ba-8f67-457d26c52bbc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65"
Sep 09 16:25:47.228450 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.228431    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50b79790-9bed-46ba-8f67-457d26c52bbc-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65\" (UID: \"50b79790-9bed-46ba-8f67-457d26c52bbc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65"
Sep 09 16:25:47.232422 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.232393    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg"]
Sep 09 16:25:47.237630 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.237606    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kv7q\" (UniqueName: \"kubernetes.io/projected/50b79790-9bed-46ba-8f67-457d26c52bbc-kube-api-access-4kv7q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65\" (UID: \"50b79790-9bed-46ba-8f67-457d26c52bbc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65"
Sep 09 16:25:47.321467 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.321377    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m"]
Sep 09 16:25:47.323811 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.323793    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m"
Sep 09 16:25:47.329483 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.329451    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8882e519-41bf-4d40-b740-f5d93b85f9bb-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9\" (UID: \"8882e519-41bf-4d40-b740-f5d93b85f9bb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9"
Sep 09 16:25:47.329652 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.329498    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vslcb\" (UniqueName: \"kubernetes.io/projected/8882e519-41bf-4d40-b740-f5d93b85f9bb-kube-api-access-vslcb\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9\" (UID: \"8882e519-41bf-4d40-b740-f5d93b85f9bb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9"
Sep 09 16:25:47.329652 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.329530    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8882e519-41bf-4d40-b740-f5d93b85f9bb-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9\" (UID: \"8882e519-41bf-4d40-b740-f5d93b85f9bb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9"
Sep 09 16:25:47.329652 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.329562    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/657f250d-c4cb-4261-8b68-7de2d582d944-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg\" (UID: \"657f250d-c4cb-4261-8b68-7de2d582d944\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg"
Sep 09 16:25:47.329652 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.329582    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/657f250d-c4cb-4261-8b68-7de2d582d944-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg\" (UID: \"657f250d-c4cb-4261-8b68-7de2d582d944\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg"
Sep 09 16:25:47.329652 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.329614    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4sfw\" (UniqueName: \"kubernetes.io/projected/657f250d-c4cb-4261-8b68-7de2d582d944-kube-api-access-l4sfw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg\" (UID: \"657f250d-c4cb-4261-8b68-7de2d582d944\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg"
Sep 09 16:25:47.329910 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.329875    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8882e519-41bf-4d40-b740-f5d93b85f9bb-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9\" (UID: \"8882e519-41bf-4d40-b740-f5d93b85f9bb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9"
Sep 09 16:25:47.329910 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.329900    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8882e519-41bf-4d40-b740-f5d93b85f9bb-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9\" (UID: \"8882e519-41bf-4d40-b740-f5d93b85f9bb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9"
Sep 09 16:25:47.334547 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.334517    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m"]
Sep 09 16:25:47.336368 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.336345    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65"
Sep 09 16:25:47.338973 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.338952    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vslcb\" (UniqueName: \"kubernetes.io/projected/8882e519-41bf-4d40-b740-f5d93b85f9bb-kube-api-access-vslcb\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9\" (UID: \"8882e519-41bf-4d40-b740-f5d93b85f9bb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9"
Sep 09 16:25:47.430024 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.429977    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9"
Sep 09 16:25:47.430233 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.430209    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/657f250d-c4cb-4261-8b68-7de2d582d944-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg\" (UID: \"657f250d-c4cb-4261-8b68-7de2d582d944\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg"
Sep 09 16:25:47.430306 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.430252    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/657f250d-c4cb-4261-8b68-7de2d582d944-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg\" (UID: \"657f250d-c4cb-4261-8b68-7de2d582d944\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg"
Sep 09 16:25:47.430358 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.430316    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvkxz\" (UniqueName: \"kubernetes.io/projected/17687187-b4e0-499f-b3ca-119306cd76ba-kube-api-access-zvkxz\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m\" (UID: \"17687187-b4e0-499f-b3ca-119306cd76ba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m"
Sep 09 16:25:47.430358 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.430340    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4sfw\" (UniqueName: \"kubernetes.io/projected/657f250d-c4cb-4261-8b68-7de2d582d944-kube-api-access-l4sfw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg\" (UID: \"657f250d-c4cb-4261-8b68-7de2d582d944\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg"
Sep 09 16:25:47.430427 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.430362    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17687187-b4e0-499f-b3ca-119306cd76ba-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m\" (UID: \"17687187-b4e0-499f-b3ca-119306cd76ba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m"
Sep 09 16:25:47.430427 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.430385    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17687187-b4e0-499f-b3ca-119306cd76ba-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m\" (UID: \"17687187-b4e0-499f-b3ca-119306cd76ba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m"
Sep 09 16:25:47.430701 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.430678    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/657f250d-c4cb-4261-8b68-7de2d582d944-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg\" (UID: \"657f250d-c4cb-4261-8b68-7de2d582d944\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg"
Sep 09 16:25:47.430744 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.430718    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/657f250d-c4cb-4261-8b68-7de2d582d944-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg\" (UID: \"657f250d-c4cb-4261-8b68-7de2d582d944\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg"
Sep 09 16:25:47.439633 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.439607    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4sfw\" (UniqueName: \"kubernetes.io/projected/657f250d-c4cb-4261-8b68-7de2d582d944-kube-api-access-l4sfw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg\" (UID: \"657f250d-c4cb-4261-8b68-7de2d582d944\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg"
Sep 09 16:25:47.531880 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.531832    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvkxz\" (UniqueName: \"kubernetes.io/projected/17687187-b4e0-499f-b3ca-119306cd76ba-kube-api-access-zvkxz\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m\" (UID: \"17687187-b4e0-499f-b3ca-119306cd76ba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m"
Sep 09 16:25:47.532094 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.531898    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17687187-b4e0-499f-b3ca-119306cd76ba-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m\" (UID: \"17687187-b4e0-499f-b3ca-119306cd76ba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m"
Sep 09 16:25:47.532094 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.531912    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg"
Sep 09 16:25:47.532094 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.531929    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17687187-b4e0-499f-b3ca-119306cd76ba-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m\" (UID: \"17687187-b4e0-499f-b3ca-119306cd76ba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m"
Sep 09 16:25:47.532337 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.532287    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17687187-b4e0-499f-b3ca-119306cd76ba-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m\" (UID: \"17687187-b4e0-499f-b3ca-119306cd76ba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m"
Sep 09 16:25:47.532411 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.532382    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17687187-b4e0-499f-b3ca-119306cd76ba-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m\" (UID: \"17687187-b4e0-499f-b3ca-119306cd76ba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m"
Sep 09 16:25:47.542137 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.542108    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvkxz\" (UniqueName: \"kubernetes.io/projected/17687187-b4e0-499f-b3ca-119306cd76ba-kube-api-access-zvkxz\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m\" (UID: \"17687187-b4e0-499f-b3ca-119306cd76ba\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m"
Sep 09 16:25:47.614942 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.614912    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65"]
Sep 09 16:25:47.616433 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:25:47.616404    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b79790_9bed_46ba_8f67_457d26c52bbc.slice/crio-49fb8ab1fd9ef4d2609218eb3636ddfdc76eb1870ca6f42f75fdf5e2c34066d7 WatchSource:0}: Error finding container 49fb8ab1fd9ef4d2609218eb3636ddfdc76eb1870ca6f42f75fdf5e2c34066d7: Status 404 returned error can't find the container with id 49fb8ab1fd9ef4d2609218eb3636ddfdc76eb1870ca6f42f75fdf5e2c34066d7
Sep 09 16:25:47.634059 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.634022    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m"
Sep 09 16:25:47.710091 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.709902    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9"]
Sep 09 16:25:47.712219 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:25:47.712176    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8882e519_41bf_4d40_b740_f5d93b85f9bb.slice/crio-bff6d8faa919a816994220684c840d2106185000752a667e0bb840db9b28b0c4 WatchSource:0}: Error finding container bff6d8faa919a816994220684c840d2106185000752a667e0bb840db9b28b0c4: Status 404 returned error can't find the container with id bff6d8faa919a816994220684c840d2106185000752a667e0bb840db9b28b0c4
Sep 09 16:25:47.808952 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.808929    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg"]
Sep 09 16:25:47.816464 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:25:47.816437    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657f250d_c4cb_4261_8b68_7de2d582d944.slice/crio-5da275acb256df1985dde1dfd72e5069ae56b8741048d30224963c640678f52d WatchSource:0}: Error finding container 5da275acb256df1985dde1dfd72e5069ae56b8741048d30224963c640678f52d: Status 404 returned error can't find the container with id 5da275acb256df1985dde1dfd72e5069ae56b8741048d30224963c640678f52d
Sep 09 16:25:47.910970 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:47.910943    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m"]
Sep 09 16:25:47.915250 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:25:47.915218    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17687187_b4e0_499f_b3ca_119306cd76ba.slice/crio-347eddced53e2f41885c2f22b0cce673cacd90337fc014627efbff467a589053 WatchSource:0}: Error finding container 347eddced53e2f41885c2f22b0cce673cacd90337fc014627efbff467a589053: Status 404 returned error can't find the container with id 347eddced53e2f41885c2f22b0cce673cacd90337fc014627efbff467a589053
Sep 09 16:25:48.418440 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:48.418398    2555 generic.go:356] "Generic (PLEG): container finished" podID="8882e519-41bf-4d40-b740-f5d93b85f9bb" containerID="9e6c1d653602e9a514a4a3f0aa4bb03d04a39271798c3073afa8347e396bc480" exitCode=0
Sep 09 16:25:48.418870 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:48.418486    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9" event={"ID":"8882e519-41bf-4d40-b740-f5d93b85f9bb","Type":"ContainerDied","Data":"9e6c1d653602e9a514a4a3f0aa4bb03d04a39271798c3073afa8347e396bc480"}
Sep 09 16:25:48.418870 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:48.418545    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9" event={"ID":"8882e519-41bf-4d40-b740-f5d93b85f9bb","Type":"ContainerStarted","Data":"bff6d8faa919a816994220684c840d2106185000752a667e0bb840db9b28b0c4"}
Sep 09 16:25:48.420211 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:48.420181    2555 generic.go:356] "Generic (PLEG): container finished" podID="17687187-b4e0-499f-b3ca-119306cd76ba" containerID="ff9dba65b237f54483c43ed4432f2949540358f243f07871fbeff936183cf3e2" exitCode=0
Sep 09 16:25:48.420325 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:48.420262    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m" event={"ID":"17687187-b4e0-499f-b3ca-119306cd76ba","Type":"ContainerDied","Data":"ff9dba65b237f54483c43ed4432f2949540358f243f07871fbeff936183cf3e2"}
Sep 09 16:25:48.420394 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:48.420332    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m" event={"ID":"17687187-b4e0-499f-b3ca-119306cd76ba","Type":"ContainerStarted","Data":"347eddced53e2f41885c2f22b0cce673cacd90337fc014627efbff467a589053"}
Sep 09 16:25:48.421951 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:48.421935    2555 generic.go:356] "Generic (PLEG): container finished" podID="657f250d-c4cb-4261-8b68-7de2d582d944" containerID="cbe3f6e3134ea816896aa8164d0c28a4a4515e2bc4425d61d18e7ae296f19c4a" exitCode=0
Sep 09 16:25:48.422041 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:48.422026    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg" event={"ID":"657f250d-c4cb-4261-8b68-7de2d582d944","Type":"ContainerDied","Data":"cbe3f6e3134ea816896aa8164d0c28a4a4515e2bc4425d61d18e7ae296f19c4a"}
Sep 09 16:25:48.422094 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:48.422059    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg" event={"ID":"657f250d-c4cb-4261-8b68-7de2d582d944","Type":"ContainerStarted","Data":"5da275acb256df1985dde1dfd72e5069ae56b8741048d30224963c640678f52d"}
Sep 09 16:25:48.423655 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:48.423636    2555 generic.go:356] "Generic (PLEG): container finished" podID="50b79790-9bed-46ba-8f67-457d26c52bbc" containerID="7e7ece7ea5a28e2997fc247dfc2cd6cf787b1f72049a350db0d26d80a1976ee0" exitCode=0
Sep 09 16:25:48.423736 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:48.423700    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65" event={"ID":"50b79790-9bed-46ba-8f67-457d26c52bbc","Type":"ContainerDied","Data":"7e7ece7ea5a28e2997fc247dfc2cd6cf787b1f72049a350db0d26d80a1976ee0"}
Sep 09 16:25:48.423789 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:48.423737    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65" event={"ID":"50b79790-9bed-46ba-8f67-457d26c52bbc","Type":"ContainerStarted","Data":"49fb8ab1fd9ef4d2609218eb3636ddfdc76eb1870ca6f42f75fdf5e2c34066d7"}
Sep 09 16:25:50.439959 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:50.439923    2555 generic.go:356] "Generic (PLEG): container finished" podID="8882e519-41bf-4d40-b740-f5d93b85f9bb" containerID="092d24c6db40aeddfdf4fe252b06229ef04102cc3a4d5a4758c9b5a1424d4858" exitCode=0
Sep 09 16:25:50.440455 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:50.440001    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9" event={"ID":"8882e519-41bf-4d40-b740-f5d93b85f9bb","Type":"ContainerDied","Data":"092d24c6db40aeddfdf4fe252b06229ef04102cc3a4d5a4758c9b5a1424d4858"}
Sep 09 16:25:50.442086 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:50.442064    2555 generic.go:356] "Generic (PLEG): container finished" podID="17687187-b4e0-499f-b3ca-119306cd76ba" containerID="baf32822e1c9033911a9446f3b4b4160da920ff2e24119653307be470dc69f4e" exitCode=0
Sep 09 16:25:50.442174 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:50.442146    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m" event={"ID":"17687187-b4e0-499f-b3ca-119306cd76ba","Type":"ContainerDied","Data":"baf32822e1c9033911a9446f3b4b4160da920ff2e24119653307be470dc69f4e"}
Sep 09 16:25:50.444286 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:50.444185    2555 generic.go:356] "Generic (PLEG): container finished" podID="657f250d-c4cb-4261-8b68-7de2d582d944" containerID="0ddff6a007b7f896261028eedd692247f5f2c0a4705c40198ae8771f11ef2566" exitCode=0
Sep 09 16:25:50.444286 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:50.444235    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg" event={"ID":"657f250d-c4cb-4261-8b68-7de2d582d944","Type":"ContainerDied","Data":"0ddff6a007b7f896261028eedd692247f5f2c0a4705c40198ae8771f11ef2566"}
Sep 09 16:25:50.446497 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:50.446476    2555 generic.go:356] "Generic (PLEG): container finished" podID="50b79790-9bed-46ba-8f67-457d26c52bbc" containerID="023ffedeeef34e8447c27f174be7f04e0eb51b46462709a244c131523e2de727" exitCode=0
Sep 09 16:25:50.446583 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:50.446564    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65" event={"ID":"50b79790-9bed-46ba-8f67-457d26c52bbc","Type":"ContainerDied","Data":"023ffedeeef34e8447c27f174be7f04e0eb51b46462709a244c131523e2de727"}
Sep 09 16:25:51.454003 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:51.453969    2555 generic.go:356] "Generic (PLEG): container finished" podID="50b79790-9bed-46ba-8f67-457d26c52bbc" containerID="8b90f353ed128429c7c6c79056b500fb1d2c9e35998604acebcb1c7bbb58690a" exitCode=0
Sep 09 16:25:51.454437 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:51.454065    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65" event={"ID":"50b79790-9bed-46ba-8f67-457d26c52bbc","Type":"ContainerDied","Data":"8b90f353ed128429c7c6c79056b500fb1d2c9e35998604acebcb1c7bbb58690a"}
Sep 09 16:25:51.456250 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:51.456224    2555 generic.go:356] "Generic (PLEG): container finished" podID="8882e519-41bf-4d40-b740-f5d93b85f9bb" containerID="e3e6866b3c5e9e3cf93380b379ecc219a90841e27e08067ef90dd9b6efb9a907" exitCode=0
Sep 09 16:25:51.456386 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:51.456319    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9" event={"ID":"8882e519-41bf-4d40-b740-f5d93b85f9bb","Type":"ContainerDied","Data":"e3e6866b3c5e9e3cf93380b379ecc219a90841e27e08067ef90dd9b6efb9a907"}
Sep 09 16:25:51.458490 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:51.458464    2555 generic.go:356] "Generic (PLEG): container finished" podID="17687187-b4e0-499f-b3ca-119306cd76ba" containerID="fdbc27ae0d42f74be6cc50ef2df52b7ed176cc429de1efbb5cfd9869ef875c4c" exitCode=0
Sep 09 16:25:51.458599 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:51.458541    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m" event={"ID":"17687187-b4e0-499f-b3ca-119306cd76ba","Type":"ContainerDied","Data":"fdbc27ae0d42f74be6cc50ef2df52b7ed176cc429de1efbb5cfd9869ef875c4c"}
Sep 09 16:25:51.460589 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:51.460561    2555 generic.go:356] "Generic (PLEG): container finished" podID="657f250d-c4cb-4261-8b68-7de2d582d944" containerID="9cb14165ea24ee3cb0eca8fc292f2dfb82326fb3cc0a2312ec585c8115cdd8b0" exitCode=0
Sep 09 16:25:51.460696 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:51.460624    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg" event={"ID":"657f250d-c4cb-4261-8b68-7de2d582d944","Type":"ContainerDied","Data":"9cb14165ea24ee3cb0eca8fc292f2dfb82326fb3cc0a2312ec585c8115cdd8b0"}
Sep 09 16:25:52.620545 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.620510    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9"
Sep 09 16:25:52.665248 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.665222    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m"
Sep 09 16:25:52.675157 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.675136    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65"
Sep 09 16:25:52.680200 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.680178    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg"
Sep 09 16:25:52.782087 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.782058    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kv7q\" (UniqueName: \"kubernetes.io/projected/50b79790-9bed-46ba-8f67-457d26c52bbc-kube-api-access-4kv7q\") pod \"50b79790-9bed-46ba-8f67-457d26c52bbc\" (UID: \"50b79790-9bed-46ba-8f67-457d26c52bbc\") "
Sep 09 16:25:52.782087 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.782091    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4sfw\" (UniqueName: \"kubernetes.io/projected/657f250d-c4cb-4261-8b68-7de2d582d944-kube-api-access-l4sfw\") pod \"657f250d-c4cb-4261-8b68-7de2d582d944\" (UID: \"657f250d-c4cb-4261-8b68-7de2d582d944\") "
Sep 09 16:25:52.782323 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.782116    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17687187-b4e0-499f-b3ca-119306cd76ba-util\") pod \"17687187-b4e0-499f-b3ca-119306cd76ba\" (UID: \"17687187-b4e0-499f-b3ca-119306cd76ba\") "
Sep 09 16:25:52.782323 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.782143    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8882e519-41bf-4d40-b740-f5d93b85f9bb-util\") pod \"8882e519-41bf-4d40-b740-f5d93b85f9bb\" (UID: \"8882e519-41bf-4d40-b740-f5d93b85f9bb\") "
Sep 09 16:25:52.782323 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.782177    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vslcb\" (UniqueName: \"kubernetes.io/projected/8882e519-41bf-4d40-b740-f5d93b85f9bb-kube-api-access-vslcb\") pod \"8882e519-41bf-4d40-b740-f5d93b85f9bb\" (UID: \"8882e519-41bf-4d40-b740-f5d93b85f9bb\") "
Sep 09 16:25:52.782323 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.782216    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50b79790-9bed-46ba-8f67-457d26c52bbc-util\") pod \"50b79790-9bed-46ba-8f67-457d26c52bbc\" (UID: \"50b79790-9bed-46ba-8f67-457d26c52bbc\") "
Sep 09 16:25:52.782323 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.782246    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvkxz\" (UniqueName: \"kubernetes.io/projected/17687187-b4e0-499f-b3ca-119306cd76ba-kube-api-access-zvkxz\") pod \"17687187-b4e0-499f-b3ca-119306cd76ba\" (UID: \"17687187-b4e0-499f-b3ca-119306cd76ba\") "
Sep 09 16:25:52.782323 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.782283    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8882e519-41bf-4d40-b740-f5d93b85f9bb-bundle\") pod \"8882e519-41bf-4d40-b740-f5d93b85f9bb\" (UID: \"8882e519-41bf-4d40-b740-f5d93b85f9bb\") "
Sep 09 16:25:52.782620 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.782334    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50b79790-9bed-46ba-8f67-457d26c52bbc-bundle\") pod \"50b79790-9bed-46ba-8f67-457d26c52bbc\" (UID: \"50b79790-9bed-46ba-8f67-457d26c52bbc\") "
Sep 09 16:25:52.783039 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.782953    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b79790-9bed-46ba-8f67-457d26c52bbc-bundle" (OuterVolumeSpecName: "bundle") pod "50b79790-9bed-46ba-8f67-457d26c52bbc" (UID: "50b79790-9bed-46ba-8f67-457d26c52bbc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:25:52.783158 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.783067    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/657f250d-c4cb-4261-8b68-7de2d582d944-bundle\") pod \"657f250d-c4cb-4261-8b68-7de2d582d944\" (UID: \"657f250d-c4cb-4261-8b68-7de2d582d944\") "
Sep 09 16:25:52.783158 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.783119    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17687187-b4e0-499f-b3ca-119306cd76ba-bundle\") pod \"17687187-b4e0-499f-b3ca-119306cd76ba\" (UID: \"17687187-b4e0-499f-b3ca-119306cd76ba\") "
Sep 09 16:25:52.783268 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.783162    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/657f250d-c4cb-4261-8b68-7de2d582d944-util\") pod \"657f250d-c4cb-4261-8b68-7de2d582d944\" (UID: \"657f250d-c4cb-4261-8b68-7de2d582d944\") "
Sep 09 16:25:52.783627 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.783457    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50b79790-9bed-46ba-8f67-457d26c52bbc-bundle\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:25:52.784702 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.783854    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17687187-b4e0-499f-b3ca-119306cd76ba-bundle" (OuterVolumeSpecName: "bundle") pod "17687187-b4e0-499f-b3ca-119306cd76ba" (UID: "17687187-b4e0-499f-b3ca-119306cd76ba"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:25:52.784702 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.783988    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8882e519-41bf-4d40-b740-f5d93b85f9bb-bundle" (OuterVolumeSpecName: "bundle") pod "8882e519-41bf-4d40-b740-f5d93b85f9bb" (UID: "8882e519-41bf-4d40-b740-f5d93b85f9bb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:25:52.784702 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.784077    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657f250d-c4cb-4261-8b68-7de2d582d944-bundle" (OuterVolumeSpecName: "bundle") pod "657f250d-c4cb-4261-8b68-7de2d582d944" (UID: "657f250d-c4cb-4261-8b68-7de2d582d944"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:25:52.785038 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.785002    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657f250d-c4cb-4261-8b68-7de2d582d944-kube-api-access-l4sfw" (OuterVolumeSpecName: "kube-api-access-l4sfw") pod "657f250d-c4cb-4261-8b68-7de2d582d944" (UID: "657f250d-c4cb-4261-8b68-7de2d582d944"). InnerVolumeSpecName "kube-api-access-l4sfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:25:52.785872 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.785847    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8882e519-41bf-4d40-b740-f5d93b85f9bb-kube-api-access-vslcb" (OuterVolumeSpecName: "kube-api-access-vslcb") pod "8882e519-41bf-4d40-b740-f5d93b85f9bb" (UID: "8882e519-41bf-4d40-b740-f5d93b85f9bb"). InnerVolumeSpecName "kube-api-access-vslcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:25:52.785967 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.785882    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17687187-b4e0-499f-b3ca-119306cd76ba-kube-api-access-zvkxz" (OuterVolumeSpecName: "kube-api-access-zvkxz") pod "17687187-b4e0-499f-b3ca-119306cd76ba" (UID: "17687187-b4e0-499f-b3ca-119306cd76ba"). InnerVolumeSpecName "kube-api-access-zvkxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:25:52.786024 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.786003    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b79790-9bed-46ba-8f67-457d26c52bbc-kube-api-access-4kv7q" (OuterVolumeSpecName: "kube-api-access-4kv7q") pod "50b79790-9bed-46ba-8f67-457d26c52bbc" (UID: "50b79790-9bed-46ba-8f67-457d26c52bbc"). InnerVolumeSpecName "kube-api-access-4kv7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:25:52.789219 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.789198    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657f250d-c4cb-4261-8b68-7de2d582d944-util" (OuterVolumeSpecName: "util") pod "657f250d-c4cb-4261-8b68-7de2d582d944" (UID: "657f250d-c4cb-4261-8b68-7de2d582d944"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:25:52.790127 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.790108    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b79790-9bed-46ba-8f67-457d26c52bbc-util" (OuterVolumeSpecName: "util") pod "50b79790-9bed-46ba-8f67-457d26c52bbc" (UID: "50b79790-9bed-46ba-8f67-457d26c52bbc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:25:52.791418 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.791394    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8882e519-41bf-4d40-b740-f5d93b85f9bb-util" (OuterVolumeSpecName: "util") pod "8882e519-41bf-4d40-b740-f5d93b85f9bb" (UID: "8882e519-41bf-4d40-b740-f5d93b85f9bb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:25:52.791888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.791873    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17687187-b4e0-499f-b3ca-119306cd76ba-util" (OuterVolumeSpecName: "util") pod "17687187-b4e0-499f-b3ca-119306cd76ba" (UID: "17687187-b4e0-499f-b3ca-119306cd76ba"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:25:52.884778 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.884732    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17687187-b4e0-499f-b3ca-119306cd76ba-bundle\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:25:52.884778 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.884767    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/657f250d-c4cb-4261-8b68-7de2d582d944-util\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:25:52.884778 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.884778    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kv7q\" (UniqueName: \"kubernetes.io/projected/50b79790-9bed-46ba-8f67-457d26c52bbc-kube-api-access-4kv7q\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:25:52.884778 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.884790    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4sfw\" (UniqueName: \"kubernetes.io/projected/657f250d-c4cb-4261-8b68-7de2d582d944-kube-api-access-l4sfw\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:25:52.885045 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.884799    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17687187-b4e0-499f-b3ca-119306cd76ba-util\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:25:52.885045 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.884806    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8882e519-41bf-4d40-b740-f5d93b85f9bb-util\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:25:52.885045 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.884815    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vslcb\" (UniqueName: \"kubernetes.io/projected/8882e519-41bf-4d40-b740-f5d93b85f9bb-kube-api-access-vslcb\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:25:52.885045 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.884823    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50b79790-9bed-46ba-8f67-457d26c52bbc-util\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:25:52.885045 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.884831    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvkxz\" (UniqueName: \"kubernetes.io/projected/17687187-b4e0-499f-b3ca-119306cd76ba-kube-api-access-zvkxz\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:25:52.885045 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.884839    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8882e519-41bf-4d40-b740-f5d93b85f9bb-bundle\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:25:52.885045 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:52.884847    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/657f250d-c4cb-4261-8b68-7de2d582d944-bundle\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:25:53.474273 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:53.474243    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65"
Sep 09 16:25:53.474469 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:53.474273    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7fxz65" event={"ID":"50b79790-9bed-46ba-8f67-457d26c52bbc","Type":"ContainerDied","Data":"49fb8ab1fd9ef4d2609218eb3636ddfdc76eb1870ca6f42f75fdf5e2c34066d7"}
Sep 09 16:25:53.474469 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:53.474348    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49fb8ab1fd9ef4d2609218eb3636ddfdc76eb1870ca6f42f75fdf5e2c34066d7"
Sep 09 16:25:53.476256 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:53.476235    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9"
Sep 09 16:25:53.476256 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:53.476241    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dgcn9" event={"ID":"8882e519-41bf-4d40-b740-f5d93b85f9bb","Type":"ContainerDied","Data":"bff6d8faa919a816994220684c840d2106185000752a667e0bb840db9b28b0c4"}
Sep 09 16:25:53.476487 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:53.476282    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bff6d8faa919a816994220684c840d2106185000752a667e0bb840db9b28b0c4"
Sep 09 16:25:53.478220 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:53.478202    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m"
Sep 09 16:25:53.478372 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:53.478233    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8q22m" event={"ID":"17687187-b4e0-499f-b3ca-119306cd76ba","Type":"ContainerDied","Data":"347eddced53e2f41885c2f22b0cce673cacd90337fc014627efbff467a589053"}
Sep 09 16:25:53.478372 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:53.478268    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="347eddced53e2f41885c2f22b0cce673cacd90337fc014627efbff467a589053"
Sep 09 16:25:53.480412 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:53.480382    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg" event={"ID":"657f250d-c4cb-4261-8b68-7de2d582d944","Type":"ContainerDied","Data":"5da275acb256df1985dde1dfd72e5069ae56b8741048d30224963c640678f52d"}
Sep 09 16:25:53.480412 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:53.480412    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akgsmg"
Sep 09 16:25:53.480595 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:53.480428    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5da275acb256df1985dde1dfd72e5069ae56b8741048d30224963c640678f52d"
Sep 09 16:25:59.020533 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020495    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n267w"]
Sep 09 16:25:59.020917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020784    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17687187-b4e0-499f-b3ca-119306cd76ba" containerName="util"
Sep 09 16:25:59.020917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020795    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="17687187-b4e0-499f-b3ca-119306cd76ba" containerName="util"
Sep 09 16:25:59.020917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020804    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="657f250d-c4cb-4261-8b68-7de2d582d944" containerName="pull"
Sep 09 16:25:59.020917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020809    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="657f250d-c4cb-4261-8b68-7de2d582d944" containerName="pull"
Sep 09 16:25:59.020917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020817    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17687187-b4e0-499f-b3ca-119306cd76ba" containerName="pull"
Sep 09 16:25:59.020917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020823    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="17687187-b4e0-499f-b3ca-119306cd76ba" containerName="pull"
Sep 09 16:25:59.020917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020830    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50b79790-9bed-46ba-8f67-457d26c52bbc" containerName="util"
Sep 09 16:25:59.020917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020835    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="50b79790-9bed-46ba-8f67-457d26c52bbc" containerName="util"
Sep 09 16:25:59.020917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020843    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="657f250d-c4cb-4261-8b68-7de2d582d944" containerName="util"
Sep 09 16:25:59.020917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020848    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="657f250d-c4cb-4261-8b68-7de2d582d944" containerName="util"
Sep 09 16:25:59.020917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020855    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17687187-b4e0-499f-b3ca-119306cd76ba" containerName="extract"
Sep 09 16:25:59.020917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020860    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="17687187-b4e0-499f-b3ca-119306cd76ba" containerName="extract"
Sep 09 16:25:59.020917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020868    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8882e519-41bf-4d40-b740-f5d93b85f9bb" containerName="pull"
Sep 09 16:25:59.020917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020873    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8882e519-41bf-4d40-b740-f5d93b85f9bb" containerName="pull"
Sep 09 16:25:59.020917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020883    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50b79790-9bed-46ba-8f67-457d26c52bbc" containerName="extract"
Sep 09 16:25:59.020917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020889    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="50b79790-9bed-46ba-8f67-457d26c52bbc" containerName="extract"
Sep 09 16:25:59.020917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020893    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="657f250d-c4cb-4261-8b68-7de2d582d944" containerName="extract"
Sep 09 16:25:59.020917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020899    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="657f250d-c4cb-4261-8b68-7de2d582d944" containerName="extract"
Sep 09 16:25:59.020917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020906    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8882e519-41bf-4d40-b740-f5d93b85f9bb" containerName="util"
Sep 09 16:25:59.020917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020911    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8882e519-41bf-4d40-b740-f5d93b85f9bb" containerName="util"
Sep 09 16:25:59.020917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020920    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50b79790-9bed-46ba-8f67-457d26c52bbc" containerName="pull"
Sep 09 16:25:59.020917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020925    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="50b79790-9bed-46ba-8f67-457d26c52bbc" containerName="pull"
Sep 09 16:25:59.021557 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020931    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8882e519-41bf-4d40-b740-f5d93b85f9bb" containerName="extract"
Sep 09 16:25:59.021557 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020936    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8882e519-41bf-4d40-b740-f5d93b85f9bb" containerName="extract"
Sep 09 16:25:59.021557 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020989    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8882e519-41bf-4d40-b740-f5d93b85f9bb" containerName="extract"
Sep 09 16:25:59.021557 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.020998    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="17687187-b4e0-499f-b3ca-119306cd76ba" containerName="extract"
Sep 09 16:25:59.021557 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.021005    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="50b79790-9bed-46ba-8f67-457d26c52bbc" containerName="extract"
Sep 09 16:25:59.021557 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.021012    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="657f250d-c4cb-4261-8b68-7de2d582d944" containerName="extract"
Sep 09 16:25:59.023851 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.023817    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n267w"
Sep 09 16:25:59.028821 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.028792    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 09 16:25:59.028977 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.028870    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-h5zfh\""
Sep 09 16:25:59.028977 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.028806    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 16:25:59.031144 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.031121    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 16:25:59.040832 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.040802    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n267w"]
Sep 09 16:25:59.139352 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.139307    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttdzk\" (UniqueName: \"kubernetes.io/projected/2defd344-8312-4ef2-a1f7-4a18cf2866a4-kube-api-access-ttdzk\") pod \"dns-operator-controller-manager-7b84cd7c9c-n267w\" (UID: \"2defd344-8312-4ef2-a1f7-4a18cf2866a4\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n267w"
Sep 09 16:25:59.240629 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.240583    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttdzk\" (UniqueName: \"kubernetes.io/projected/2defd344-8312-4ef2-a1f7-4a18cf2866a4-kube-api-access-ttdzk\") pod \"dns-operator-controller-manager-7b84cd7c9c-n267w\" (UID: \"2defd344-8312-4ef2-a1f7-4a18cf2866a4\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n267w"
Sep 09 16:25:59.250544 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.250507    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttdzk\" (UniqueName: \"kubernetes.io/projected/2defd344-8312-4ef2-a1f7-4a18cf2866a4-kube-api-access-ttdzk\") pod \"dns-operator-controller-manager-7b84cd7c9c-n267w\" (UID: \"2defd344-8312-4ef2-a1f7-4a18cf2866a4\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n267w"
Sep 09 16:25:59.334001 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.333912    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n267w"
Sep 09 16:25:59.609153 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:25:59.609122    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n267w"]
Sep 09 16:25:59.611870 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:25:59.611840    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2defd344_8312_4ef2_a1f7_4a18cf2866a4.slice/crio-d734aeae5d721c776acf7e780173883ecc6bc0acf45f104da2ad7af481add55e WatchSource:0}: Error finding container d734aeae5d721c776acf7e780173883ecc6bc0acf45f104da2ad7af481add55e: Status 404 returned error can't find the container with id d734aeae5d721c776acf7e780173883ecc6bc0acf45f104da2ad7af481add55e
Sep 09 16:26:00.525602 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:00.525561    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n267w" event={"ID":"2defd344-8312-4ef2-a1f7-4a18cf2866a4","Type":"ContainerStarted","Data":"d734aeae5d721c776acf7e780173883ecc6bc0acf45f104da2ad7af481add55e"}
Sep 09 16:26:01.119385 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:01.119344    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq"]
Sep 09 16:26:01.122405 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:01.122374    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq"
Sep 09 16:26:01.130690 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:01.130664    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dfvhh\""
Sep 09 16:26:01.140589 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:01.140551    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq"]
Sep 09 16:26:01.256646 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:01.256599    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4dcs\" (UniqueName: \"kubernetes.io/projected/4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b-kube-api-access-v4dcs\") pod \"limitador-operator-controller-manager-74495c69f8-mqhzq\" (UID: \"4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq"
Sep 09 16:26:01.357416 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:01.357378    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4dcs\" (UniqueName: \"kubernetes.io/projected/4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b-kube-api-access-v4dcs\") pod \"limitador-operator-controller-manager-74495c69f8-mqhzq\" (UID: \"4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq"
Sep 09 16:26:01.382067 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:01.381987    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4dcs\" (UniqueName: \"kubernetes.io/projected/4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b-kube-api-access-v4dcs\") pod \"limitador-operator-controller-manager-74495c69f8-mqhzq\" (UID: \"4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq"
Sep 09 16:26:01.433979 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:01.433944    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq"
Sep 09 16:26:01.725281 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:01.725244    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq"]
Sep 09 16:26:01.725936 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:26:01.725912    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4066d30a_8ee8_4eea_8bc5_2c3aab58ec0b.slice/crio-f063b524fd5cc5d07f08e852b78218d9e21d83dd90df1ba20c132c448fc5aa6f WatchSource:0}: Error finding container f063b524fd5cc5d07f08e852b78218d9e21d83dd90df1ba20c132c448fc5aa6f: Status 404 returned error can't find the container with id f063b524fd5cc5d07f08e852b78218d9e21d83dd90df1ba20c132c448fc5aa6f
Sep 09 16:26:02.539334 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:02.539277    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq" event={"ID":"4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b","Type":"ContainerStarted","Data":"f063b524fd5cc5d07f08e852b78218d9e21d83dd90df1ba20c132c448fc5aa6f"}
Sep 09 16:26:04.554930 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:04.554880    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq" event={"ID":"4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b","Type":"ContainerStarted","Data":"2ea3a926c5e3d60001dbbbf62f29aa13c8442200dea7acb6551a8a79c046e1cd"}
Sep 09 16:26:04.555435 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:04.555060    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq"
Sep 09 16:26:04.573244 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:04.573126    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq" podStartSLOduration=0.980717351 podStartE2EDuration="3.573102232s" totalImagesPullingTime="2.592384881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:26:04.571895695 +0000 UTC m=+730.402103756" watchObservedRunningTime="2026-09-09 16:26:04.573102232 +0000 UTC m=+730.403310295"
Sep 09 16:26:05.431459 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:05.431421    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"]
Sep 09 16:26:05.434952 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:05.434931    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"
Sep 09 16:26:05.439445 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:05.439418    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bghjj\""
Sep 09 16:26:05.447204 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:05.447177    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"]
Sep 09 16:26:05.562758 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:05.562711    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n267w" event={"ID":"2defd344-8312-4ef2-a1f7-4a18cf2866a4","Type":"ContainerStarted","Data":"b63ec6790d13df7673cb6a5c1d8450b3cdf4480296ccf514067e3a864077f04b"}
Sep 09 16:26:05.563175 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:05.562901    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n267w"
Sep 09 16:26:05.585925 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:05.585863    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n267w" podStartSLOduration=0.992243114 podStartE2EDuration="6.585845516s" totalImagesPullingTime="5.593602402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:25:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:26:05.584937496 +0000 UTC m=+731.415145555" watchObservedRunningTime="2026-09-09 16:26:05.585845516 +0000 UTC m=+731.416053576"
Sep 09 16:26:05.595873 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:05.595831    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0e9b482-f6db-4b3f-a149-5ed37f40210a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qgzm7\" (UID: \"f0e9b482-f6db-4b3f-a149-5ed37f40210a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"
Sep 09 16:26:05.596036 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:05.595881    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nddvj\" (UniqueName: \"kubernetes.io/projected/f0e9b482-f6db-4b3f-a149-5ed37f40210a-kube-api-access-nddvj\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qgzm7\" (UID: \"f0e9b482-f6db-4b3f-a149-5ed37f40210a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"
Sep 09 16:26:05.697377 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:05.697328    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0e9b482-f6db-4b3f-a149-5ed37f40210a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qgzm7\" (UID: \"f0e9b482-f6db-4b3f-a149-5ed37f40210a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"
Sep 09 16:26:05.697586 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:05.697395    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nddvj\" (UniqueName: \"kubernetes.io/projected/f0e9b482-f6db-4b3f-a149-5ed37f40210a-kube-api-access-nddvj\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qgzm7\" (UID: \"f0e9b482-f6db-4b3f-a149-5ed37f40210a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"
Sep 09 16:26:05.697790 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:05.697765    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0e9b482-f6db-4b3f-a149-5ed37f40210a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qgzm7\" (UID: \"f0e9b482-f6db-4b3f-a149-5ed37f40210a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"
Sep 09 16:26:05.707116 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:05.707080    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nddvj\" (UniqueName: \"kubernetes.io/projected/f0e9b482-f6db-4b3f-a149-5ed37f40210a-kube-api-access-nddvj\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qgzm7\" (UID: \"f0e9b482-f6db-4b3f-a149-5ed37f40210a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"
Sep 09 16:26:05.746966 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:05.746927    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"
Sep 09 16:26:06.026716 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:06.026688    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"]
Sep 09 16:26:06.028057 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:26:06.028024    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e9b482_f6db_4b3f_a149_5ed37f40210a.slice/crio-c33b4aae478038cca1fc7a017516a2f2b6e0bfdfe9099b770fe2b5431279d77a WatchSource:0}: Error finding container c33b4aae478038cca1fc7a017516a2f2b6e0bfdfe9099b770fe2b5431279d77a: Status 404 returned error can't find the container with id c33b4aae478038cca1fc7a017516a2f2b6e0bfdfe9099b770fe2b5431279d77a
Sep 09 16:26:06.569489 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:06.569447    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7" event={"ID":"f0e9b482-f6db-4b3f-a149-5ed37f40210a","Type":"ContainerStarted","Data":"c33b4aae478038cca1fc7a017516a2f2b6e0bfdfe9099b770fe2b5431279d77a"}
Sep 09 16:26:07.850925 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:07.850884    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-89jtl"]
Sep 09 16:26:07.854407 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:07.854378    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-89jtl"
Sep 09 16:26:07.857452 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:07.857420    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-x5ch4\""
Sep 09 16:26:07.866600 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:07.866573    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-89jtl"]
Sep 09 16:26:08.019972 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:08.019931    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whxwr\" (UniqueName: \"kubernetes.io/projected/5882ad0b-e8d0-42f5-ab1e-67e64234cc2b-kube-api-access-whxwr\") pod \"authorino-operator-76b478bd78-89jtl\" (UID: \"5882ad0b-e8d0-42f5-ab1e-67e64234cc2b\") " pod="kuadrant-system/authorino-operator-76b478bd78-89jtl"
Sep 09 16:26:08.120702 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:08.120601    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whxwr\" (UniqueName: \"kubernetes.io/projected/5882ad0b-e8d0-42f5-ab1e-67e64234cc2b-kube-api-access-whxwr\") pod \"authorino-operator-76b478bd78-89jtl\" (UID: \"5882ad0b-e8d0-42f5-ab1e-67e64234cc2b\") " pod="kuadrant-system/authorino-operator-76b478bd78-89jtl"
Sep 09 16:26:08.132104 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:08.132067    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whxwr\" (UniqueName: \"kubernetes.io/projected/5882ad0b-e8d0-42f5-ab1e-67e64234cc2b-kube-api-access-whxwr\") pod \"authorino-operator-76b478bd78-89jtl\" (UID: \"5882ad0b-e8d0-42f5-ab1e-67e64234cc2b\") " pod="kuadrant-system/authorino-operator-76b478bd78-89jtl"
Sep 09 16:26:08.167756 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:08.167714    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-89jtl"
Sep 09 16:26:08.452202 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:08.452171    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-89jtl"]
Sep 09 16:26:08.455316 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:26:08.455253    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5882ad0b_e8d0_42f5_ab1e_67e64234cc2b.slice/crio-ce9167dc02c69a450a665e0a8960f830bb7da472279dc449e80b629c887af176 WatchSource:0}: Error finding container ce9167dc02c69a450a665e0a8960f830bb7da472279dc449e80b629c887af176: Status 404 returned error can't find the container with id ce9167dc02c69a450a665e0a8960f830bb7da472279dc449e80b629c887af176
Sep 09 16:26:08.584580 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:08.584539    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-89jtl" event={"ID":"5882ad0b-e8d0-42f5-ab1e-67e64234cc2b","Type":"ContainerStarted","Data":"ce9167dc02c69a450a665e0a8960f830bb7da472279dc449e80b629c887af176"}
Sep 09 16:26:10.600382 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:10.600316    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7" event={"ID":"f0e9b482-f6db-4b3f-a149-5ed37f40210a","Type":"ContainerStarted","Data":"95d85b67b62faa72506db959e1232574b752ad93323ebd3032863f592711e517"}
Sep 09 16:26:10.622702 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:10.622632    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7" podStartSLOduration=1.211398912 podStartE2EDuration="5.622609034s" totalImagesPullingTime="4.411210122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:26:10.620108103 +0000 UTC m=+736.450316160" watchObservedRunningTime="2026-09-09 16:26:10.622609034 +0000 UTC m=+736.452817092"
Sep 09 16:26:11.607665 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:11.607619    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-89jtl" event={"ID":"5882ad0b-e8d0-42f5-ab1e-67e64234cc2b","Type":"ContainerStarted","Data":"a3133435239ab02e1bdb4f273ff2b353a4e7f08a660675d4ed363b66002ae218"}
Sep 09 16:26:11.608095 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:11.607807    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-89jtl"
Sep 09 16:26:11.608095 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:11.607849    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"
Sep 09 16:26:11.626015 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:11.625961    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-89jtl" podStartSLOduration=1.978487189 podStartE2EDuration="4.625945616s" totalImagesPullingTime="2.647458427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:26:11.624166306 +0000 UTC m=+737.454374364" watchObservedRunningTime="2026-09-09 16:26:11.625945616 +0000 UTC m=+737.456153672"
Sep 09 16:26:15.565362 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:15.565324    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq"
Sep 09 16:26:16.572319 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:16.572263    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-n267w"
Sep 09 16:26:22.614325 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:22.614277    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"
Sep 09 16:26:22.614792 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:22.614371    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-89jtl"
Sep 09 16:26:24.044837 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:24.044801    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s74g6"]
Sep 09 16:26:24.079613 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:24.079576    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s74g6"
Sep 09 16:26:24.085164 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:24.085133    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s74g6"]
Sep 09 16:26:24.139121 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:24.139076    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5vt9\" (UniqueName: \"kubernetes.io/projected/eaff5f45-47ec-4968-87f0-6f5a81333bc0-kube-api-access-r5vt9\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-s74g6\" (UID: \"eaff5f45-47ec-4968-87f0-6f5a81333bc0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s74g6"
Sep 09 16:26:24.139321 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:24.139133    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eaff5f45-47ec-4968-87f0-6f5a81333bc0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-s74g6\" (UID: \"eaff5f45-47ec-4968-87f0-6f5a81333bc0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s74g6"
Sep 09 16:26:24.239747 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:24.239706    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5vt9\" (UniqueName: \"kubernetes.io/projected/eaff5f45-47ec-4968-87f0-6f5a81333bc0-kube-api-access-r5vt9\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-s74g6\" (UID: \"eaff5f45-47ec-4968-87f0-6f5a81333bc0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s74g6"
Sep 09 16:26:24.239747 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:24.239757    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eaff5f45-47ec-4968-87f0-6f5a81333bc0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-s74g6\" (UID: \"eaff5f45-47ec-4968-87f0-6f5a81333bc0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s74g6"
Sep 09 16:26:24.240169 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:24.240147    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eaff5f45-47ec-4968-87f0-6f5a81333bc0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-s74g6\" (UID: \"eaff5f45-47ec-4968-87f0-6f5a81333bc0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s74g6"
Sep 09 16:26:24.254120 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:24.254092    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5vt9\" (UniqueName: \"kubernetes.io/projected/eaff5f45-47ec-4968-87f0-6f5a81333bc0-kube-api-access-r5vt9\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-s74g6\" (UID: \"eaff5f45-47ec-4968-87f0-6f5a81333bc0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s74g6"
Sep 09 16:26:24.392359 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:24.392223    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s74g6"
Sep 09 16:26:24.685042 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:24.684980    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s74g6"]
Sep 09 16:26:24.687385 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:26:24.687340    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaff5f45_47ec_4968_87f0_6f5a81333bc0.slice/crio-99198bedd092efad2e3b537e8507b5a57179d532571d4951ba2d2465993ea3fe WatchSource:0}: Error finding container 99198bedd092efad2e3b537e8507b5a57179d532571d4951ba2d2465993ea3fe: Status 404 returned error can't find the container with id 99198bedd092efad2e3b537e8507b5a57179d532571d4951ba2d2465993ea3fe
Sep 09 16:26:25.159311 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.159248    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s74g6"]
Sep 09 16:26:25.172929 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.172889    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s74g6"]
Sep 09 16:26:25.192899 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.192862    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"]
Sep 09 16:26:25.193096 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.193071    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7" podUID="f0e9b482-f6db-4b3f-a149-5ed37f40210a" containerName="manager" containerID="cri-o://95d85b67b62faa72506db959e1232574b752ad93323ebd3032863f592711e517" gracePeriod=2
Sep 09 16:26:25.199495 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.199467    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h4b8s"]
Sep 09 16:26:25.200090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.200063    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaff5f45-47ec-4968-87f0-6f5a81333bc0" containerName="manager"
Sep 09 16:26:25.200233 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.200219    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaff5f45-47ec-4968-87f0-6f5a81333bc0" containerName="manager"
Sep 09 16:26:25.200451 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.200437    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaff5f45-47ec-4968-87f0-6f5a81333bc0" containerName="manager"
Sep 09 16:26:25.203664 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:25.203627    2555 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"f0e9b482-f6db-4b3f-a149-5ed37f40210a\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-qgzm7\": pods \"kuadrant-operator-controller-manager-76f8f686bf-qgzm7\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"
Sep 09 16:26:25.219143 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.219104    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"]
Sep 09 16:26:25.219345 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.219158    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq"]
Sep 09 16:26:25.219345 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.219220    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h4b8s"
Sep 09 16:26:25.219823 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.219768    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq" podUID="4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b" containerName="manager" containerID="cri-o://2ea3a926c5e3d60001dbbbf62f29aa13c8442200dea7acb6551a8a79c046e1cd" gracePeriod=2
Sep 09 16:26:25.222251 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:25.222216    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-qgzm7\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="f0e9b482-f6db-4b3f-a149-5ed37f40210a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"
Sep 09 16:26:25.225070 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.225032    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h4b8s"]
Sep 09 16:26:25.228112 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.228084    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2s7h5"]
Sep 09 16:26:25.228730 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.228707    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0e9b482-f6db-4b3f-a149-5ed37f40210a" containerName="manager"
Sep 09 16:26:25.228730 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.228730    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0e9b482-f6db-4b3f-a149-5ed37f40210a" containerName="manager"
Sep 09 16:26:25.228892 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.228835    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0e9b482-f6db-4b3f-a149-5ed37f40210a" containerName="manager"
Sep 09 16:26:25.247334 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:25.246560    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mqhzq\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq"
Sep 09 16:26:25.249099 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:25.249060    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-qgzm7\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="f0e9b482-f6db-4b3f-a149-5ed37f40210a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"
Sep 09 16:26:25.249323 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.249283    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m85w9\" (UniqueName: \"kubernetes.io/projected/29e2a722-d0e9-44ab-8554-d2874dc89b63-kube-api-access-m85w9\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-h4b8s\" (UID: \"29e2a722-d0e9-44ab-8554-d2874dc89b63\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h4b8s"
Sep 09 16:26:25.249509 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.249328    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq"]
Sep 09 16:26:25.249680 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.249532    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29e2a722-d0e9-44ab-8554-d2874dc89b63-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-h4b8s\" (UID: \"29e2a722-d0e9-44ab-8554-d2874dc89b63\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h4b8s"
Sep 09 16:26:25.249680 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.249557    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2s7h5"]
Sep 09 16:26:25.249680 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.249395    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2s7h5"
Sep 09 16:26:25.252922 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:25.252891    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-qgzm7\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="f0e9b482-f6db-4b3f-a149-5ed37f40210a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"
Sep 09 16:26:25.281628 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:25.281585    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mqhzq\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq"
Sep 09 16:26:25.352186 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.350691    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m85w9\" (UniqueName: \"kubernetes.io/projected/29e2a722-d0e9-44ab-8554-d2874dc89b63-kube-api-access-m85w9\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-h4b8s\" (UID: \"29e2a722-d0e9-44ab-8554-d2874dc89b63\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h4b8s"
Sep 09 16:26:25.352186 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.350864    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29e2a722-d0e9-44ab-8554-d2874dc89b63-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-h4b8s\" (UID: \"29e2a722-d0e9-44ab-8554-d2874dc89b63\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h4b8s"
Sep 09 16:26:25.352186 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.350908    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/becb9765-a7de-48c2-b9ee-e83af6e6c395-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2s7h5\" (UID: \"becb9765-a7de-48c2-b9ee-e83af6e6c395\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2s7h5"
Sep 09 16:26:25.352186 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.350950    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbl59\" (UniqueName: \"kubernetes.io/projected/becb9765-a7de-48c2-b9ee-e83af6e6c395-kube-api-access-cbl59\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2s7h5\" (UID: \"becb9765-a7de-48c2-b9ee-e83af6e6c395\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2s7h5"
Sep 09 16:26:25.352186 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.351662    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29e2a722-d0e9-44ab-8554-d2874dc89b63-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-h4b8s\" (UID: \"29e2a722-d0e9-44ab-8554-d2874dc89b63\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h4b8s"
Sep 09 16:26:25.363370 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.363341    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m85w9\" (UniqueName: \"kubernetes.io/projected/29e2a722-d0e9-44ab-8554-d2874dc89b63-kube-api-access-m85w9\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-h4b8s\" (UID: \"29e2a722-d0e9-44ab-8554-d2874dc89b63\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h4b8s"
Sep 09 16:26:25.452541 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.452498    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/becb9765-a7de-48c2-b9ee-e83af6e6c395-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2s7h5\" (UID: \"becb9765-a7de-48c2-b9ee-e83af6e6c395\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2s7h5"
Sep 09 16:26:25.452719 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.452556    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbl59\" (UniqueName: \"kubernetes.io/projected/becb9765-a7de-48c2-b9ee-e83af6e6c395-kube-api-access-cbl59\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2s7h5\" (UID: \"becb9765-a7de-48c2-b9ee-e83af6e6c395\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2s7h5"
Sep 09 16:26:25.452939 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.452915    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/becb9765-a7de-48c2-b9ee-e83af6e6c395-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2s7h5\" (UID: \"becb9765-a7de-48c2-b9ee-e83af6e6c395\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2s7h5"
Sep 09 16:26:25.463895 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.463868    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbl59\" (UniqueName: \"kubernetes.io/projected/becb9765-a7de-48c2-b9ee-e83af6e6c395-kube-api-access-cbl59\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2s7h5\" (UID: \"becb9765-a7de-48c2-b9ee-e83af6e6c395\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2s7h5"
Sep 09 16:26:25.503731 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.503695    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"
Sep 09 16:26:25.506395 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:25.506359    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-qgzm7\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="f0e9b482-f6db-4b3f-a149-5ed37f40210a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"
Sep 09 16:26:25.508669 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:25.508633    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mqhzq\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq"
Sep 09 16:26:25.517633 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.517610    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq"
Sep 09 16:26:25.520068 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:25.520037    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-qgzm7\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="f0e9b482-f6db-4b3f-a149-5ed37f40210a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"
Sep 09 16:26:25.522009 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:25.521985    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mqhzq\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq"
Sep 09 16:26:25.553454 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.553417    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4dcs\" (UniqueName: \"kubernetes.io/projected/4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b-kube-api-access-v4dcs\") pod \"4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b\" (UID: \"4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b\") "
Sep 09 16:26:25.553658 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.553516    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0e9b482-f6db-4b3f-a149-5ed37f40210a-extensions-socket-volume\") pod \"f0e9b482-f6db-4b3f-a149-5ed37f40210a\" (UID: \"f0e9b482-f6db-4b3f-a149-5ed37f40210a\") "
Sep 09 16:26:25.553658 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.553540    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nddvj\" (UniqueName: \"kubernetes.io/projected/f0e9b482-f6db-4b3f-a149-5ed37f40210a-kube-api-access-nddvj\") pod \"f0e9b482-f6db-4b3f-a149-5ed37f40210a\" (UID: \"f0e9b482-f6db-4b3f-a149-5ed37f40210a\") "
Sep 09 16:26:25.554033 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.553994    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e9b482-f6db-4b3f-a149-5ed37f40210a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f0e9b482-f6db-4b3f-a149-5ed37f40210a" (UID: "f0e9b482-f6db-4b3f-a149-5ed37f40210a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:26:25.555691 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.555667    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e9b482-f6db-4b3f-a149-5ed37f40210a-kube-api-access-nddvj" (OuterVolumeSpecName: "kube-api-access-nddvj") pod "f0e9b482-f6db-4b3f-a149-5ed37f40210a" (UID: "f0e9b482-f6db-4b3f-a149-5ed37f40210a"). InnerVolumeSpecName "kube-api-access-nddvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:26:25.555785 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.555729    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b-kube-api-access-v4dcs" (OuterVolumeSpecName: "kube-api-access-v4dcs") pod "4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b" (UID: "4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b"). InnerVolumeSpecName "kube-api-access-v4dcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:26:25.639988 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.639951    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h4b8s"
Sep 09 16:26:25.647984 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.647945    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2s7h5"
Sep 09 16:26:25.654467 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.654437    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4dcs\" (UniqueName: \"kubernetes.io/projected/4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b-kube-api-access-v4dcs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:26:25.654467 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.654464    2555 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0e9b482-f6db-4b3f-a149-5ed37f40210a-extensions-socket-volume\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:26:25.654467 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.654476    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nddvj\" (UniqueName: \"kubernetes.io/projected/f0e9b482-f6db-4b3f-a149-5ed37f40210a-kube-api-access-nddvj\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:26:25.689489 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.689454    2555 generic.go:356] "Generic (PLEG): container finished" podID="f0e9b482-f6db-4b3f-a149-5ed37f40210a" containerID="95d85b67b62faa72506db959e1232574b752ad93323ebd3032863f592711e517" exitCode=0
Sep 09 16:26:25.689645 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.689504    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"
Sep 09 16:26:25.689645 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.689561    2555 scope.go:117] "RemoveContainer" containerID="95d85b67b62faa72506db959e1232574b752ad93323ebd3032863f592711e517"
Sep 09 16:26:25.691649 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.691618    2555 generic.go:356] "Generic (PLEG): container finished" podID="4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b" containerID="2ea3a926c5e3d60001dbbbf62f29aa13c8442200dea7acb6551a8a79c046e1cd" exitCode=0
Sep 09 16:26:25.691764 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.691673    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq"
Sep 09 16:26:25.692665 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:25.692621    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mqhzq\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq"
Sep 09 16:26:25.695084 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:25.695060    2555 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s74g6" container="manager"
Sep 09 16:26:25.695371 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:25.695325    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-qgzm7\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="f0e9b482-f6db-4b3f-a149-5ed37f40210a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"
Sep 09 16:26:25.699259 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:25.699159    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mqhzq\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq"
Sep 09 16:26:25.702308 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:25.702254    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-s74g6\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="eaff5f45-47ec-4968-87f0-6f5a81333bc0" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s74g6"
Sep 09 16:26:25.703360 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.702935    2555 scope.go:117] "RemoveContainer" containerID="95d85b67b62faa72506db959e1232574b752ad93323ebd3032863f592711e517"
Sep 09 16:26:25.704058 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:25.703866    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95d85b67b62faa72506db959e1232574b752ad93323ebd3032863f592711e517\": container with ID starting with 95d85b67b62faa72506db959e1232574b752ad93323ebd3032863f592711e517 not found: ID does not exist" containerID="95d85b67b62faa72506db959e1232574b752ad93323ebd3032863f592711e517"
Sep 09 16:26:25.704058 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.703904    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d85b67b62faa72506db959e1232574b752ad93323ebd3032863f592711e517"} err="failed to get container status \"95d85b67b62faa72506db959e1232574b752ad93323ebd3032863f592711e517\": rpc error: code = NotFound desc = could not find container \"95d85b67b62faa72506db959e1232574b752ad93323ebd3032863f592711e517\": container with ID starting with 95d85b67b62faa72506db959e1232574b752ad93323ebd3032863f592711e517 not found: ID does not exist"
Sep 09 16:26:25.704058 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.703928    2555 scope.go:117] "RemoveContainer" containerID="2ea3a926c5e3d60001dbbbf62f29aa13c8442200dea7acb6551a8a79c046e1cd"
Sep 09 16:26:25.704771 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:25.704626    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-qgzm7\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="f0e9b482-f6db-4b3f-a149-5ed37f40210a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"
Sep 09 16:26:25.706839 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:25.706811    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mqhzq\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq"
Sep 09 16:26:25.709344 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:25.709311    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-s74g6\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="eaff5f45-47ec-4968-87f0-6f5a81333bc0" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s74g6"
Sep 09 16:26:25.711577 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:25.711544    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-qgzm7\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="f0e9b482-f6db-4b3f-a149-5ed37f40210a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"
Sep 09 16:26:25.714161 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:25.714052    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-s74g6\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="eaff5f45-47ec-4968-87f0-6f5a81333bc0" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s74g6"
Sep 09 16:26:25.716075 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.715878    2555 scope.go:117] "RemoveContainer" containerID="2ea3a926c5e3d60001dbbbf62f29aa13c8442200dea7acb6551a8a79c046e1cd"
Sep 09 16:26:25.716711 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:25.716152    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-qgzm7\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="f0e9b482-f6db-4b3f-a149-5ed37f40210a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qgzm7"
Sep 09 16:26:25.716802 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:25.716722    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea3a926c5e3d60001dbbbf62f29aa13c8442200dea7acb6551a8a79c046e1cd\": container with ID starting with 2ea3a926c5e3d60001dbbbf62f29aa13c8442200dea7acb6551a8a79c046e1cd not found: ID does not exist" containerID="2ea3a926c5e3d60001dbbbf62f29aa13c8442200dea7acb6551a8a79c046e1cd"
Sep 09 16:26:25.716802 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.716761    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea3a926c5e3d60001dbbbf62f29aa13c8442200dea7acb6551a8a79c046e1cd"} err="failed to get container status \"2ea3a926c5e3d60001dbbbf62f29aa13c8442200dea7acb6551a8a79c046e1cd\": rpc error: code = NotFound desc = could not find container \"2ea3a926c5e3d60001dbbbf62f29aa13c8442200dea7acb6551a8a79c046e1cd\": container with ID starting with 2ea3a926c5e3d60001dbbbf62f29aa13c8442200dea7acb6551a8a79c046e1cd not found: ID does not exist"
Sep 09 16:26:25.718716 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:25.718681    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mqhzq\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mqhzq"
Sep 09 16:26:25.946130 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.945950    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h4b8s"]
Sep 09 16:26:25.949228 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:26:25.949199    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e2a722_d0e9_44ab_8554_d2874dc89b63.slice/crio-01d498c2cb553b323b45d475458fb6962b7b8f5834aba90d7ecb2fd1c87c9d8a WatchSource:0}: Error finding container 01d498c2cb553b323b45d475458fb6962b7b8f5834aba90d7ecb2fd1c87c9d8a: Status 404 returned error can't find the container with id 01d498c2cb553b323b45d475458fb6962b7b8f5834aba90d7ecb2fd1c87c9d8a
Sep 09 16:26:25.973057 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:25.973031    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2s7h5"]
Sep 09 16:26:25.974432 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:26:25.974404    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbecb9765_a7de_48c2_b9ee_e83af6e6c395.slice/crio-ca672a83f116f7dbcda3b25828dbb4d915412734b9dc5eb8289c6f2aaed0129a WatchSource:0}: Error finding container ca672a83f116f7dbcda3b25828dbb4d915412734b9dc5eb8289c6f2aaed0129a: Status 404 returned error can't find the container with id ca672a83f116f7dbcda3b25828dbb4d915412734b9dc5eb8289c6f2aaed0129a
Sep 09 16:26:26.608040 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:26.607958    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b" path="/var/lib/kubelet/pods/4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b/volumes"
Sep 09 16:26:26.608440 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:26.608395    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0e9b482-f6db-4b3f-a149-5ed37f40210a" path="/var/lib/kubelet/pods/f0e9b482-f6db-4b3f-a149-5ed37f40210a/volumes"
Sep 09 16:26:26.704546 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:26.704510    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s74g6" podUID="eaff5f45-47ec-4968-87f0-6f5a81333bc0" containerName="manager" containerID="cri-o://783817d7d74f8e53e6dee968e685a38c77795e6886df8ed9c8cfb3a2ba3f981d" gracePeriod=2
Sep 09 16:26:26.706347 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:26.706321    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h4b8s" event={"ID":"29e2a722-d0e9-44ab-8554-d2874dc89b63","Type":"ContainerStarted","Data":"82345b25e6513b8bed34589e3a7d8317842f509295210b326a356ea1e09afd35"}
Sep 09 16:26:26.706455 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:26.706367    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h4b8s" event={"ID":"29e2a722-d0e9-44ab-8554-d2874dc89b63","Type":"ContainerStarted","Data":"01d498c2cb553b323b45d475458fb6962b7b8f5834aba90d7ecb2fd1c87c9d8a"}
Sep 09 16:26:26.706510 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:26.706491    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h4b8s"
Sep 09 16:26:26.708222 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:26.708202    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2s7h5" event={"ID":"becb9765-a7de-48c2-b9ee-e83af6e6c395","Type":"ContainerStarted","Data":"e02c50ddea75da77052352eb0e2d64b4b77d227000222367b75baa14854182c7"}
Sep 09 16:26:26.708343 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:26.708242    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2s7h5" event={"ID":"becb9765-a7de-48c2-b9ee-e83af6e6c395","Type":"ContainerStarted","Data":"ca672a83f116f7dbcda3b25828dbb4d915412734b9dc5eb8289c6f2aaed0129a"}
Sep 09 16:26:26.708343 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:26.708331    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2s7h5"
Sep 09 16:26:26.731974 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:26.731911    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h4b8s" podStartSLOduration=1.7318951980000001 podStartE2EDuration="1.731895198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:26:26.728694433 +0000 UTC m=+752.558902492" watchObservedRunningTime="2026-09-09 16:26:26.731895198 +0000 UTC m=+752.562103257"
Sep 09 16:26:26.750232 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:26.750167    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2s7h5" podStartSLOduration=1.750151095 podStartE2EDuration="1.750151095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:26:26.748547675 +0000 UTC m=+752.578755728" watchObservedRunningTime="2026-09-09 16:26:26.750151095 +0000 UTC m=+752.580359154"
Sep 09 16:26:26.977636 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:26.977600    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s74g6"
Sep 09 16:26:26.980238 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:26.980206    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-s74g6\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="eaff5f45-47ec-4968-87f0-6f5a81333bc0" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s74g6"
Sep 09 16:26:27.065631 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:27.065596    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eaff5f45-47ec-4968-87f0-6f5a81333bc0-extensions-socket-volume\") pod \"eaff5f45-47ec-4968-87f0-6f5a81333bc0\" (UID: \"eaff5f45-47ec-4968-87f0-6f5a81333bc0\") "
Sep 09 16:26:27.065810 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:27.065696    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5vt9\" (UniqueName: \"kubernetes.io/projected/eaff5f45-47ec-4968-87f0-6f5a81333bc0-kube-api-access-r5vt9\") pod \"eaff5f45-47ec-4968-87f0-6f5a81333bc0\" (UID: \"eaff5f45-47ec-4968-87f0-6f5a81333bc0\") "
Sep 09 16:26:27.066042 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:27.066019    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaff5f45-47ec-4968-87f0-6f5a81333bc0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "eaff5f45-47ec-4968-87f0-6f5a81333bc0" (UID: "eaff5f45-47ec-4968-87f0-6f5a81333bc0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:26:27.067851 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:27.067813    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaff5f45-47ec-4968-87f0-6f5a81333bc0-kube-api-access-r5vt9" (OuterVolumeSpecName: "kube-api-access-r5vt9") pod "eaff5f45-47ec-4968-87f0-6f5a81333bc0" (UID: "eaff5f45-47ec-4968-87f0-6f5a81333bc0"). InnerVolumeSpecName "kube-api-access-r5vt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:26:27.167100 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:27.167005    2555 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eaff5f45-47ec-4968-87f0-6f5a81333bc0-extensions-socket-volume\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:26:27.167100 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:27.167041    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5vt9\" (UniqueName: \"kubernetes.io/projected/eaff5f45-47ec-4968-87f0-6f5a81333bc0-kube-api-access-r5vt9\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:26:27.715259 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:27.715222    2555 generic.go:356] "Generic (PLEG): container finished" podID="eaff5f45-47ec-4968-87f0-6f5a81333bc0" containerID="783817d7d74f8e53e6dee968e685a38c77795e6886df8ed9c8cfb3a2ba3f981d" exitCode=0
Sep 09 16:26:27.715757 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:27.715267    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s74g6"
Sep 09 16:26:27.715757 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:27.715353    2555 scope.go:117] "RemoveContainer" containerID="783817d7d74f8e53e6dee968e685a38c77795e6886df8ed9c8cfb3a2ba3f981d"
Sep 09 16:26:27.717828 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:27.717803    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-s74g6\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="eaff5f45-47ec-4968-87f0-6f5a81333bc0" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s74g6"
Sep 09 16:26:27.724367 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:27.724342    2555 scope.go:117] "RemoveContainer" containerID="783817d7d74f8e53e6dee968e685a38c77795e6886df8ed9c8cfb3a2ba3f981d"
Sep 09 16:26:27.724635 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:27.724616    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"783817d7d74f8e53e6dee968e685a38c77795e6886df8ed9c8cfb3a2ba3f981d\": container with ID starting with 783817d7d74f8e53e6dee968e685a38c77795e6886df8ed9c8cfb3a2ba3f981d not found: ID does not exist" containerID="783817d7d74f8e53e6dee968e685a38c77795e6886df8ed9c8cfb3a2ba3f981d"
Sep 09 16:26:27.724687 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:27.724645    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"783817d7d74f8e53e6dee968e685a38c77795e6886df8ed9c8cfb3a2ba3f981d"} err="failed to get container status \"783817d7d74f8e53e6dee968e685a38c77795e6886df8ed9c8cfb3a2ba3f981d\": rpc error: code = NotFound desc = could not find container \"783817d7d74f8e53e6dee968e685a38c77795e6886df8ed9c8cfb3a2ba3f981d\": container with ID starting with 783817d7d74f8e53e6dee968e685a38c77795e6886df8ed9c8cfb3a2ba3f981d not found: ID does not exist"
Sep 09 16:26:27.725687 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:27.725666    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-s74g6\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="eaff5f45-47ec-4968-87f0-6f5a81333bc0" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-s74g6"
Sep 09 16:26:28.607780 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:28.607745    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaff5f45-47ec-4968-87f0-6f5a81333bc0" path="/var/lib/kubelet/pods/eaff5f45-47ec-4968-87f0-6f5a81333bc0/volumes"
Sep 09 16:26:37.718087 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:37.718050    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2s7h5"
Sep 09 16:26:37.718591 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:37.718134    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h4b8s"
Sep 09 16:26:37.780538 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:37.780502    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2s7h5"]
Sep 09 16:26:37.780741 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:37.780714    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2s7h5" podUID="becb9765-a7de-48c2-b9ee-e83af6e6c395" containerName="manager" containerID="cri-o://e02c50ddea75da77052352eb0e2d64b4b77d227000222367b75baa14854182c7" gracePeriod=10
Sep 09 16:26:37.973003 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:37.972317    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w5vbz"]
Sep 09 16:26:37.973003 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:37.972825    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b" containerName="manager"
Sep 09 16:26:37.973003 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:37.972841    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b" containerName="manager"
Sep 09 16:26:37.973003 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:37.972941    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4066d30a-8ee8-4eea-8bc5-2c3aab58ec0b" containerName="manager"
Sep 09 16:26:37.975313 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:37.975272    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w5vbz"
Sep 09 16:26:37.985996 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:37.985962    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w5vbz"]
Sep 09 16:26:38.033998 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.033968    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2s7h5"
Sep 09 16:26:38.067376 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.067343    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/becb9765-a7de-48c2-b9ee-e83af6e6c395-extensions-socket-volume\") pod \"becb9765-a7de-48c2-b9ee-e83af6e6c395\" (UID: \"becb9765-a7de-48c2-b9ee-e83af6e6c395\") "
Sep 09 16:26:38.067567 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.067423    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbl59\" (UniqueName: \"kubernetes.io/projected/becb9765-a7de-48c2-b9ee-e83af6e6c395-kube-api-access-cbl59\") pod \"becb9765-a7de-48c2-b9ee-e83af6e6c395\" (UID: \"becb9765-a7de-48c2-b9ee-e83af6e6c395\") "
Sep 09 16:26:38.067567 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.067554    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0ecae37-92af-4dfc-98d1-3741d1366000-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w5vbz\" (UID: \"b0ecae37-92af-4dfc-98d1-3741d1366000\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w5vbz"
Sep 09 16:26:38.067798 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.067623    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr472\" (UniqueName: \"kubernetes.io/projected/b0ecae37-92af-4dfc-98d1-3741d1366000-kube-api-access-jr472\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w5vbz\" (UID: \"b0ecae37-92af-4dfc-98d1-3741d1366000\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w5vbz"
Sep 09 16:26:38.067798 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.067757    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/becb9765-a7de-48c2-b9ee-e83af6e6c395-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "becb9765-a7de-48c2-b9ee-e83af6e6c395" (UID: "becb9765-a7de-48c2-b9ee-e83af6e6c395"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:26:38.069594 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.069567    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/becb9765-a7de-48c2-b9ee-e83af6e6c395-kube-api-access-cbl59" (OuterVolumeSpecName: "kube-api-access-cbl59") pod "becb9765-a7de-48c2-b9ee-e83af6e6c395" (UID: "becb9765-a7de-48c2-b9ee-e83af6e6c395"). InnerVolumeSpecName "kube-api-access-cbl59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:26:38.168472 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.168433    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr472\" (UniqueName: \"kubernetes.io/projected/b0ecae37-92af-4dfc-98d1-3741d1366000-kube-api-access-jr472\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w5vbz\" (UID: \"b0ecae37-92af-4dfc-98d1-3741d1366000\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w5vbz"
Sep 09 16:26:38.168660 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.168499    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0ecae37-92af-4dfc-98d1-3741d1366000-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w5vbz\" (UID: \"b0ecae37-92af-4dfc-98d1-3741d1366000\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w5vbz"
Sep 09 16:26:38.168660 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.168565    2555 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/becb9765-a7de-48c2-b9ee-e83af6e6c395-extensions-socket-volume\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:26:38.168660 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.168580    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbl59\" (UniqueName: \"kubernetes.io/projected/becb9765-a7de-48c2-b9ee-e83af6e6c395-kube-api-access-cbl59\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:26:38.168866 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.168848    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0ecae37-92af-4dfc-98d1-3741d1366000-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w5vbz\" (UID: \"b0ecae37-92af-4dfc-98d1-3741d1366000\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w5vbz"
Sep 09 16:26:38.176506 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.176472    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr472\" (UniqueName: \"kubernetes.io/projected/b0ecae37-92af-4dfc-98d1-3741d1366000-kube-api-access-jr472\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w5vbz\" (UID: \"b0ecae37-92af-4dfc-98d1-3741d1366000\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w5vbz"
Sep 09 16:26:38.288518 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.288476    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w5vbz"
Sep 09 16:26:38.561628 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.561594    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w5vbz"]
Sep 09 16:26:38.563030 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:26:38.562996    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ecae37_92af_4dfc_98d1_3741d1366000.slice/crio-4613fff4ec8f606d7f33bfac8cbc39a1b9b97d97daf84c82f173215f3228ff84 WatchSource:0}: Error finding container 4613fff4ec8f606d7f33bfac8cbc39a1b9b97d97daf84c82f173215f3228ff84: Status 404 returned error can't find the container with id 4613fff4ec8f606d7f33bfac8cbc39a1b9b97d97daf84c82f173215f3228ff84
Sep 09 16:26:38.786079 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.786037    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w5vbz" event={"ID":"b0ecae37-92af-4dfc-98d1-3741d1366000","Type":"ContainerStarted","Data":"8ef50e5d6b7f905a56799e4c56579f89ebaa6fc3905001f2618aca48560d5107"}
Sep 09 16:26:38.786552 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.786204    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w5vbz"
Sep 09 16:26:38.786552 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.786241    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w5vbz" event={"ID":"b0ecae37-92af-4dfc-98d1-3741d1366000","Type":"ContainerStarted","Data":"4613fff4ec8f606d7f33bfac8cbc39a1b9b97d97daf84c82f173215f3228ff84"}
Sep 09 16:26:38.787571 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.787543    2555 generic.go:356] "Generic (PLEG): container finished" podID="becb9765-a7de-48c2-b9ee-e83af6e6c395" containerID="e02c50ddea75da77052352eb0e2d64b4b77d227000222367b75baa14854182c7" exitCode=0
Sep 09 16:26:38.787686 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.787584    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2s7h5"
Sep 09 16:26:38.787686 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.787600    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2s7h5" event={"ID":"becb9765-a7de-48c2-b9ee-e83af6e6c395","Type":"ContainerDied","Data":"e02c50ddea75da77052352eb0e2d64b4b77d227000222367b75baa14854182c7"}
Sep 09 16:26:38.787686 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.787647    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2s7h5" event={"ID":"becb9765-a7de-48c2-b9ee-e83af6e6c395","Type":"ContainerDied","Data":"ca672a83f116f7dbcda3b25828dbb4d915412734b9dc5eb8289c6f2aaed0129a"}
Sep 09 16:26:38.787686 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.787676    2555 scope.go:117] "RemoveContainer" containerID="e02c50ddea75da77052352eb0e2d64b4b77d227000222367b75baa14854182c7"
Sep 09 16:26:38.805479 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.805150    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w5vbz" podStartSLOduration=1.805126628 podStartE2EDuration="1.805126628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:26:38.802660078 +0000 UTC m=+764.632868139" watchObservedRunningTime="2026-09-09 16:26:38.805126628 +0000 UTC m=+764.635334690"
Sep 09 16:26:38.809000 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.808979    2555 scope.go:117] "RemoveContainer" containerID="e02c50ddea75da77052352eb0e2d64b4b77d227000222367b75baa14854182c7"
Sep 09 16:26:38.809468 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:38.809434    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e02c50ddea75da77052352eb0e2d64b4b77d227000222367b75baa14854182c7\": container with ID starting with e02c50ddea75da77052352eb0e2d64b4b77d227000222367b75baa14854182c7 not found: ID does not exist" containerID="e02c50ddea75da77052352eb0e2d64b4b77d227000222367b75baa14854182c7"
Sep 09 16:26:38.809600 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.809480    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e02c50ddea75da77052352eb0e2d64b4b77d227000222367b75baa14854182c7"} err="failed to get container status \"e02c50ddea75da77052352eb0e2d64b4b77d227000222367b75baa14854182c7\": rpc error: code = NotFound desc = could not find container \"e02c50ddea75da77052352eb0e2d64b4b77d227000222367b75baa14854182c7\": container with ID starting with e02c50ddea75da77052352eb0e2d64b4b77d227000222367b75baa14854182c7 not found: ID does not exist"
Sep 09 16:26:38.814836 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.814745    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2s7h5"]
Sep 09 16:26:38.817551 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:38.817525    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2s7h5"]
Sep 09 16:26:40.607254 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:40.607218    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="becb9765-a7de-48c2-b9ee-e83af6e6c395" path="/var/lib/kubelet/pods/becb9765-a7de-48c2-b9ee-e83af6e6c395/volumes"
Sep 09 16:26:49.801948 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:49.801911    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w5vbz"
Sep 09 16:26:49.844836 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:49.844794    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h4b8s"]
Sep 09 16:26:49.845104 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:49.845067    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h4b8s" podUID="29e2a722-d0e9-44ab-8554-d2874dc89b63" containerName="manager" containerID="cri-o://82345b25e6513b8bed34589e3a7d8317842f509295210b326a356ea1e09afd35" gracePeriod=10
Sep 09 16:26:50.099453 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:50.099423    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h4b8s"
Sep 09 16:26:50.175574 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:50.175535    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29e2a722-d0e9-44ab-8554-d2874dc89b63-extensions-socket-volume\") pod \"29e2a722-d0e9-44ab-8554-d2874dc89b63\" (UID: \"29e2a722-d0e9-44ab-8554-d2874dc89b63\") "
Sep 09 16:26:50.175776 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:50.175588    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m85w9\" (UniqueName: \"kubernetes.io/projected/29e2a722-d0e9-44ab-8554-d2874dc89b63-kube-api-access-m85w9\") pod \"29e2a722-d0e9-44ab-8554-d2874dc89b63\" (UID: \"29e2a722-d0e9-44ab-8554-d2874dc89b63\") "
Sep 09 16:26:50.175949 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:50.175925    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e2a722-d0e9-44ab-8554-d2874dc89b63-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "29e2a722-d0e9-44ab-8554-d2874dc89b63" (UID: "29e2a722-d0e9-44ab-8554-d2874dc89b63"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:26:50.177767 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:50.177732    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e2a722-d0e9-44ab-8554-d2874dc89b63-kube-api-access-m85w9" (OuterVolumeSpecName: "kube-api-access-m85w9") pod "29e2a722-d0e9-44ab-8554-d2874dc89b63" (UID: "29e2a722-d0e9-44ab-8554-d2874dc89b63"). InnerVolumeSpecName "kube-api-access-m85w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:26:50.276330 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:50.276246    2555 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29e2a722-d0e9-44ab-8554-d2874dc89b63-extensions-socket-volume\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:26:50.276330 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:50.276278    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m85w9\" (UniqueName: \"kubernetes.io/projected/29e2a722-d0e9-44ab-8554-d2874dc89b63-kube-api-access-m85w9\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:26:50.867530 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:50.867439    2555 generic.go:356] "Generic (PLEG): container finished" podID="29e2a722-d0e9-44ab-8554-d2874dc89b63" containerID="82345b25e6513b8bed34589e3a7d8317842f509295210b326a356ea1e09afd35" exitCode=0
Sep 09 16:26:50.867530 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:50.867483    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h4b8s"
Sep 09 16:26:50.867959 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:50.867523    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h4b8s" event={"ID":"29e2a722-d0e9-44ab-8554-d2874dc89b63","Type":"ContainerDied","Data":"82345b25e6513b8bed34589e3a7d8317842f509295210b326a356ea1e09afd35"}
Sep 09 16:26:50.867959 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:50.867576    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h4b8s" event={"ID":"29e2a722-d0e9-44ab-8554-d2874dc89b63","Type":"ContainerDied","Data":"01d498c2cb553b323b45d475458fb6962b7b8f5834aba90d7ecb2fd1c87c9d8a"}
Sep 09 16:26:50.867959 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:50.867603    2555 scope.go:117] "RemoveContainer" containerID="82345b25e6513b8bed34589e3a7d8317842f509295210b326a356ea1e09afd35"
Sep 09 16:26:50.877233 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:50.877211    2555 scope.go:117] "RemoveContainer" containerID="82345b25e6513b8bed34589e3a7d8317842f509295210b326a356ea1e09afd35"
Sep 09 16:26:50.877579 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:26:50.877554    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82345b25e6513b8bed34589e3a7d8317842f509295210b326a356ea1e09afd35\": container with ID starting with 82345b25e6513b8bed34589e3a7d8317842f509295210b326a356ea1e09afd35 not found: ID does not exist" containerID="82345b25e6513b8bed34589e3a7d8317842f509295210b326a356ea1e09afd35"
Sep 09 16:26:50.877671 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:50.877587    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82345b25e6513b8bed34589e3a7d8317842f509295210b326a356ea1e09afd35"} err="failed to get container status \"82345b25e6513b8bed34589e3a7d8317842f509295210b326a356ea1e09afd35\": rpc error: code = NotFound desc = could not find container \"82345b25e6513b8bed34589e3a7d8317842f509295210b326a356ea1e09afd35\": container with ID starting with 82345b25e6513b8bed34589e3a7d8317842f509295210b326a356ea1e09afd35 not found: ID does not exist"
Sep 09 16:26:50.884393 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:50.884358    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h4b8s"]
Sep 09 16:26:50.887309 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:50.887275    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h4b8s"]
Sep 09 16:26:52.608049 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:52.608009    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29e2a722-d0e9-44ab-8554-d2874dc89b63" path="/var/lib/kubelet/pods/29e2a722-d0e9-44ab-8554-d2874dc89b63/volumes"
Sep 09 16:26:54.129080 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.129028    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"]
Sep 09 16:26:54.129700 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.129422    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e2a722-d0e9-44ab-8554-d2874dc89b63" containerName="manager"
Sep 09 16:26:54.129700 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.129437    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e2a722-d0e9-44ab-8554-d2874dc89b63" containerName="manager"
Sep 09 16:26:54.129700 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.129449    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="becb9765-a7de-48c2-b9ee-e83af6e6c395" containerName="manager"
Sep 09 16:26:54.129700 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.129454    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="becb9765-a7de-48c2-b9ee-e83af6e6c395" containerName="manager"
Sep 09 16:26:54.129700 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.129532    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="29e2a722-d0e9-44ab-8554-d2874dc89b63" containerName="manager"
Sep 09 16:26:54.129700 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.129540    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="becb9765-a7de-48c2-b9ee-e83af6e6c395" containerName="manager"
Sep 09 16:26:54.132322 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.132270    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.134956 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.134928    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-ntb6r\""
Sep 09 16:26:54.134956 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.134942    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 16:26:54.143276 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.143247    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"]
Sep 09 16:26:54.209742 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.209693    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b512776f-22c4-40b8-89fd-c3cd475e6d09-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.209941 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.209771    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b512776f-22c4-40b8-89fd-c3cd475e6d09-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.209941 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.209797    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b512776f-22c4-40b8-89fd-c3cd475e6d09-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.209941 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.209813    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b512776f-22c4-40b8-89fd-c3cd475e6d09-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.209941 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.209844    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b512776f-22c4-40b8-89fd-c3cd475e6d09-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.209941 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.209890    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b512776f-22c4-40b8-89fd-c3cd475e6d09-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.209941 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.209921    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkjqj\" (UniqueName: \"kubernetes.io/projected/b512776f-22c4-40b8-89fd-c3cd475e6d09-kube-api-access-gkjqj\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.210164 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.209955    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b512776f-22c4-40b8-89fd-c3cd475e6d09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.210164 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.209985    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b512776f-22c4-40b8-89fd-c3cd475e6d09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.311496 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.311443    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b512776f-22c4-40b8-89fd-c3cd475e6d09-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.311496 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.311489    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b512776f-22c4-40b8-89fd-c3cd475e6d09-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.311773 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.311507    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b512776f-22c4-40b8-89fd-c3cd475e6d09-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.311773 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.311544    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b512776f-22c4-40b8-89fd-c3cd475e6d09-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.311773 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.311579    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b512776f-22c4-40b8-89fd-c3cd475e6d09-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.311773 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.311624    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkjqj\" (UniqueName: \"kubernetes.io/projected/b512776f-22c4-40b8-89fd-c3cd475e6d09-kube-api-access-gkjqj\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.311773 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.311669    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b512776f-22c4-40b8-89fd-c3cd475e6d09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.311773 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.311703    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b512776f-22c4-40b8-89fd-c3cd475e6d09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.311773 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.311751    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b512776f-22c4-40b8-89fd-c3cd475e6d09-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.312135 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.312019    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b512776f-22c4-40b8-89fd-c3cd475e6d09-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.312135 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.312021    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b512776f-22c4-40b8-89fd-c3cd475e6d09-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.312249 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.312216    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b512776f-22c4-40b8-89fd-c3cd475e6d09-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.312512 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.312491    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b512776f-22c4-40b8-89fd-c3cd475e6d09-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.312639 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.312618    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b512776f-22c4-40b8-89fd-c3cd475e6d09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.314250 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.314229    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b512776f-22c4-40b8-89fd-c3cd475e6d09-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.314380 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.314362    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b512776f-22c4-40b8-89fd-c3cd475e6d09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.319585 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.319551    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b512776f-22c4-40b8-89fd-c3cd475e6d09-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.319728 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.319706    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkjqj\" (UniqueName: \"kubernetes.io/projected/b512776f-22c4-40b8-89fd-c3cd475e6d09-kube-api-access-gkjqj\") pod \"maas-default-gateway-openshift-default-5ccb968c66-cnkj9\" (UID: \"b512776f-22c4-40b8-89fd-c3cd475e6d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.447027 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.446933    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:54.723320 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.723276    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"]
Sep 09 16:26:54.726406 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:26:54.726380    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb512776f_22c4_40b8_89fd_c3cd475e6d09.slice/crio-94e135364ca745519e8093c78af8b439983070b448d670e0a95a98128aa474c3 WatchSource:0}: Error finding container 94e135364ca745519e8093c78af8b439983070b448d670e0a95a98128aa474c3: Status 404 returned error can't find the container with id 94e135364ca745519e8093c78af8b439983070b448d670e0a95a98128aa474c3
Sep 09 16:26:54.894361 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:54.894320    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9" event={"ID":"b512776f-22c4-40b8-89fd-c3cd475e6d09","Type":"ContainerStarted","Data":"94e135364ca745519e8093c78af8b439983070b448d670e0a95a98128aa474c3"}
Sep 09 16:26:57.733520 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:57.733479    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:26:57.733817 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:57.733549    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:26:57.733817 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:57.733582    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:26:57.916267 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:57.916224    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9" event={"ID":"b512776f-22c4-40b8-89fd-c3cd475e6d09","Type":"ContainerStarted","Data":"c7eff200e2e7488dac14869f9579cd015da731fa3a054f13338eab4a93270610"}
Sep 09 16:26:57.938607 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:57.938535    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9" podStartSLOduration=0.933571488 podStartE2EDuration="3.938509472s" totalImagesPullingTime="3.004937984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:26:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:26:57.934273953 +0000 UTC m=+783.764482014" watchObservedRunningTime="2026-09-09 16:26:57.938509472 +0000 UTC m=+783.768717532"
Sep 09 16:26:58.447754 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:58.447716    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:58.452426 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:58.452398    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:58.922169 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:58.922133    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:26:58.923361 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:26:58.923341    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:27:08.638192 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:08.638151    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tk9gj"]
Sep 09 16:27:08.641716 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:08.641696    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-tk9gj"
Sep 09 16:27:08.644322 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:08.644283    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 16:27:08.644413 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:08.644283    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6rwjw\""
Sep 09 16:27:08.651451 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:08.650280    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tk9gj"]
Sep 09 16:27:08.738612 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:08.738573    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tk9gj"]
Sep 09 16:27:08.745588 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:08.745555    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/67ad5941-db88-4e93-b06b-1e661cc8d4a5-config-file\") pod \"limitador-limitador-8588749977-tk9gj\" (UID: \"67ad5941-db88-4e93-b06b-1e661cc8d4a5\") " pod="kuadrant-system/limitador-limitador-8588749977-tk9gj"
Sep 09 16:27:08.745749 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:08.745645    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-946dk\" (UniqueName: \"kubernetes.io/projected/67ad5941-db88-4e93-b06b-1e661cc8d4a5-kube-api-access-946dk\") pod \"limitador-limitador-8588749977-tk9gj\" (UID: \"67ad5941-db88-4e93-b06b-1e661cc8d4a5\") " pod="kuadrant-system/limitador-limitador-8588749977-tk9gj"
Sep 09 16:27:08.847105 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:08.847069    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-946dk\" (UniqueName: \"kubernetes.io/projected/67ad5941-db88-4e93-b06b-1e661cc8d4a5-kube-api-access-946dk\") pod \"limitador-limitador-8588749977-tk9gj\" (UID: \"67ad5941-db88-4e93-b06b-1e661cc8d4a5\") " pod="kuadrant-system/limitador-limitador-8588749977-tk9gj"
Sep 09 16:27:08.847303 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:08.847128    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/67ad5941-db88-4e93-b06b-1e661cc8d4a5-config-file\") pod \"limitador-limitador-8588749977-tk9gj\" (UID: \"67ad5941-db88-4e93-b06b-1e661cc8d4a5\") " pod="kuadrant-system/limitador-limitador-8588749977-tk9gj"
Sep 09 16:27:08.847768 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:08.847747    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/67ad5941-db88-4e93-b06b-1e661cc8d4a5-config-file\") pod \"limitador-limitador-8588749977-tk9gj\" (UID: \"67ad5941-db88-4e93-b06b-1e661cc8d4a5\") " pod="kuadrant-system/limitador-limitador-8588749977-tk9gj"
Sep 09 16:27:08.855847 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:08.855810    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-946dk\" (UniqueName: \"kubernetes.io/projected/67ad5941-db88-4e93-b06b-1e661cc8d4a5-kube-api-access-946dk\") pod \"limitador-limitador-8588749977-tk9gj\" (UID: \"67ad5941-db88-4e93-b06b-1e661cc8d4a5\") " pod="kuadrant-system/limitador-limitador-8588749977-tk9gj"
Sep 09 16:27:08.960170 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:08.960064    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-tk9gj"
Sep 09 16:27:09.182425 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:09.182381    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:27:09.186950 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:09.186923    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"
Sep 09 16:27:09.194391 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:09.194363    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:27:09.219868 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:09.219767    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:27:09.233411 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:09.233223    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tk9gj"]
Sep 09 16:27:09.235764 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:27:09.235725    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ad5941_db88_4e93_b06b_1e661cc8d4a5.slice/crio-7c973c1c4f72f3715c73b2d111dd89cb0f7e4b6ed35f3e48e4822b3738d37c44 WatchSource:0}: Error finding container 7c973c1c4f72f3715c73b2d111dd89cb0f7e4b6ed35f3e48e4822b3738d37c44: Status 404 returned error can't find the container with id 7c973c1c4f72f3715c73b2d111dd89cb0f7e4b6ed35f3e48e4822b3738d37c44
Sep 09 16:27:09.354430 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:09.354389    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wblvv\" (UniqueName: \"kubernetes.io/projected/40a70a72-34ff-4e7e-a553-63449eb5b379-kube-api-access-wblvv\") pod \"limitador-limitador-84994f7ffc-5lx6m\" (UID: \"40a70a72-34ff-4e7e-a553-63449eb5b379\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"
Sep 09 16:27:09.354612 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:09.354446    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40a70a72-34ff-4e7e-a553-63449eb5b379-config-file\") pod \"limitador-limitador-84994f7ffc-5lx6m\" (UID: \"40a70a72-34ff-4e7e-a553-63449eb5b379\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"
Sep 09 16:27:09.455810 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:09.455773    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wblvv\" (UniqueName: \"kubernetes.io/projected/40a70a72-34ff-4e7e-a553-63449eb5b379-kube-api-access-wblvv\") pod \"limitador-limitador-84994f7ffc-5lx6m\" (UID: \"40a70a72-34ff-4e7e-a553-63449eb5b379\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"
Sep 09 16:27:09.455978 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:09.455835    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40a70a72-34ff-4e7e-a553-63449eb5b379-config-file\") pod \"limitador-limitador-84994f7ffc-5lx6m\" (UID: \"40a70a72-34ff-4e7e-a553-63449eb5b379\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"
Sep 09 16:27:09.456497 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:09.456476    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40a70a72-34ff-4e7e-a553-63449eb5b379-config-file\") pod \"limitador-limitador-84994f7ffc-5lx6m\" (UID: \"40a70a72-34ff-4e7e-a553-63449eb5b379\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"
Sep 09 16:27:09.464084 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:09.464053    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wblvv\" (UniqueName: \"kubernetes.io/projected/40a70a72-34ff-4e7e-a553-63449eb5b379-kube-api-access-wblvv\") pod \"limitador-limitador-84994f7ffc-5lx6m\" (UID: \"40a70a72-34ff-4e7e-a553-63449eb5b379\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"
Sep 09 16:27:09.498889 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:09.498852    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"
Sep 09 16:27:09.536723 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:09.536684    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-52dq9"]
Sep 09 16:27:09.541566 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:09.541528    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-52dq9"
Sep 09 16:27:09.544384 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:09.544350    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-84zp8\""
Sep 09 16:27:09.547319 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:09.547277    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-52dq9"]
Sep 09 16:27:09.658320 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:09.658256    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7ngr\" (UniqueName: \"kubernetes.io/projected/f0f0c0a7-68e0-46bf-8bf5-1247ae2b87d1-kube-api-access-x7ngr\") pod \"authorino-59c974bd48-52dq9\" (UID: \"f0f0c0a7-68e0-46bf-8bf5-1247ae2b87d1\") " pod="kuadrant-system/authorino-59c974bd48-52dq9"
Sep 09 16:27:09.759513 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:09.759377    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ngr\" (UniqueName: \"kubernetes.io/projected/f0f0c0a7-68e0-46bf-8bf5-1247ae2b87d1-kube-api-access-x7ngr\") pod \"authorino-59c974bd48-52dq9\" (UID: \"f0f0c0a7-68e0-46bf-8bf5-1247ae2b87d1\") " pod="kuadrant-system/authorino-59c974bd48-52dq9"
Sep 09 16:27:09.770176 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:09.770149    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7ngr\" (UniqueName: \"kubernetes.io/projected/f0f0c0a7-68e0-46bf-8bf5-1247ae2b87d1-kube-api-access-x7ngr\") pod \"authorino-59c974bd48-52dq9\" (UID: \"f0f0c0a7-68e0-46bf-8bf5-1247ae2b87d1\") " pod="kuadrant-system/authorino-59c974bd48-52dq9"
Sep 09 16:27:09.777040 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:09.777010    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:27:09.778002 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:27:09.777978    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a70a72_34ff_4e7e_a553_63449eb5b379.slice/crio-d09d8bf4d1cfeba0e9a57f34db38d8a2d1036828eda875d81fbce33259d4c3e8 WatchSource:0}: Error finding container d09d8bf4d1cfeba0e9a57f34db38d8a2d1036828eda875d81fbce33259d4c3e8: Status 404 returned error can't find the container with id d09d8bf4d1cfeba0e9a57f34db38d8a2d1036828eda875d81fbce33259d4c3e8
Sep 09 16:27:09.854898 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:09.854854    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-52dq9"
Sep 09 16:27:09.983732 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:09.983694    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-tk9gj" event={"ID":"67ad5941-db88-4e93-b06b-1e661cc8d4a5","Type":"ContainerStarted","Data":"7c973c1c4f72f3715c73b2d111dd89cb0f7e4b6ed35f3e48e4822b3738d37c44"}
Sep 09 16:27:09.985022 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:09.984997    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-5lx6m" event={"ID":"40a70a72-34ff-4e7e-a553-63449eb5b379","Type":"ContainerStarted","Data":"d09d8bf4d1cfeba0e9a57f34db38d8a2d1036828eda875d81fbce33259d4c3e8"}
Sep 09 16:27:10.131328 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:10.131281    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-52dq9"]
Sep 09 16:27:10.132809 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:27:10.132781    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0f0c0a7_68e0_46bf_8bf5_1247ae2b87d1.slice/crio-2f8ae12c99ae793a24fd94d3b16407fc7347a62e456b2edb357d7794869b5a91 WatchSource:0}: Error finding container 2f8ae12c99ae793a24fd94d3b16407fc7347a62e456b2edb357d7794869b5a91: Status 404 returned error can't find the container with id 2f8ae12c99ae793a24fd94d3b16407fc7347a62e456b2edb357d7794869b5a91
Sep 09 16:27:10.993113 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:10.993070    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-tk9gj" event={"ID":"67ad5941-db88-4e93-b06b-1e661cc8d4a5","Type":"ContainerStarted","Data":"c7ce4db39175b18a7280ccb15716ac5175197fbbcd10d55f08d69a6a0fee7c5f"}
Sep 09 16:27:10.993572 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:10.993257    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-tk9gj"
Sep 09 16:27:10.994456 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:10.994428    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-52dq9" event={"ID":"f0f0c0a7-68e0-46bf-8bf5-1247ae2b87d1","Type":"ContainerStarted","Data":"2f8ae12c99ae793a24fd94d3b16407fc7347a62e456b2edb357d7794869b5a91"}
Sep 09 16:27:10.995944 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:10.995922    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-5lx6m" event={"ID":"40a70a72-34ff-4e7e-a553-63449eb5b379","Type":"ContainerStarted","Data":"9ff2ad8a40b732b88fc1ed39905af3d091554af009ebad2beba890be61e40e53"}
Sep 09 16:27:10.996038 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:10.995967    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"
Sep 09 16:27:11.023757 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:11.023697    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-5lx6m" podStartSLOduration=1.209399253 podStartE2EDuration="2.023665727s" totalImagesPullingTime="814.266474ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:27:11.022716205 +0000 UTC m=+796.852924265" watchObservedRunningTime="2026-09-09 16:27:11.023665727 +0000 UTC m=+796.853873786"
Sep 09 16:27:11.025126 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:11.025088    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-tk9gj" podStartSLOduration=1.670675967 podStartE2EDuration="3.025077667s" totalImagesPullingTime="1.3544017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:27:11.008561973 +0000 UTC m=+796.838770032" watchObservedRunningTime="2026-09-09 16:27:11.025077667 +0000 UTC m=+796.855285725"
Sep 09 16:27:13.011434 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:13.011391    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-52dq9" event={"ID":"f0f0c0a7-68e0-46bf-8bf5-1247ae2b87d1","Type":"ContainerStarted","Data":"3d3a49a1a9f3dbe4ca8f51667356ae9202c3451a4bed6fb4b8bac2442861758a"}
Sep 09 16:27:13.028483 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:13.028421    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-52dq9" podStartSLOduration=2.067371517 podStartE2EDuration="4.028403891s" totalImagesPullingTime="1.961032374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:27:13.026690619 +0000 UTC m=+798.856898679" watchObservedRunningTime="2026-09-09 16:27:13.028403891 +0000 UTC m=+798.858611949"
Sep 09 16:27:22.005447 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:22.005412    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-tk9gj"
Sep 09 16:27:22.005927 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:22.005488    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"
Sep 09 16:27:22.062351 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:22.062315    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tk9gj"]
Sep 09 16:27:22.062551 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:22.062524    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-tk9gj" podUID="67ad5941-db88-4e93-b06b-1e661cc8d4a5" containerName="limitador" containerID="cri-o://c7ce4db39175b18a7280ccb15716ac5175197fbbcd10d55f08d69a6a0fee7c5f" gracePeriod=30
Sep 09 16:27:22.618384 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:22.618352    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-tk9gj"
Sep 09 16:27:22.678510 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:22.678411    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-946dk\" (UniqueName: \"kubernetes.io/projected/67ad5941-db88-4e93-b06b-1e661cc8d4a5-kube-api-access-946dk\") pod \"67ad5941-db88-4e93-b06b-1e661cc8d4a5\" (UID: \"67ad5941-db88-4e93-b06b-1e661cc8d4a5\") "
Sep 09 16:27:22.678671 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:22.678513    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/67ad5941-db88-4e93-b06b-1e661cc8d4a5-config-file\") pod \"67ad5941-db88-4e93-b06b-1e661cc8d4a5\" (UID: \"67ad5941-db88-4e93-b06b-1e661cc8d4a5\") "
Sep 09 16:27:22.678866 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:22.678841    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67ad5941-db88-4e93-b06b-1e661cc8d4a5-config-file" (OuterVolumeSpecName: "config-file") pod "67ad5941-db88-4e93-b06b-1e661cc8d4a5" (UID: "67ad5941-db88-4e93-b06b-1e661cc8d4a5"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:27:22.680624 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:22.680603    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ad5941-db88-4e93-b06b-1e661cc8d4a5-kube-api-access-946dk" (OuterVolumeSpecName: "kube-api-access-946dk") pod "67ad5941-db88-4e93-b06b-1e661cc8d4a5" (UID: "67ad5941-db88-4e93-b06b-1e661cc8d4a5"). InnerVolumeSpecName "kube-api-access-946dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:27:22.779547 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:22.779500    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-946dk\" (UniqueName: \"kubernetes.io/projected/67ad5941-db88-4e93-b06b-1e661cc8d4a5-kube-api-access-946dk\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:27:22.779547 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:22.779536    2555 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/67ad5941-db88-4e93-b06b-1e661cc8d4a5-config-file\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:27:23.068066 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:23.068026    2555 generic.go:356] "Generic (PLEG): container finished" podID="67ad5941-db88-4e93-b06b-1e661cc8d4a5" containerID="c7ce4db39175b18a7280ccb15716ac5175197fbbcd10d55f08d69a6a0fee7c5f" exitCode=0
Sep 09 16:27:23.068582 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:23.068078    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-tk9gj"
Sep 09 16:27:23.068582 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:23.068105    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-tk9gj" event={"ID":"67ad5941-db88-4e93-b06b-1e661cc8d4a5","Type":"ContainerDied","Data":"c7ce4db39175b18a7280ccb15716ac5175197fbbcd10d55f08d69a6a0fee7c5f"}
Sep 09 16:27:23.068582 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:23.068157    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-tk9gj" event={"ID":"67ad5941-db88-4e93-b06b-1e661cc8d4a5","Type":"ContainerDied","Data":"7c973c1c4f72f3715c73b2d111dd89cb0f7e4b6ed35f3e48e4822b3738d37c44"}
Sep 09 16:27:23.068582 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:23.068175    2555 scope.go:117] "RemoveContainer" containerID="c7ce4db39175b18a7280ccb15716ac5175197fbbcd10d55f08d69a6a0fee7c5f"
Sep 09 16:27:23.079567 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:23.079539    2555 scope.go:117] "RemoveContainer" containerID="c7ce4db39175b18a7280ccb15716ac5175197fbbcd10d55f08d69a6a0fee7c5f"
Sep 09 16:27:23.079920 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:27:23.079899    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7ce4db39175b18a7280ccb15716ac5175197fbbcd10d55f08d69a6a0fee7c5f\": container with ID starting with c7ce4db39175b18a7280ccb15716ac5175197fbbcd10d55f08d69a6a0fee7c5f not found: ID does not exist" containerID="c7ce4db39175b18a7280ccb15716ac5175197fbbcd10d55f08d69a6a0fee7c5f"
Sep 09 16:27:23.079997 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:23.079932    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7ce4db39175b18a7280ccb15716ac5175197fbbcd10d55f08d69a6a0fee7c5f"} err="failed to get container status \"c7ce4db39175b18a7280ccb15716ac5175197fbbcd10d55f08d69a6a0fee7c5f\": rpc error: code = NotFound desc = could not find container \"c7ce4db39175b18a7280ccb15716ac5175197fbbcd10d55f08d69a6a0fee7c5f\": container with ID starting with c7ce4db39175b18a7280ccb15716ac5175197fbbcd10d55f08d69a6a0fee7c5f not found: ID does not exist"
Sep 09 16:27:23.091452 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:23.091415    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tk9gj"]
Sep 09 16:27:23.099674 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:23.094926    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tk9gj"]
Sep 09 16:27:23.111861 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:27:23.111813    2555 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ad5941_db88_4e93_b06b_1e661cc8d4a5.slice\": RecentStats: unable to find data in memory cache]"
Sep 09 16:27:24.607778 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:24.607741    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67ad5941-db88-4e93-b06b-1e661cc8d4a5" path="/var/lib/kubelet/pods/67ad5941-db88-4e93-b06b-1e661cc8d4a5/volumes"
Sep 09 16:27:38.023736 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.023689    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-dfc55"]
Sep 09 16:27:38.024250 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.024141    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67ad5941-db88-4e93-b06b-1e661cc8d4a5" containerName="limitador"
Sep 09 16:27:38.024250 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.024158    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="67ad5941-db88-4e93-b06b-1e661cc8d4a5" containerName="limitador"
Sep 09 16:27:38.024250 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.024223    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="67ad5941-db88-4e93-b06b-1e661cc8d4a5" containerName="limitador"
Sep 09 16:27:38.032691 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.032662    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dfc55"
Sep 09 16:27:38.033592 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.033563    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-dfc55"]
Sep 09 16:27:38.115474 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.115427    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7wsh\" (UniqueName: \"kubernetes.io/projected/79130403-08ca-41f6-890a-14a17a5b5551-kube-api-access-r7wsh\") pod \"authorino-557f876f69-dfc55\" (UID: \"79130403-08ca-41f6-890a-14a17a5b5551\") " pod="kuadrant-system/authorino-557f876f69-dfc55"
Sep 09 16:27:38.142747 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.142707    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-dfc55"]
Sep 09 16:27:38.143006 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:27:38.142984    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-r7wsh], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-dfc55" podUID="79130403-08ca-41f6-890a-14a17a5b5551"
Sep 09 16:27:38.156205 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.156170    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dfc55"
Sep 09 16:27:38.163946 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.163913    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dfc55"
Sep 09 16:27:38.169142 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.169110    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-t8h4x"]
Sep 09 16:27:38.172858 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.172831    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-t8h4x"
Sep 09 16:27:38.175224 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.175195    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 16:27:38.179691 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.179662    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-t8h4x"]
Sep 09 16:27:38.216029 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.215994    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7wsh\" (UniqueName: \"kubernetes.io/projected/79130403-08ca-41f6-890a-14a17a5b5551-kube-api-access-r7wsh\") pod \"authorino-557f876f69-dfc55\" (UID: \"79130403-08ca-41f6-890a-14a17a5b5551\") " pod="kuadrant-system/authorino-557f876f69-dfc55"
Sep 09 16:27:38.224614 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.224583    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7wsh\" (UniqueName: \"kubernetes.io/projected/79130403-08ca-41f6-890a-14a17a5b5551-kube-api-access-r7wsh\") pod \"authorino-557f876f69-dfc55\" (UID: \"79130403-08ca-41f6-890a-14a17a5b5551\") " pod="kuadrant-system/authorino-557f876f69-dfc55"
Sep 09 16:27:38.302734 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.302644    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-t8h4x"]
Sep 09 16:27:38.302939 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:27:38.302914    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fbqpn tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-t8h4x" podUID="8fbae2bf-6c44-41b4-bceb-82b566566a46"
Sep 09 16:27:38.317461 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.317405    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7wsh\" (UniqueName: \"kubernetes.io/projected/79130403-08ca-41f6-890a-14a17a5b5551-kube-api-access-r7wsh\") pod \"79130403-08ca-41f6-890a-14a17a5b5551\" (UID: \"79130403-08ca-41f6-890a-14a17a5b5551\") "
Sep 09 16:27:38.317671 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.317643    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbqpn\" (UniqueName: \"kubernetes.io/projected/8fbae2bf-6c44-41b4-bceb-82b566566a46-kube-api-access-fbqpn\") pod \"authorino-6d4dfc649f-t8h4x\" (UID: \"8fbae2bf-6c44-41b4-bceb-82b566566a46\") " pod="kuadrant-system/authorino-6d4dfc649f-t8h4x"
Sep 09 16:27:38.317761 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.317682    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8fbae2bf-6c44-41b4-bceb-82b566566a46-tls-cert\") pod \"authorino-6d4dfc649f-t8h4x\" (UID: \"8fbae2bf-6c44-41b4-bceb-82b566566a46\") " pod="kuadrant-system/authorino-6d4dfc649f-t8h4x"
Sep 09 16:27:38.319704 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.319675    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79130403-08ca-41f6-890a-14a17a5b5551-kube-api-access-r7wsh" (OuterVolumeSpecName: "kube-api-access-r7wsh") pod "79130403-08ca-41f6-890a-14a17a5b5551" (UID: "79130403-08ca-41f6-890a-14a17a5b5551"). InnerVolumeSpecName "kube-api-access-r7wsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:27:38.332146 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.332117    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6b554cb7f9-ptjq9"]
Sep 09 16:27:38.336014 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.335992    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b554cb7f9-ptjq9"
Sep 09 16:27:38.343647 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.343616    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b554cb7f9-ptjq9"]
Sep 09 16:27:38.418647 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.418605    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49p8r\" (UniqueName: \"kubernetes.io/projected/1ba52872-5a39-45e9-bd34-e6bfb9caca13-kube-api-access-49p8r\") pod \"authorino-6b554cb7f9-ptjq9\" (UID: \"1ba52872-5a39-45e9-bd34-e6bfb9caca13\") " pod="kuadrant-system/authorino-6b554cb7f9-ptjq9"
Sep 09 16:27:38.418845 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.418668    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbqpn\" (UniqueName: \"kubernetes.io/projected/8fbae2bf-6c44-41b4-bceb-82b566566a46-kube-api-access-fbqpn\") pod \"authorino-6d4dfc649f-t8h4x\" (UID: \"8fbae2bf-6c44-41b4-bceb-82b566566a46\") " pod="kuadrant-system/authorino-6d4dfc649f-t8h4x"
Sep 09 16:27:38.418845 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.418757    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8fbae2bf-6c44-41b4-bceb-82b566566a46-tls-cert\") pod \"authorino-6d4dfc649f-t8h4x\" (UID: \"8fbae2bf-6c44-41b4-bceb-82b566566a46\") " pod="kuadrant-system/authorino-6d4dfc649f-t8h4x"
Sep 09 16:27:38.418845 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.418803    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1ba52872-5a39-45e9-bd34-e6bfb9caca13-tls-cert\") pod \"authorino-6b554cb7f9-ptjq9\" (UID: \"1ba52872-5a39-45e9-bd34-e6bfb9caca13\") " pod="kuadrant-system/authorino-6b554cb7f9-ptjq9"
Sep 09 16:27:38.419041 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.418910    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7wsh\" (UniqueName: \"kubernetes.io/projected/79130403-08ca-41f6-890a-14a17a5b5551-kube-api-access-r7wsh\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:27:38.421183 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.421165    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8fbae2bf-6c44-41b4-bceb-82b566566a46-tls-cert\") pod \"authorino-6d4dfc649f-t8h4x\" (UID: \"8fbae2bf-6c44-41b4-bceb-82b566566a46\") " pod="kuadrant-system/authorino-6d4dfc649f-t8h4x"
Sep 09 16:27:38.426164 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.426135    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbqpn\" (UniqueName: \"kubernetes.io/projected/8fbae2bf-6c44-41b4-bceb-82b566566a46-kube-api-access-fbqpn\") pod \"authorino-6d4dfc649f-t8h4x\" (UID: \"8fbae2bf-6c44-41b4-bceb-82b566566a46\") " pod="kuadrant-system/authorino-6d4dfc649f-t8h4x"
Sep 09 16:27:38.519678 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.519630    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1ba52872-5a39-45e9-bd34-e6bfb9caca13-tls-cert\") pod \"authorino-6b554cb7f9-ptjq9\" (UID: \"1ba52872-5a39-45e9-bd34-e6bfb9caca13\") " pod="kuadrant-system/authorino-6b554cb7f9-ptjq9"
Sep 09 16:27:38.519858 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.519701    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49p8r\" (UniqueName: \"kubernetes.io/projected/1ba52872-5a39-45e9-bd34-e6bfb9caca13-kube-api-access-49p8r\") pod \"authorino-6b554cb7f9-ptjq9\" (UID: \"1ba52872-5a39-45e9-bd34-e6bfb9caca13\") " pod="kuadrant-system/authorino-6b554cb7f9-ptjq9"
Sep 09 16:27:38.522109 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.522083    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1ba52872-5a39-45e9-bd34-e6bfb9caca13-tls-cert\") pod \"authorino-6b554cb7f9-ptjq9\" (UID: \"1ba52872-5a39-45e9-bd34-e6bfb9caca13\") " pod="kuadrant-system/authorino-6b554cb7f9-ptjq9"
Sep 09 16:27:38.530979 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.530952    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49p8r\" (UniqueName: \"kubernetes.io/projected/1ba52872-5a39-45e9-bd34-e6bfb9caca13-kube-api-access-49p8r\") pod \"authorino-6b554cb7f9-ptjq9\" (UID: \"1ba52872-5a39-45e9-bd34-e6bfb9caca13\") " pod="kuadrant-system/authorino-6b554cb7f9-ptjq9"
Sep 09 16:27:38.646459 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.646417    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b554cb7f9-ptjq9"
Sep 09 16:27:38.918779 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:38.918724    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b554cb7f9-ptjq9"]
Sep 09 16:27:38.921498 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:27:38.921464    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ba52872_5a39_45e9_bd34_e6bfb9caca13.slice/crio-e86e17837f6d085ad6ab4d9523b45a4c704d701a2e6e4e63843e8ad003b51f15 WatchSource:0}: Error finding container e86e17837f6d085ad6ab4d9523b45a4c704d701a2e6e4e63843e8ad003b51f15: Status 404 returned error can't find the container with id e86e17837f6d085ad6ab4d9523b45a4c704d701a2e6e4e63843e8ad003b51f15
Sep 09 16:27:39.163177 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:39.163136    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b554cb7f9-ptjq9" event={"ID":"1ba52872-5a39-45e9-bd34-e6bfb9caca13","Type":"ContainerStarted","Data":"e86e17837f6d085ad6ab4d9523b45a4c704d701a2e6e4e63843e8ad003b51f15"}
Sep 09 16:27:39.163677 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:39.163211    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dfc55"
Sep 09 16:27:39.163677 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:39.163224    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-t8h4x"
Sep 09 16:27:39.172910 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:39.172827    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-t8h4x"
Sep 09 16:27:39.203191 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:39.203156    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-dfc55"]
Sep 09 16:27:39.205360 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:39.205338    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-dfc55"]
Sep 09 16:27:39.327073 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:39.327028    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8fbae2bf-6c44-41b4-bceb-82b566566a46-tls-cert\") pod \"8fbae2bf-6c44-41b4-bceb-82b566566a46\" (UID: \"8fbae2bf-6c44-41b4-bceb-82b566566a46\") "
Sep 09 16:27:39.327283 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:39.327134    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbqpn\" (UniqueName: \"kubernetes.io/projected/8fbae2bf-6c44-41b4-bceb-82b566566a46-kube-api-access-fbqpn\") pod \"8fbae2bf-6c44-41b4-bceb-82b566566a46\" (UID: \"8fbae2bf-6c44-41b4-bceb-82b566566a46\") "
Sep 09 16:27:39.329168 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:39.329135    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fbae2bf-6c44-41b4-bceb-82b566566a46-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "8fbae2bf-6c44-41b4-bceb-82b566566a46" (UID: "8fbae2bf-6c44-41b4-bceb-82b566566a46"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:27:39.329263 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:39.329231    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbae2bf-6c44-41b4-bceb-82b566566a46-kube-api-access-fbqpn" (OuterVolumeSpecName: "kube-api-access-fbqpn") pod "8fbae2bf-6c44-41b4-bceb-82b566566a46" (UID: "8fbae2bf-6c44-41b4-bceb-82b566566a46"). InnerVolumeSpecName "kube-api-access-fbqpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:27:39.428165 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:39.428131    2555 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8fbae2bf-6c44-41b4-bceb-82b566566a46-tls-cert\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:27:39.428165 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:39.428161    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbqpn\" (UniqueName: \"kubernetes.io/projected/8fbae2bf-6c44-41b4-bceb-82b566566a46-kube-api-access-fbqpn\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:27:40.169849 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:40.169808    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b554cb7f9-ptjq9" event={"ID":"1ba52872-5a39-45e9-bd34-e6bfb9caca13","Type":"ContainerStarted","Data":"625aacd76f9e3f3a0f957806bceca6992b7491d376bd9accb9e923fe165a1c5b"}
Sep 09 16:27:40.170252 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:40.169884    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-t8h4x"
Sep 09 16:27:40.186421 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:40.186368    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6b554cb7f9-ptjq9" podStartSLOduration=1.7252886429999998 podStartE2EDuration="2.186350967s" totalImagesPullingTime="461.062324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:27:40.184792622 +0000 UTC m=+826.015000680" watchObservedRunningTime="2026-09-09 16:27:40.186350967 +0000 UTC m=+826.016559027"
Sep 09 16:27:40.210408 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:40.210371    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-t8h4x"]
Sep 09 16:27:40.217350 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:40.217314    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-52dq9"]
Sep 09 16:27:40.217546 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:40.217522    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-52dq9" podUID="f0f0c0a7-68e0-46bf-8bf5-1247ae2b87d1" containerName="authorino" containerID="cri-o://3d3a49a1a9f3dbe4ca8f51667356ae9202c3451a4bed6fb4b8bac2442861758a" gracePeriod=30
Sep 09 16:27:40.219982 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:40.219953    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-t8h4x"]
Sep 09 16:27:40.465199 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:40.465162    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-52dq9"
Sep 09 16:27:40.538411 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:40.538375    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7ngr\" (UniqueName: \"kubernetes.io/projected/f0f0c0a7-68e0-46bf-8bf5-1247ae2b87d1-kube-api-access-x7ngr\") pod \"f0f0c0a7-68e0-46bf-8bf5-1247ae2b87d1\" (UID: \"f0f0c0a7-68e0-46bf-8bf5-1247ae2b87d1\") "
Sep 09 16:27:40.540485 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:40.540452    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f0c0a7-68e0-46bf-8bf5-1247ae2b87d1-kube-api-access-x7ngr" (OuterVolumeSpecName: "kube-api-access-x7ngr") pod "f0f0c0a7-68e0-46bf-8bf5-1247ae2b87d1" (UID: "f0f0c0a7-68e0-46bf-8bf5-1247ae2b87d1"). InnerVolumeSpecName "kube-api-access-x7ngr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:27:40.607876 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:40.607843    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79130403-08ca-41f6-890a-14a17a5b5551" path="/var/lib/kubelet/pods/79130403-08ca-41f6-890a-14a17a5b5551/volumes"
Sep 09 16:27:40.608123 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:40.608110    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fbae2bf-6c44-41b4-bceb-82b566566a46" path="/var/lib/kubelet/pods/8fbae2bf-6c44-41b4-bceb-82b566566a46/volumes"
Sep 09 16:27:40.639714 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:40.639671    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7ngr\" (UniqueName: \"kubernetes.io/projected/f0f0c0a7-68e0-46bf-8bf5-1247ae2b87d1-kube-api-access-x7ngr\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:27:41.176054 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:41.176020    2555 generic.go:356] "Generic (PLEG): container finished" podID="f0f0c0a7-68e0-46bf-8bf5-1247ae2b87d1" containerID="3d3a49a1a9f3dbe4ca8f51667356ae9202c3451a4bed6fb4b8bac2442861758a" exitCode=0
Sep 09 16:27:41.176582 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:41.176069    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-52dq9"
Sep 09 16:27:41.176582 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:41.176111    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-52dq9" event={"ID":"f0f0c0a7-68e0-46bf-8bf5-1247ae2b87d1","Type":"ContainerDied","Data":"3d3a49a1a9f3dbe4ca8f51667356ae9202c3451a4bed6fb4b8bac2442861758a"}
Sep 09 16:27:41.176582 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:41.176166    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-52dq9" event={"ID":"f0f0c0a7-68e0-46bf-8bf5-1247ae2b87d1","Type":"ContainerDied","Data":"2f8ae12c99ae793a24fd94d3b16407fc7347a62e456b2edb357d7794869b5a91"}
Sep 09 16:27:41.176582 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:41.176189    2555 scope.go:117] "RemoveContainer" containerID="3d3a49a1a9f3dbe4ca8f51667356ae9202c3451a4bed6fb4b8bac2442861758a"
Sep 09 16:27:41.185272 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:41.185248    2555 scope.go:117] "RemoveContainer" containerID="3d3a49a1a9f3dbe4ca8f51667356ae9202c3451a4bed6fb4b8bac2442861758a"
Sep 09 16:27:41.185622 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:27:41.185600    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d3a49a1a9f3dbe4ca8f51667356ae9202c3451a4bed6fb4b8bac2442861758a\": container with ID starting with 3d3a49a1a9f3dbe4ca8f51667356ae9202c3451a4bed6fb4b8bac2442861758a not found: ID does not exist" containerID="3d3a49a1a9f3dbe4ca8f51667356ae9202c3451a4bed6fb4b8bac2442861758a"
Sep 09 16:27:41.185692 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:41.185632    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d3a49a1a9f3dbe4ca8f51667356ae9202c3451a4bed6fb4b8bac2442861758a"} err="failed to get container status \"3d3a49a1a9f3dbe4ca8f51667356ae9202c3451a4bed6fb4b8bac2442861758a\": rpc error: code = NotFound desc = could not find container \"3d3a49a1a9f3dbe4ca8f51667356ae9202c3451a4bed6fb4b8bac2442861758a\": container with ID starting with 3d3a49a1a9f3dbe4ca8f51667356ae9202c3451a4bed6fb4b8bac2442861758a not found: ID does not exist"
Sep 09 16:27:41.192304 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:41.192262    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-52dq9"]
Sep 09 16:27:41.196007 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:41.195982    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-52dq9"]
Sep 09 16:27:42.607697 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:42.607660    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0f0c0a7-68e0-46bf-8bf5-1247ae2b87d1" path="/var/lib/kubelet/pods/f0f0c0a7-68e0-46bf-8bf5-1247ae2b87d1/volumes"
Sep 09 16:27:45.460409 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:45.460371    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67549fc748-cbbzp"]
Sep 09 16:27:45.460871 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:45.460856    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0f0c0a7-68e0-46bf-8bf5-1247ae2b87d1" containerName="authorino"
Sep 09 16:27:45.460914 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:45.460873    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0f0c0a7-68e0-46bf-8bf5-1247ae2b87d1" containerName="authorino"
Sep 09 16:27:45.460963 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:45.460953    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0f0c0a7-68e0-46bf-8bf5-1247ae2b87d1" containerName="authorino"
Sep 09 16:27:45.463999 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:45.463978    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67549fc748-cbbzp"
Sep 09 16:27:45.467040 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:45.467015    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 16:27:45.467163 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:45.467035    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 16:27:45.468051 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:45.468032    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nm5jk\""
Sep 09 16:27:45.468141 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:45.468034    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 16:27:45.474162 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:45.474130    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67549fc748-cbbzp"]
Sep 09 16:27:45.582710 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:45.582675    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d3fe335-933e-4f01-85d4-00ca07ac0d99-metrics-certs\") pod \"maas-controller-67549fc748-cbbzp\" (UID: \"6d3fe335-933e-4f01-85d4-00ca07ac0d99\") " pod="opendatahub/maas-controller-67549fc748-cbbzp"
Sep 09 16:27:45.582906 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:45.582733    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvf48\" (UniqueName: \"kubernetes.io/projected/6d3fe335-933e-4f01-85d4-00ca07ac0d99-kube-api-access-mvf48\") pod \"maas-controller-67549fc748-cbbzp\" (UID: \"6d3fe335-933e-4f01-85d4-00ca07ac0d99\") " pod="opendatahub/maas-controller-67549fc748-cbbzp"
Sep 09 16:27:45.582906 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:45.582825    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d3fe335-933e-4f01-85d4-00ca07ac0d99-webhook-cert\") pod \"maas-controller-67549fc748-cbbzp\" (UID: \"6d3fe335-933e-4f01-85d4-00ca07ac0d99\") " pod="opendatahub/maas-controller-67549fc748-cbbzp"
Sep 09 16:27:45.683896 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:45.683852    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d3fe335-933e-4f01-85d4-00ca07ac0d99-webhook-cert\") pod \"maas-controller-67549fc748-cbbzp\" (UID: \"6d3fe335-933e-4f01-85d4-00ca07ac0d99\") " pod="opendatahub/maas-controller-67549fc748-cbbzp"
Sep 09 16:27:45.684086 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:45.683921    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d3fe335-933e-4f01-85d4-00ca07ac0d99-metrics-certs\") pod \"maas-controller-67549fc748-cbbzp\" (UID: \"6d3fe335-933e-4f01-85d4-00ca07ac0d99\") " pod="opendatahub/maas-controller-67549fc748-cbbzp"
Sep 09 16:27:45.684086 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:45.683961    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvf48\" (UniqueName: \"kubernetes.io/projected/6d3fe335-933e-4f01-85d4-00ca07ac0d99-kube-api-access-mvf48\") pod \"maas-controller-67549fc748-cbbzp\" (UID: \"6d3fe335-933e-4f01-85d4-00ca07ac0d99\") " pod="opendatahub/maas-controller-67549fc748-cbbzp"
Sep 09 16:27:45.686598 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:45.686568    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d3fe335-933e-4f01-85d4-00ca07ac0d99-metrics-certs\") pod \"maas-controller-67549fc748-cbbzp\" (UID: \"6d3fe335-933e-4f01-85d4-00ca07ac0d99\") " pod="opendatahub/maas-controller-67549fc748-cbbzp"
Sep 09 16:27:45.686715 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:45.686683    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d3fe335-933e-4f01-85d4-00ca07ac0d99-webhook-cert\") pod \"maas-controller-67549fc748-cbbzp\" (UID: \"6d3fe335-933e-4f01-85d4-00ca07ac0d99\") " pod="opendatahub/maas-controller-67549fc748-cbbzp"
Sep 09 16:27:45.691960 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:45.691930    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvf48\" (UniqueName: \"kubernetes.io/projected/6d3fe335-933e-4f01-85d4-00ca07ac0d99-kube-api-access-mvf48\") pod \"maas-controller-67549fc748-cbbzp\" (UID: \"6d3fe335-933e-4f01-85d4-00ca07ac0d99\") " pod="opendatahub/maas-controller-67549fc748-cbbzp"
Sep 09 16:27:45.776069 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:45.776017    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67549fc748-cbbzp"
Sep 09 16:27:46.048417 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:46.048381    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67549fc748-cbbzp"]
Sep 09 16:27:46.049932 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:27:46.049901    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3fe335_933e_4f01_85d4_00ca07ac0d99.slice/crio-1c4336924a5591582343a80dfc3a20f556995c64936220b5211d1c8d1bd753a5 WatchSource:0}: Error finding container 1c4336924a5591582343a80dfc3a20f556995c64936220b5211d1c8d1bd753a5: Status 404 returned error can't find the container with id 1c4336924a5591582343a80dfc3a20f556995c64936220b5211d1c8d1bd753a5
Sep 09 16:27:46.212547 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:46.212501    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67549fc748-cbbzp" event={"ID":"6d3fe335-933e-4f01-85d4-00ca07ac0d99","Type":"ContainerStarted","Data":"1c4336924a5591582343a80dfc3a20f556995c64936220b5211d1c8d1bd753a5"}
Sep 09 16:27:49.233595 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:49.233552    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67549fc748-cbbzp" event={"ID":"6d3fe335-933e-4f01-85d4-00ca07ac0d99","Type":"ContainerStarted","Data":"b0007f168a13cced33434eba3ff7f6b0c064ee39e2a5a6097d29f6334f09ab57"}
Sep 09 16:27:49.234066 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:49.233621    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67549fc748-cbbzp"
Sep 09 16:27:49.252274 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:27:49.252201    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-67549fc748-cbbzp" podStartSLOduration=1.899808084 podStartE2EDuration="4.252178251s" totalImagesPullingTime="2.352370167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:27:49.250393027 +0000 UTC m=+835.080601085" watchObservedRunningTime="2026-09-09 16:27:49.252178251 +0000 UTC m=+835.082386332"
Sep 09 16:28:00.241171 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:00.241136    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67549fc748-cbbzp"
Sep 09 16:28:00.566326 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:00.566211    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67549fc748-cbbzp"]
Sep 09 16:28:00.566498 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:00.566445    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-67549fc748-cbbzp" podUID="6d3fe335-933e-4f01-85d4-00ca07ac0d99" containerName="manager" containerID="cri-o://b0007f168a13cced33434eba3ff7f6b0c064ee39e2a5a6097d29f6334f09ab57" gracePeriod=30
Sep 09 16:28:00.807209 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:00.807181    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67549fc748-cbbzp"
Sep 09 16:28:00.822169 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:00.822101    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d3fe335-933e-4f01-85d4-00ca07ac0d99-webhook-cert\") pod \"6d3fe335-933e-4f01-85d4-00ca07ac0d99\" (UID: \"6d3fe335-933e-4f01-85d4-00ca07ac0d99\") "
Sep 09 16:28:00.822169 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:00.822168    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d3fe335-933e-4f01-85d4-00ca07ac0d99-metrics-certs\") pod \"6d3fe335-933e-4f01-85d4-00ca07ac0d99\" (UID: \"6d3fe335-933e-4f01-85d4-00ca07ac0d99\") "
Sep 09 16:28:00.822362 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:00.822193    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvf48\" (UniqueName: \"kubernetes.io/projected/6d3fe335-933e-4f01-85d4-00ca07ac0d99-kube-api-access-mvf48\") pod \"6d3fe335-933e-4f01-85d4-00ca07ac0d99\" (UID: \"6d3fe335-933e-4f01-85d4-00ca07ac0d99\") "
Sep 09 16:28:00.824963 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:00.824920    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d3fe335-933e-4f01-85d4-00ca07ac0d99-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6d3fe335-933e-4f01-85d4-00ca07ac0d99" (UID: "6d3fe335-933e-4f01-85d4-00ca07ac0d99"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:28:00.825078 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:00.825023    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d3fe335-933e-4f01-85d4-00ca07ac0d99-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6d3fe335-933e-4f01-85d4-00ca07ac0d99" (UID: "6d3fe335-933e-4f01-85d4-00ca07ac0d99"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:28:00.825240 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:00.825201    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3fe335-933e-4f01-85d4-00ca07ac0d99-kube-api-access-mvf48" (OuterVolumeSpecName: "kube-api-access-mvf48") pod "6d3fe335-933e-4f01-85d4-00ca07ac0d99" (UID: "6d3fe335-933e-4f01-85d4-00ca07ac0d99"). InnerVolumeSpecName "kube-api-access-mvf48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:28:00.923427 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:00.923377    2555 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d3fe335-933e-4f01-85d4-00ca07ac0d99-webhook-cert\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:28:00.923427 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:00.923422    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d3fe335-933e-4f01-85d4-00ca07ac0d99-metrics-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:28:00.923427 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:00.923432    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvf48\" (UniqueName: \"kubernetes.io/projected/6d3fe335-933e-4f01-85d4-00ca07ac0d99-kube-api-access-mvf48\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:28:01.303753 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:01.303713    2555 generic.go:356] "Generic (PLEG): container finished" podID="6d3fe335-933e-4f01-85d4-00ca07ac0d99" containerID="b0007f168a13cced33434eba3ff7f6b0c064ee39e2a5a6097d29f6334f09ab57" exitCode=0
Sep 09 16:28:01.303753 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:01.303754    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67549fc748-cbbzp"
Sep 09 16:28:01.304240 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:01.303791    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67549fc748-cbbzp" event={"ID":"6d3fe335-933e-4f01-85d4-00ca07ac0d99","Type":"ContainerDied","Data":"b0007f168a13cced33434eba3ff7f6b0c064ee39e2a5a6097d29f6334f09ab57"}
Sep 09 16:28:01.304240 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:01.303840    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67549fc748-cbbzp" event={"ID":"6d3fe335-933e-4f01-85d4-00ca07ac0d99","Type":"ContainerDied","Data":"1c4336924a5591582343a80dfc3a20f556995c64936220b5211d1c8d1bd753a5"}
Sep 09 16:28:01.304240 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:01.303869    2555 scope.go:117] "RemoveContainer" containerID="b0007f168a13cced33434eba3ff7f6b0c064ee39e2a5a6097d29f6334f09ab57"
Sep 09 16:28:01.323553 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:01.323533    2555 scope.go:117] "RemoveContainer" containerID="b0007f168a13cced33434eba3ff7f6b0c064ee39e2a5a6097d29f6334f09ab57"
Sep 09 16:28:01.323824 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:28:01.323805    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0007f168a13cced33434eba3ff7f6b0c064ee39e2a5a6097d29f6334f09ab57\": container with ID starting with b0007f168a13cced33434eba3ff7f6b0c064ee39e2a5a6097d29f6334f09ab57 not found: ID does not exist" containerID="b0007f168a13cced33434eba3ff7f6b0c064ee39e2a5a6097d29f6334f09ab57"
Sep 09 16:28:01.323888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:01.323830    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0007f168a13cced33434eba3ff7f6b0c064ee39e2a5a6097d29f6334f09ab57"} err="failed to get container status \"b0007f168a13cced33434eba3ff7f6b0c064ee39e2a5a6097d29f6334f09ab57\": rpc error: code = NotFound desc = could not find container \"b0007f168a13cced33434eba3ff7f6b0c064ee39e2a5a6097d29f6334f09ab57\": container with ID starting with b0007f168a13cced33434eba3ff7f6b0c064ee39e2a5a6097d29f6334f09ab57 not found: ID does not exist"
Sep 09 16:28:01.332491 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:01.332464    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67549fc748-cbbzp"]
Sep 09 16:28:01.336259 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:01.336238    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-67549fc748-cbbzp"]
Sep 09 16:28:02.608159 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:02.608124    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d3fe335-933e-4f01-85d4-00ca07ac0d99" path="/var/lib/kubelet/pods/6d3fe335-933e-4f01-85d4-00ca07ac0d99/volumes"
Sep 09 16:28:19.924768 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:19.924735    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"]
Sep 09 16:28:19.925188 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:19.925050    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d3fe335-933e-4f01-85d4-00ca07ac0d99" containerName="manager"
Sep 09 16:28:19.925188 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:19.925062    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d3fe335-933e-4f01-85d4-00ca07ac0d99" containerName="manager"
Sep 09 16:28:19.925188 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:19.925134    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d3fe335-933e-4f01-85d4-00ca07ac0d99" containerName="manager"
Sep 09 16:28:19.929408 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:19.929385    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"
Sep 09 16:28:19.932004 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:19.931985    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 09 16:28:19.933392 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:19.933362    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:28:19.933516 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:19.933362    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 16:28:19.933595 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:19.933371    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k96v8\""
Sep 09 16:28:19.933667 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:19.933419    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 16:28:19.941441 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:19.941420    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"]
Sep 09 16:28:19.982023 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:19.981991    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/531455b0-04f0-42bc-948f-ba17688f2d93-openshift-service-ca\") pod \"maas-api-7f975847c4-8rlg2\" (UID: \"531455b0-04f0-42bc-948f-ba17688f2d93\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"
Sep 09 16:28:19.982023 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:19.982026    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531455b0-04f0-42bc-948f-ba17688f2d93-metrics-certs\") pod \"maas-api-7f975847c4-8rlg2\" (UID: \"531455b0-04f0-42bc-948f-ba17688f2d93\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"
Sep 09 16:28:19.982230 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:19.982052    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531455b0-04f0-42bc-948f-ba17688f2d93-maas-api-tls\") pod \"maas-api-7f975847c4-8rlg2\" (UID: \"531455b0-04f0-42bc-948f-ba17688f2d93\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"
Sep 09 16:28:19.982230 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:19.982131    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxkqc\" (UniqueName: \"kubernetes.io/projected/531455b0-04f0-42bc-948f-ba17688f2d93-kube-api-access-hxkqc\") pod \"maas-api-7f975847c4-8rlg2\" (UID: \"531455b0-04f0-42bc-948f-ba17688f2d93\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"
Sep 09 16:28:20.082681 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:20.082640    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/531455b0-04f0-42bc-948f-ba17688f2d93-openshift-service-ca\") pod \"maas-api-7f975847c4-8rlg2\" (UID: \"531455b0-04f0-42bc-948f-ba17688f2d93\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"
Sep 09 16:28:20.082681 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:20.082683    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531455b0-04f0-42bc-948f-ba17688f2d93-metrics-certs\") pod \"maas-api-7f975847c4-8rlg2\" (UID: \"531455b0-04f0-42bc-948f-ba17688f2d93\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"
Sep 09 16:28:20.082945 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:20.082719    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531455b0-04f0-42bc-948f-ba17688f2d93-maas-api-tls\") pod \"maas-api-7f975847c4-8rlg2\" (UID: \"531455b0-04f0-42bc-948f-ba17688f2d93\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"
Sep 09 16:28:20.082945 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:20.082768    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxkqc\" (UniqueName: \"kubernetes.io/projected/531455b0-04f0-42bc-948f-ba17688f2d93-kube-api-access-hxkqc\") pod \"maas-api-7f975847c4-8rlg2\" (UID: \"531455b0-04f0-42bc-948f-ba17688f2d93\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"
Sep 09 16:28:20.082945 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:28:20.082873    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Sep 09 16:28:20.083129 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:28:20.082954    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/531455b0-04f0-42bc-948f-ba17688f2d93-maas-api-tls podName:531455b0-04f0-42bc-948f-ba17688f2d93 nodeName:}" failed. No retries permitted until 2026-09-09 16:28:20.582930412 +0000 UTC m=+866.413138454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/531455b0-04f0-42bc-948f-ba17688f2d93-maas-api-tls") pod "maas-api-7f975847c4-8rlg2" (UID: "531455b0-04f0-42bc-948f-ba17688f2d93") : secret "maas-api-serving-cert" not found
Sep 09 16:28:20.083534 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:20.083510    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/531455b0-04f0-42bc-948f-ba17688f2d93-openshift-service-ca\") pod \"maas-api-7f975847c4-8rlg2\" (UID: \"531455b0-04f0-42bc-948f-ba17688f2d93\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"
Sep 09 16:28:20.085372 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:20.085282    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531455b0-04f0-42bc-948f-ba17688f2d93-metrics-certs\") pod \"maas-api-7f975847c4-8rlg2\" (UID: \"531455b0-04f0-42bc-948f-ba17688f2d93\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"
Sep 09 16:28:20.093452 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:20.093427    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxkqc\" (UniqueName: \"kubernetes.io/projected/531455b0-04f0-42bc-948f-ba17688f2d93-kube-api-access-hxkqc\") pod \"maas-api-7f975847c4-8rlg2\" (UID: \"531455b0-04f0-42bc-948f-ba17688f2d93\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"
Sep 09 16:28:20.539476 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:20.539437    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:28:20.587760 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:20.587715    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531455b0-04f0-42bc-948f-ba17688f2d93-maas-api-tls\") pod \"maas-api-7f975847c4-8rlg2\" (UID: \"531455b0-04f0-42bc-948f-ba17688f2d93\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"
Sep 09 16:28:20.590340 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:20.590283    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531455b0-04f0-42bc-948f-ba17688f2d93-maas-api-tls\") pod \"maas-api-7f975847c4-8rlg2\" (UID: \"531455b0-04f0-42bc-948f-ba17688f2d93\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"
Sep 09 16:28:20.841174 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:20.841085    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"
Sep 09 16:28:21.123665 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:21.123540    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"]
Sep 09 16:28:21.125984 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:28:21.125954    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod531455b0_04f0_42bc_948f_ba17688f2d93.slice/crio-2cb8a9ba6b87cfbb3563fe42a4eab75c5967005221cfcacc0294eb7041bd1108 WatchSource:0}: Error finding container 2cb8a9ba6b87cfbb3563fe42a4eab75c5967005221cfcacc0294eb7041bd1108: Status 404 returned error can't find the container with id 2cb8a9ba6b87cfbb3563fe42a4eab75c5967005221cfcacc0294eb7041bd1108
Sep 09 16:28:21.406781 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:21.406692    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2" event={"ID":"531455b0-04f0-42bc-948f-ba17688f2d93","Type":"ContainerStarted","Data":"2cb8a9ba6b87cfbb3563fe42a4eab75c5967005221cfcacc0294eb7041bd1108"}
Sep 09 16:28:24.425934 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:24.425830    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2" event={"ID":"531455b0-04f0-42bc-948f-ba17688f2d93","Type":"ContainerStarted","Data":"5eb6d59f52113930bd123fe50341094dae36fefe5f718c8b60f8c7c932a20f13"}
Sep 09 16:28:24.425934 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:24.425901    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"
Sep 09 16:28:24.441959 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:24.441894    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2" podStartSLOduration=2.469344013 podStartE2EDuration="5.441854385s" totalImagesPullingTime="2.972510372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:28:24.440266224 +0000 UTC m=+870.270474287" watchObservedRunningTime="2026-09-09 16:28:24.441854385 +0000 UTC m=+870.272062445"
Sep 09 16:28:29.469799 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:29.469755    2555 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-cnkj9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": read tcp 10.134.0.2:37016->10.134.0.46:15021: read: connection reset by peer" start-of-body=
Sep 09 16:28:29.470204 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:29.469851    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9" podUID="b512776f-22c4-40b8-89fd-c3cd475e6d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": read tcp 10.134.0.2:37016->10.134.0.46:15021: read: connection reset by peer"
Sep 09 16:28:30.436665 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:30.436636    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"
Sep 09 16:28:30.461104 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:30.461072    2555 generic.go:356] "Generic (PLEG): container finished" podID="b512776f-22c4-40b8-89fd-c3cd475e6d09" containerID="c7eff200e2e7488dac14869f9579cd015da731fa3a054f13338eab4a93270610" exitCode=0
Sep 09 16:28:30.461280 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:30.461135    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9" event={"ID":"b512776f-22c4-40b8-89fd-c3cd475e6d09","Type":"ContainerDied","Data":"c7eff200e2e7488dac14869f9579cd015da731fa3a054f13338eab4a93270610"}
Sep 09 16:28:30.461662 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:30.461638    2555 scope.go:117] "RemoveContainer" containerID="c7eff200e2e7488dac14869f9579cd015da731fa3a054f13338eab4a93270610"
Sep 09 16:28:30.462921 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:30.462888    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:28:30.463006 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:30.462960    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:28:30.463006 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:30.462994    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:28:31.470905 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:31.470866    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9" event={"ID":"b512776f-22c4-40b8-89fd-c3cd475e6d09","Type":"ContainerStarted","Data":"2768335c8a79f849341a4b10663e5677363f952ef06abf6f5fdee1f75e8a8874"}
Sep 09 16:28:32.448017 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:32.447979    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:28:32.448450 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:32.448426    2555 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-cnkj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 09 16:28:32.448628 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:32.448603    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9" podUID="b512776f-22c4-40b8-89fd-c3cd475e6d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 09 16:28:33.448019 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:33.447977    2555 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-cnkj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 09 16:28:33.448639 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:33.448604    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9" podUID="b512776f-22c4-40b8-89fd-c3cd475e6d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 09 16:28:34.447763 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:34.447718    2555 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-cnkj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 09 16:28:34.448067 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:34.448032    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9" podUID="b512776f-22c4-40b8-89fd-c3cd475e6d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 09 16:28:35.447520 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:35.447479    2555 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-cnkj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 09 16:28:35.447809 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:35.447775    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9" podUID="b512776f-22c4-40b8-89fd-c3cd475e6d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 09 16:28:36.447785 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:36.447738    2555 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-cnkj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 09 16:28:36.448194 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:36.447819    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9" podUID="b512776f-22c4-40b8-89fd-c3cd475e6d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 09 16:28:37.452388 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:37.452355    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:28:37.452746 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:37.452613    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:28:37.453217 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:37.453201    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-cnkj9"
Sep 09 16:28:45.405765 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.405728    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"]
Sep 09 16:28:45.408343 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.408318    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"
Sep 09 16:28:45.412248 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.412223    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 16:28:45.412391 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.412225    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 09 16:28:45.412391 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.412236    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 16:28:45.412391 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.412284    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-x7dhn\""
Sep 09 16:28:45.415727 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.415707    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"]
Sep 09 16:28:45.511601 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.511575    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a263ea-58bc-4f76-961e-4231faea205a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp\" (UID: \"28a263ea-58bc-4f76-961e-4231faea205a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"
Sep 09 16:28:45.511705 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.511626    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a263ea-58bc-4f76-961e-4231faea205a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp\" (UID: \"28a263ea-58bc-4f76-961e-4231faea205a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"
Sep 09 16:28:45.511776 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.511756    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a263ea-58bc-4f76-961e-4231faea205a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp\" (UID: \"28a263ea-58bc-4f76-961e-4231faea205a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"
Sep 09 16:28:45.511814 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.511799    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5czl\" (UniqueName: \"kubernetes.io/projected/28a263ea-58bc-4f76-961e-4231faea205a-kube-api-access-d5czl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp\" (UID: \"28a263ea-58bc-4f76-961e-4231faea205a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"
Sep 09 16:28:45.511852 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.511825    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a263ea-58bc-4f76-961e-4231faea205a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp\" (UID: \"28a263ea-58bc-4f76-961e-4231faea205a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"
Sep 09 16:28:45.511887 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.511849    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a263ea-58bc-4f76-961e-4231faea205a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp\" (UID: \"28a263ea-58bc-4f76-961e-4231faea205a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"
Sep 09 16:28:45.612974 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.612938    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a263ea-58bc-4f76-961e-4231faea205a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp\" (UID: \"28a263ea-58bc-4f76-961e-4231faea205a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"
Sep 09 16:28:45.613152 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.612996    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a263ea-58bc-4f76-961e-4231faea205a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp\" (UID: \"28a263ea-58bc-4f76-961e-4231faea205a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"
Sep 09 16:28:45.613152 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.613038    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a263ea-58bc-4f76-961e-4231faea205a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp\" (UID: \"28a263ea-58bc-4f76-961e-4231faea205a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"
Sep 09 16:28:45.613152 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.613068    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5czl\" (UniqueName: \"kubernetes.io/projected/28a263ea-58bc-4f76-961e-4231faea205a-kube-api-access-d5czl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp\" (UID: \"28a263ea-58bc-4f76-961e-4231faea205a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"
Sep 09 16:28:45.613152 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.613102    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a263ea-58bc-4f76-961e-4231faea205a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp\" (UID: \"28a263ea-58bc-4f76-961e-4231faea205a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"
Sep 09 16:28:45.613152 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.613131    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a263ea-58bc-4f76-961e-4231faea205a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp\" (UID: \"28a263ea-58bc-4f76-961e-4231faea205a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"
Sep 09 16:28:45.613516 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.613485    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a263ea-58bc-4f76-961e-4231faea205a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp\" (UID: \"28a263ea-58bc-4f76-961e-4231faea205a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"
Sep 09 16:28:45.613516 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.613502    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a263ea-58bc-4f76-961e-4231faea205a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp\" (UID: \"28a263ea-58bc-4f76-961e-4231faea205a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"
Sep 09 16:28:45.613698 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.613543    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a263ea-58bc-4f76-961e-4231faea205a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp\" (UID: \"28a263ea-58bc-4f76-961e-4231faea205a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"
Sep 09 16:28:45.615367 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.615344    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a263ea-58bc-4f76-961e-4231faea205a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp\" (UID: \"28a263ea-58bc-4f76-961e-4231faea205a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"
Sep 09 16:28:45.615556 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.615541    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a263ea-58bc-4f76-961e-4231faea205a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp\" (UID: \"28a263ea-58bc-4f76-961e-4231faea205a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"
Sep 09 16:28:45.622725 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.622703    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5czl\" (UniqueName: \"kubernetes.io/projected/28a263ea-58bc-4f76-961e-4231faea205a-kube-api-access-d5czl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp\" (UID: \"28a263ea-58bc-4f76-961e-4231faea205a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"
Sep 09 16:28:45.736884 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.736793    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"
Sep 09 16:28:45.892855 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:45.892817    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:28:46.012056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:46.011540    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"]
Sep 09 16:28:46.015541 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:28:46.014660    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a263ea_58bc_4f76_961e_4231faea205a.slice/crio-50c6a1881fbfe2f77debd4de7505d1e64aa40b650d2715d1a961a33ec9b9536a WatchSource:0}: Error finding container 50c6a1881fbfe2f77debd4de7505d1e64aa40b650d2715d1a961a33ec9b9536a: Status 404 returned error can't find the container with id 50c6a1881fbfe2f77debd4de7505d1e64aa40b650d2715d1a961a33ec9b9536a
Sep 09 16:28:46.556096 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:46.556016    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp" event={"ID":"28a263ea-58bc-4f76-961e-4231faea205a","Type":"ContainerStarted","Data":"50c6a1881fbfe2f77debd4de7505d1e64aa40b650d2715d1a961a33ec9b9536a"}
Sep 09 16:28:49.583330 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:49.583274    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp" event={"ID":"28a263ea-58bc-4f76-961e-4231faea205a","Type":"ContainerStarted","Data":"9fc40df8c27f4a6009ea3c0b99a11ad442c01805911573e9a4f68307564a0790"}
Sep 09 16:28:49.602093 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:49.602045    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp" podStartSLOduration=1.688587716 podStartE2EDuration="4.602031031s" totalImagesPullingTime="2.913443315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:28:49.599507224 +0000 UTC m=+895.429715284" watchObservedRunningTime="2026-09-09 16:28:49.602031031 +0000 UTC m=+895.432239090"
Sep 09 16:28:54.582634 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:54.582606    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgcjl_561d9dd7-04c3-42dc-bcf6-2c92fd61706b/console-operator/1.log"
Sep 09 16:28:54.583573 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:54.583552    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgcjl_561d9dd7-04c3-42dc-bcf6-2c92fd61706b/console-operator/1.log"
Sep 09 16:28:55.737594 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:55.737555    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"
Sep 09 16:28:55.737994 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:55.737619    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"
Sep 09 16:28:55.749909 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:55.749880    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"
Sep 09 16:28:56.630077 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:56.630051    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp"
Sep 09 16:28:57.591396 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:28:57.591359    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:29:14.793775 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:29:14.793737    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:29:38.492877 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:29:38.492839    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:29:47.097326 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:29:47.097271    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:29:53.803035 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:29:53.803002    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:30:03.695749 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:03.695712    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"]
Sep 09 16:30:03.699173 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:03.699154    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"
Sep 09 16:30:03.701832 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:03.701814    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 09 16:30:03.707675 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:03.707644    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"]
Sep 09 16:30:03.783012 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:03.782946    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f590638-b2f6-434b-81d0-ec3a9a6d590d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7\" (UID: \"4f590638-b2f6-434b-81d0-ec3a9a6d590d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"
Sep 09 16:30:03.783314 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:03.783274    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4lzd\" (UniqueName: \"kubernetes.io/projected/4f590638-b2f6-434b-81d0-ec3a9a6d590d-kube-api-access-b4lzd\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7\" (UID: \"4f590638-b2f6-434b-81d0-ec3a9a6d590d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"
Sep 09 16:30:03.783501 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:03.783477    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f590638-b2f6-434b-81d0-ec3a9a6d590d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7\" (UID: \"4f590638-b2f6-434b-81d0-ec3a9a6d590d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"
Sep 09 16:30:03.783639 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:03.783539    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f590638-b2f6-434b-81d0-ec3a9a6d590d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7\" (UID: \"4f590638-b2f6-434b-81d0-ec3a9a6d590d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"
Sep 09 16:30:03.783639 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:03.783560    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f590638-b2f6-434b-81d0-ec3a9a6d590d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7\" (UID: \"4f590638-b2f6-434b-81d0-ec3a9a6d590d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"
Sep 09 16:30:03.783639 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:03.783585    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f590638-b2f6-434b-81d0-ec3a9a6d590d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7\" (UID: \"4f590638-b2f6-434b-81d0-ec3a9a6d590d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"
Sep 09 16:30:03.884244 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:03.884204    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f590638-b2f6-434b-81d0-ec3a9a6d590d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7\" (UID: \"4f590638-b2f6-434b-81d0-ec3a9a6d590d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"
Sep 09 16:30:03.884244 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:03.884247    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4lzd\" (UniqueName: \"kubernetes.io/projected/4f590638-b2f6-434b-81d0-ec3a9a6d590d-kube-api-access-b4lzd\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7\" (UID: \"4f590638-b2f6-434b-81d0-ec3a9a6d590d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"
Sep 09 16:30:03.884515 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:03.884313    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f590638-b2f6-434b-81d0-ec3a9a6d590d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7\" (UID: \"4f590638-b2f6-434b-81d0-ec3a9a6d590d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"
Sep 09 16:30:03.884515 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:03.884371    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f590638-b2f6-434b-81d0-ec3a9a6d590d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7\" (UID: \"4f590638-b2f6-434b-81d0-ec3a9a6d590d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"
Sep 09 16:30:03.884515 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:03.884398    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f590638-b2f6-434b-81d0-ec3a9a6d590d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7\" (UID: \"4f590638-b2f6-434b-81d0-ec3a9a6d590d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"
Sep 09 16:30:03.884515 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:03.884421    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f590638-b2f6-434b-81d0-ec3a9a6d590d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7\" (UID: \"4f590638-b2f6-434b-81d0-ec3a9a6d590d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"
Sep 09 16:30:03.884811 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:03.884781    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f590638-b2f6-434b-81d0-ec3a9a6d590d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7\" (UID: \"4f590638-b2f6-434b-81d0-ec3a9a6d590d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"
Sep 09 16:30:03.884917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:03.884901    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f590638-b2f6-434b-81d0-ec3a9a6d590d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7\" (UID: \"4f590638-b2f6-434b-81d0-ec3a9a6d590d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"
Sep 09 16:30:03.884993 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:03.884967    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f590638-b2f6-434b-81d0-ec3a9a6d590d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7\" (UID: \"4f590638-b2f6-434b-81d0-ec3a9a6d590d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"
Sep 09 16:30:03.887180 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:03.887149    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f590638-b2f6-434b-81d0-ec3a9a6d590d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7\" (UID: \"4f590638-b2f6-434b-81d0-ec3a9a6d590d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"
Sep 09 16:30:03.887283 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:03.887199    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f590638-b2f6-434b-81d0-ec3a9a6d590d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7\" (UID: \"4f590638-b2f6-434b-81d0-ec3a9a6d590d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"
Sep 09 16:30:03.891622 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:03.891595    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4lzd\" (UniqueName: \"kubernetes.io/projected/4f590638-b2f6-434b-81d0-ec3a9a6d590d-kube-api-access-b4lzd\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7\" (UID: \"4f590638-b2f6-434b-81d0-ec3a9a6d590d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"
Sep 09 16:30:04.009247 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:04.009213    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"
Sep 09 16:30:04.280268 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:04.280244    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"]
Sep 09 16:30:04.281858 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:30:04.281828    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f590638_b2f6_434b_81d0_ec3a9a6d590d.slice/crio-70e737c5da8fcb5463dabac439ef76d667c8109926bc2ffef60d677b73547cbd WatchSource:0}: Error finding container 70e737c5da8fcb5463dabac439ef76d667c8109926bc2ffef60d677b73547cbd: Status 404 returned error can't find the container with id 70e737c5da8fcb5463dabac439ef76d667c8109926bc2ffef60d677b73547cbd
Sep 09 16:30:04.283176 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:04.283157    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 16:30:04.984987 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:04.984894    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7" event={"ID":"4f590638-b2f6-434b-81d0-ec3a9a6d590d","Type":"ContainerStarted","Data":"a80b15a1e853ac15728897405ae3eb9c31c97dc8a1069b696e9fbeb227a6200d"}
Sep 09 16:30:04.984987 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:04.984941    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7" event={"ID":"4f590638-b2f6-434b-81d0-ec3a9a6d590d","Type":"ContainerStarted","Data":"70e737c5da8fcb5463dabac439ef76d667c8109926bc2ffef60d677b73547cbd"}
Sep 09 16:30:05.003267 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:05.003212    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7" podStartSLOduration=1.7393902140000002 podStartE2EDuration="2.003196557s" totalImagesPullingTime="263.806343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:30:05.000758645 +0000 UTC m=+970.830966727" watchObservedRunningTime="2026-09-09 16:30:05.003196557 +0000 UTC m=+970.833404613"
Sep 09 16:30:05.904014 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:05.903976    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:30:14.009602 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:14.009568    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"
Sep 09 16:30:14.010148 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:14.009842    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"
Sep 09 16:30:14.021953 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:14.021919    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"
Sep 09 16:30:14.045943 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:30:14.045918    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7"
Sep 09 16:31:58.753915 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:31:58.753813    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b554cb7f9-ptjq9"]
Sep 09 16:31:58.754507 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:31:58.754053    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6b554cb7f9-ptjq9" podUID="1ba52872-5a39-45e9-bd34-e6bfb9caca13" containerName="authorino" containerID="cri-o://625aacd76f9e3f3a0f957806bceca6992b7491d376bd9accb9e923fe165a1c5b" gracePeriod=30
Sep 09 16:31:59.001947 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:31:59.001917    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b554cb7f9-ptjq9"
Sep 09 16:31:59.102503 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:31:59.102410    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49p8r\" (UniqueName: \"kubernetes.io/projected/1ba52872-5a39-45e9-bd34-e6bfb9caca13-kube-api-access-49p8r\") pod \"1ba52872-5a39-45e9-bd34-e6bfb9caca13\" (UID: \"1ba52872-5a39-45e9-bd34-e6bfb9caca13\") "
Sep 09 16:31:59.102503 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:31:59.102445    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1ba52872-5a39-45e9-bd34-e6bfb9caca13-tls-cert\") pod \"1ba52872-5a39-45e9-bd34-e6bfb9caca13\" (UID: \"1ba52872-5a39-45e9-bd34-e6bfb9caca13\") "
Sep 09 16:31:59.104549 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:31:59.104517    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba52872-5a39-45e9-bd34-e6bfb9caca13-kube-api-access-49p8r" (OuterVolumeSpecName: "kube-api-access-49p8r") pod "1ba52872-5a39-45e9-bd34-e6bfb9caca13" (UID: "1ba52872-5a39-45e9-bd34-e6bfb9caca13"). InnerVolumeSpecName "kube-api-access-49p8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:31:59.112331 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:31:59.112287    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ba52872-5a39-45e9-bd34-e6bfb9caca13-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "1ba52872-5a39-45e9-bd34-e6bfb9caca13" (UID: "1ba52872-5a39-45e9-bd34-e6bfb9caca13"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:31:59.203457 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:31:59.203417    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49p8r\" (UniqueName: \"kubernetes.io/projected/1ba52872-5a39-45e9-bd34-e6bfb9caca13-kube-api-access-49p8r\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:31:59.203457 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:31:59.203459    2555 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1ba52872-5a39-45e9-bd34-e6bfb9caca13-tls-cert\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:31:59.608841 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:31:59.608804    2555 generic.go:356] "Generic (PLEG): container finished" podID="1ba52872-5a39-45e9-bd34-e6bfb9caca13" containerID="625aacd76f9e3f3a0f957806bceca6992b7491d376bd9accb9e923fe165a1c5b" exitCode=0
Sep 09 16:31:59.609029 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:31:59.608846    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b554cb7f9-ptjq9"
Sep 09 16:31:59.609029 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:31:59.608882    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b554cb7f9-ptjq9" event={"ID":"1ba52872-5a39-45e9-bd34-e6bfb9caca13","Type":"ContainerDied","Data":"625aacd76f9e3f3a0f957806bceca6992b7491d376bd9accb9e923fe165a1c5b"}
Sep 09 16:31:59.609029 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:31:59.608923    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b554cb7f9-ptjq9" event={"ID":"1ba52872-5a39-45e9-bd34-e6bfb9caca13","Type":"ContainerDied","Data":"e86e17837f6d085ad6ab4d9523b45a4c704d701a2e6e4e63843e8ad003b51f15"}
Sep 09 16:31:59.609029 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:31:59.608941    2555 scope.go:117] "RemoveContainer" containerID="625aacd76f9e3f3a0f957806bceca6992b7491d376bd9accb9e923fe165a1c5b"
Sep 09 16:31:59.618256 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:31:59.618237    2555 scope.go:117] "RemoveContainer" containerID="625aacd76f9e3f3a0f957806bceca6992b7491d376bd9accb9e923fe165a1c5b"
Sep 09 16:31:59.618587 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:31:59.618567    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"625aacd76f9e3f3a0f957806bceca6992b7491d376bd9accb9e923fe165a1c5b\": container with ID starting with 625aacd76f9e3f3a0f957806bceca6992b7491d376bd9accb9e923fe165a1c5b not found: ID does not exist" containerID="625aacd76f9e3f3a0f957806bceca6992b7491d376bd9accb9e923fe165a1c5b"
Sep 09 16:31:59.618638 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:31:59.618598    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"625aacd76f9e3f3a0f957806bceca6992b7491d376bd9accb9e923fe165a1c5b"} err="failed to get container status \"625aacd76f9e3f3a0f957806bceca6992b7491d376bd9accb9e923fe165a1c5b\": rpc error: code = NotFound desc = could not find container \"625aacd76f9e3f3a0f957806bceca6992b7491d376bd9accb9e923fe165a1c5b\": container with ID starting with 625aacd76f9e3f3a0f957806bceca6992b7491d376bd9accb9e923fe165a1c5b not found: ID does not exist"
Sep 09 16:31:59.629994 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:31:59.629966    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6b554cb7f9-ptjq9"]
Sep 09 16:31:59.633626 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:31:59.633603    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6b554cb7f9-ptjq9"]
Sep 09 16:32:00.608474 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:00.608443    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ba52872-5a39-45e9-bd34-e6bfb9caca13" path="/var/lib/kubelet/pods/1ba52872-5a39-45e9-bd34-e6bfb9caca13/volumes"
Sep 09 16:32:24.785897 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:24.785861    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"]
Sep 09 16:32:24.786418 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:24.786398    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ba52872-5a39-45e9-bd34-e6bfb9caca13" containerName="authorino"
Sep 09 16:32:24.786479 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:24.786423    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ba52872-5a39-45e9-bd34-e6bfb9caca13" containerName="authorino"
Sep 09 16:32:24.786577 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:24.786564    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ba52872-5a39-45e9-bd34-e6bfb9caca13" containerName="authorino"
Sep 09 16:32:24.799687 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:24.799655    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:24.801003 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:24.800978    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"]
Sep 09 16:32:24.802525 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:24.802499    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-630acf-openshift-default-dockercfg-j6r8l\""
Sep 09 16:32:24.916991 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:24.916953    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-workload-certs\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:24.917166 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:24.917096    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-credential-socket\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:24.917166 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:24.917137    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47bc83af-995c-4e42-bc90-29961e944677-istiod-ca-cert\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:24.917251 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:24.917200    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-istio-envoy\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:24.917251 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:24.917245    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-istio-data\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:24.917359 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:24.917278    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47bc83af-995c-4e42-bc90-29961e944677-istio-token\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:24.917359 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:24.917313    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m5bk\" (UniqueName: \"kubernetes.io/projected/47bc83af-995c-4e42-bc90-29961e944677-kube-api-access-4m5bk\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:24.917437 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:24.917375    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-workload-socket\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:24.917437 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:24.917415    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47bc83af-995c-4e42-bc90-29961e944677-istio-podinfo\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:25.018195 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.018163    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47bc83af-995c-4e42-bc90-29961e944677-istio-token\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:25.018195 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.018200    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m5bk\" (UniqueName: \"kubernetes.io/projected/47bc83af-995c-4e42-bc90-29961e944677-kube-api-access-4m5bk\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:25.018447 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.018230    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-workload-socket\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:25.018447 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.018258    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47bc83af-995c-4e42-bc90-29961e944677-istio-podinfo\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:25.018447 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.018324    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-workload-certs\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:25.018447 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.018400    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-credential-socket\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:25.018447 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.018424    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47bc83af-995c-4e42-bc90-29961e944677-istiod-ca-cert\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:25.018730 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.018466    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-istio-envoy\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:25.018730 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.018498    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-istio-data\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:25.018730 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.018705    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-workload-socket\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:25.018885 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.018753    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-workload-certs\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:25.018885 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.018801    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-credential-socket\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:25.018955 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.018926    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-istio-data\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:25.019133 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.019114    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47bc83af-995c-4e42-bc90-29961e944677-istiod-ca-cert\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:25.020696 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.020677    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-istio-envoy\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:25.021006 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.020984    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47bc83af-995c-4e42-bc90-29961e944677-istio-podinfo\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:25.026367 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.026346    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47bc83af-995c-4e42-bc90-29961e944677-istio-token\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:25.026502 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.026483    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m5bk\" (UniqueName: \"kubernetes.io/projected/47bc83af-995c-4e42-bc90-29961e944677-kube-api-access-4m5bk\") pod \"e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") " pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:25.112547 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.112457    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:25.389340 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.389314    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"]
Sep 09 16:32:25.390892 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:32:25.390866    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47bc83af_995c_4e42_bc90_29961e944677.slice/crio-316e4e08a20d21baf8cf36c6ebecf05971c3b111ee4b57db2a59c7f8a436f5d5 WatchSource:0}: Error finding container 316e4e08a20d21baf8cf36c6ebecf05971c3b111ee4b57db2a59c7f8a436f5d5: Status 404 returned error can't find the container with id 316e4e08a20d21baf8cf36c6ebecf05971c3b111ee4b57db2a59c7f8a436f5d5
Sep 09 16:32:25.393049 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.393013    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:32:25.393126 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.393078    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:32:25.393126 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.393107    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:32:25.754542 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.754504    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt" event={"ID":"47bc83af-995c-4e42-bc90-29961e944677","Type":"ContainerStarted","Data":"c358c8ff041a4b99cdb0489577849c054d6a7f5ec90401c27c0c25da76c52ee8"}
Sep 09 16:32:25.754722 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.754564    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt" event={"ID":"47bc83af-995c-4e42-bc90-29961e944677","Type":"ContainerStarted","Data":"316e4e08a20d21baf8cf36c6ebecf05971c3b111ee4b57db2a59c7f8a436f5d5"}
Sep 09 16:32:25.776361 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:25.776288    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt" podStartSLOduration=1.776274188 podStartE2EDuration="1.776274188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:32:25.773745478 +0000 UTC m=+1111.603953562" watchObservedRunningTime="2026-09-09 16:32:25.776274188 +0000 UTC m=+1111.606482241"
Sep 09 16:32:26.113425 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:26.113345    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:26.117923 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:26.117899    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:26.556907 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:26.556872    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6966-140695943030064"]
Sep 09 16:32:26.560388 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:26.560367    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6966-140695943030064"
Sep 09 16:32:26.562822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:26.562794    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-npvh9\""
Sep 09 16:32:26.567137 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:26.567113    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6966-140695943030064"]
Sep 09 16:32:26.633547 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:26.633516    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx9mv\" (UniqueName: \"kubernetes.io/projected/96ed89be-5ff5-4247-941a-19217c252636-kube-api-access-jx9mv\") pod \"test-curl-6966-140695943030064\" (UID: \"96ed89be-5ff5-4247-941a-19217c252636\") " pod="opendatahub/test-curl-6966-140695943030064"
Sep 09 16:32:26.734977 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:26.734943    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx9mv\" (UniqueName: \"kubernetes.io/projected/96ed89be-5ff5-4247-941a-19217c252636-kube-api-access-jx9mv\") pod \"test-curl-6966-140695943030064\" (UID: \"96ed89be-5ff5-4247-941a-19217c252636\") " pod="opendatahub/test-curl-6966-140695943030064"
Sep 09 16:32:26.743137 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:26.743106    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx9mv\" (UniqueName: \"kubernetes.io/projected/96ed89be-5ff5-4247-941a-19217c252636-kube-api-access-jx9mv\") pod \"test-curl-6966-140695943030064\" (UID: \"96ed89be-5ff5-4247-941a-19217c252636\") " pod="opendatahub/test-curl-6966-140695943030064"
Sep 09 16:32:26.760193 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:26.760164    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:26.761285 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:26.761263    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:26.872340 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:26.872247    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6966-140695943030064"
Sep 09 16:32:27.146571 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:27.146543    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6966-140695943030064"]
Sep 09 16:32:27.149084 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:32:27.149055    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96ed89be_5ff5_4247_941a_19217c252636.slice/crio-f5ce14cc1f464f172d9d751558a16faa14cb97b5891ec1366bf7a0e415779523 WatchSource:0}: Error finding container f5ce14cc1f464f172d9d751558a16faa14cb97b5891ec1366bf7a0e415779523: Status 404 returned error can't find the container with id f5ce14cc1f464f172d9d751558a16faa14cb97b5891ec1366bf7a0e415779523
Sep 09 16:32:27.767601 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:27.767564    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6966-140695943030064" event={"ID":"96ed89be-5ff5-4247-941a-19217c252636","Type":"ContainerStarted","Data":"f5ce14cc1f464f172d9d751558a16faa14cb97b5891ec1366bf7a0e415779523"}
Sep 09 16:32:28.396826 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:28.396740    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:32:28.774847 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:28.774807    2555 generic.go:356] "Generic (PLEG): container finished" podID="96ed89be-5ff5-4247-941a-19217c252636" containerID="051905cb901638da9400b1196d53ed29fa7c056a106bafe1d942e0544a425b83" exitCode=0
Sep 09 16:32:28.775048 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:28.774886    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6966-140695943030064" event={"ID":"96ed89be-5ff5-4247-941a-19217c252636","Type":"ContainerDied","Data":"051905cb901638da9400b1196d53ed29fa7c056a106bafe1d942e0544a425b83"}
Sep 09 16:32:28.831419 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:28.831382    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"]
Sep 09 16:32:28.835346 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:28.835325    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"
Sep 09 16:32:28.837814 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:28.837784    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-630acf-serving-cert\""
Sep 09 16:32:28.850622 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:28.850597    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"]
Sep 09 16:32:28.957031 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:28.956996    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da1cef8f-f790-4448-b049-4912347443ff-openshift-service-ca\") pod \"maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr\" (UID: \"da1cef8f-f790-4448-b049-4912347443ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"
Sep 09 16:32:28.957225 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:28.957070    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da1cef8f-f790-4448-b049-4912347443ff-metrics-certs\") pod \"maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr\" (UID: \"da1cef8f-f790-4448-b049-4912347443ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"
Sep 09 16:32:28.957225 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:28.957129    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da1cef8f-f790-4448-b049-4912347443ff-maas-api-tls\") pod \"maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr\" (UID: \"da1cef8f-f790-4448-b049-4912347443ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"
Sep 09 16:32:28.957225 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:28.957159    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq5b6\" (UniqueName: \"kubernetes.io/projected/da1cef8f-f790-4448-b049-4912347443ff-kube-api-access-zq5b6\") pod \"maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr\" (UID: \"da1cef8f-f790-4448-b049-4912347443ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"
Sep 09 16:32:29.058418 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.058315    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da1cef8f-f790-4448-b049-4912347443ff-maas-api-tls\") pod \"maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr\" (UID: \"da1cef8f-f790-4448-b049-4912347443ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"
Sep 09 16:32:29.058418 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.058369    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq5b6\" (UniqueName: \"kubernetes.io/projected/da1cef8f-f790-4448-b049-4912347443ff-kube-api-access-zq5b6\") pod \"maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr\" (UID: \"da1cef8f-f790-4448-b049-4912347443ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"
Sep 09 16:32:29.058717 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.058450    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da1cef8f-f790-4448-b049-4912347443ff-openshift-service-ca\") pod \"maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr\" (UID: \"da1cef8f-f790-4448-b049-4912347443ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"
Sep 09 16:32:29.058717 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:32:29.058460    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-serving-cert: secret "maas-api-e2e-inf-a-630acf-serving-cert" not found
Sep 09 16:32:29.058717 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.058514    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da1cef8f-f790-4448-b049-4912347443ff-metrics-certs\") pod \"maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr\" (UID: \"da1cef8f-f790-4448-b049-4912347443ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"
Sep 09 16:32:29.058717 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:32:29.058563    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da1cef8f-f790-4448-b049-4912347443ff-maas-api-tls podName:da1cef8f-f790-4448-b049-4912347443ff nodeName:}" failed. No retries permitted until 2026-09-09 16:32:29.558537407 +0000 UTC m=+1115.388745539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da1cef8f-f790-4448-b049-4912347443ff-maas-api-tls") pod "maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr" (UID: "da1cef8f-f790-4448-b049-4912347443ff") : secret "maas-api-e2e-inf-a-630acf-serving-cert" not found
Sep 09 16:32:29.059289 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.059266    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da1cef8f-f790-4448-b049-4912347443ff-openshift-service-ca\") pod \"maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr\" (UID: \"da1cef8f-f790-4448-b049-4912347443ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"
Sep 09 16:32:29.061204 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.061180    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da1cef8f-f790-4448-b049-4912347443ff-metrics-certs\") pod \"maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr\" (UID: \"da1cef8f-f790-4448-b049-4912347443ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"
Sep 09 16:32:29.078994 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.078967    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq5b6\" (UniqueName: \"kubernetes.io/projected/da1cef8f-f790-4448-b049-4912347443ff-kube-api-access-zq5b6\") pod \"maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr\" (UID: \"da1cef8f-f790-4448-b049-4912347443ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"
Sep 09 16:32:29.475362 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.475271    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd"]
Sep 09 16:32:29.478713 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.478694    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd"
Sep 09 16:32:29.481431 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.481409    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-630acf-dockercfg-pq87p\""
Sep 09 16:32:29.481677 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.481658    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-630acf\""
Sep 09 16:32:29.495401 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.495377    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd"]
Sep 09 16:32:29.563777 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.563740    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da1cef8f-f790-4448-b049-4912347443ff-maas-api-tls\") pod \"maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr\" (UID: \"da1cef8f-f790-4448-b049-4912347443ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"
Sep 09 16:32:29.563953 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.563790    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf1cf22-47e8-4729-9049-ff4c40bb6e8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd\" (UID: \"1bf1cf22-47e8-4729-9049-ff4c40bb6e8c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd"
Sep 09 16:32:29.563953 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.563814    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ctlq\" (UniqueName: \"kubernetes.io/projected/1bf1cf22-47e8-4729-9049-ff4c40bb6e8c-kube-api-access-7ctlq\") pod \"payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd\" (UID: \"1bf1cf22-47e8-4729-9049-ff4c40bb6e8c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd"
Sep 09 16:32:29.566420 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.566390    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da1cef8f-f790-4448-b049-4912347443ff-maas-api-tls\") pod \"maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr\" (UID: \"da1cef8f-f790-4448-b049-4912347443ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"
Sep 09 16:32:29.665457 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.665411    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf1cf22-47e8-4729-9049-ff4c40bb6e8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd\" (UID: \"1bf1cf22-47e8-4729-9049-ff4c40bb6e8c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd"
Sep 09 16:32:29.665627 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.665462    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ctlq\" (UniqueName: \"kubernetes.io/projected/1bf1cf22-47e8-4729-9049-ff4c40bb6e8c-kube-api-access-7ctlq\") pod \"payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd\" (UID: \"1bf1cf22-47e8-4729-9049-ff4c40bb6e8c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd"
Sep 09 16:32:29.666646 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.666622    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf1cf22-47e8-4729-9049-ff4c40bb6e8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd\" (UID: \"1bf1cf22-47e8-4729-9049-ff4c40bb6e8c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd"
Sep 09 16:32:29.673887 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.673857    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ctlq\" (UniqueName: \"kubernetes.io/projected/1bf1cf22-47e8-4729-9049-ff4c40bb6e8c-kube-api-access-7ctlq\") pod \"payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd\" (UID: \"1bf1cf22-47e8-4729-9049-ff4c40bb6e8c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd"
Sep 09 16:32:29.750670 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.750632    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"
Sep 09 16:32:29.789686 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.789098    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd"
Sep 09 16:32:29.944358 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.944322    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6966-140695943030064"
Sep 09 16:32:29.976036 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.976000    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-6966-140695943030064_96ed89be-5ff5-4247-941a-19217c252636/test-curl-6966-140695943030064/0.log"
Sep 09 16:32:29.997911 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:29.997873    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6966-140695943030064"]
Sep 09 16:32:30.001684 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:30.001618    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6966-140695943030064"]
Sep 09 16:32:30.026420 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:30.026396    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"]
Sep 09 16:32:30.027565 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:32:30.027541    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda1cef8f_f790_4448_b049_4912347443ff.slice/crio-7e5001402858bc72f0dd7ed963d0b0741725ffbdb2b867a9c291b1f95388dbf0 WatchSource:0}: Error finding container 7e5001402858bc72f0dd7ed963d0b0741725ffbdb2b867a9c291b1f95388dbf0: Status 404 returned error can't find the container with id 7e5001402858bc72f0dd7ed963d0b0741725ffbdb2b867a9c291b1f95388dbf0
Sep 09 16:32:30.068887 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:30.068862    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jx9mv\" (UniqueName: \"kubernetes.io/projected/96ed89be-5ff5-4247-941a-19217c252636-kube-api-access-jx9mv\") pod \"96ed89be-5ff5-4247-941a-19217c252636\" (UID: \"96ed89be-5ff5-4247-941a-19217c252636\") "
Sep 09 16:32:30.070950 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:30.070916    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ed89be-5ff5-4247-941a-19217c252636-kube-api-access-jx9mv" (OuterVolumeSpecName: "kube-api-access-jx9mv") pod "96ed89be-5ff5-4247-941a-19217c252636" (UID: "96ed89be-5ff5-4247-941a-19217c252636"). InnerVolumeSpecName "kube-api-access-jx9mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:32:30.081661 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:32:30.081626    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf1cf22_47e8_4729_9049_ff4c40bb6e8c.slice/crio-07e6117fe75a707afd11f6477c98739c75a07febeb15325bda71cd8bb7d01c5d WatchSource:0}: Error finding container 07e6117fe75a707afd11f6477c98739c75a07febeb15325bda71cd8bb7d01c5d: Status 404 returned error can't find the container with id 07e6117fe75a707afd11f6477c98739c75a07febeb15325bda71cd8bb7d01c5d
Sep 09 16:32:30.081757 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:30.081726    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd"]
Sep 09 16:32:30.124619 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:30.124594    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6966-140695943030064"]
Sep 09 16:32:30.125002 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:30.124988    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96ed89be-5ff5-4247-941a-19217c252636" containerName="test-curl-6966-140695943030064"
Sep 09 16:32:30.125049 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:30.125004    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="96ed89be-5ff5-4247-941a-19217c252636" containerName="test-curl-6966-140695943030064"
Sep 09 16:32:30.125085 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:30.125063    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="96ed89be-5ff5-4247-941a-19217c252636" containerName="test-curl-6966-140695943030064"
Sep 09 16:32:30.128499 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:30.128484    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6966-140695943030064"
Sep 09 16:32:30.133971 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:30.133948    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6966-140695943030064"]
Sep 09 16:32:30.169995 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:30.169969    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwgdd\" (UniqueName: \"kubernetes.io/projected/f612c4f0-e37d-43d3-96cc-6e8e2716be43-kube-api-access-mwgdd\") pod \"test-curl-6966-140695943030064\" (UID: \"f612c4f0-e37d-43d3-96cc-6e8e2716be43\") " pod="opendatahub/test-curl-6966-140695943030064"
Sep 09 16:32:30.170121 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:30.170043    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx9mv\" (UniqueName: \"kubernetes.io/projected/96ed89be-5ff5-4247-941a-19217c252636-kube-api-access-jx9mv\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:32:30.270774 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:30.270653    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwgdd\" (UniqueName: \"kubernetes.io/projected/f612c4f0-e37d-43d3-96cc-6e8e2716be43-kube-api-access-mwgdd\") pod \"test-curl-6966-140695943030064\" (UID: \"f612c4f0-e37d-43d3-96cc-6e8e2716be43\") " pod="opendatahub/test-curl-6966-140695943030064"
Sep 09 16:32:30.278522 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:30.278495    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwgdd\" (UniqueName: \"kubernetes.io/projected/f612c4f0-e37d-43d3-96cc-6e8e2716be43-kube-api-access-mwgdd\") pod \"test-curl-6966-140695943030064\" (UID: \"f612c4f0-e37d-43d3-96cc-6e8e2716be43\") " pod="opendatahub/test-curl-6966-140695943030064"
Sep 09 16:32:30.439086 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:30.439058    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6966-140695943030064"
Sep 09 16:32:30.608476 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:30.608389    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96ed89be-5ff5-4247-941a-19217c252636" path="/var/lib/kubelet/pods/96ed89be-5ff5-4247-941a-19217c252636/volumes"
Sep 09 16:32:30.710443 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:30.710415    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6966-140695943030064"]
Sep 09 16:32:30.711633 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:32:30.711610    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf612c4f0_e37d_43d3_96cc_6e8e2716be43.slice/crio-4b17304ef16fb9ddea8309224bb2ebe4b20cbd7b06430bf2ca3326e5e61a3ac7 WatchSource:0}: Error finding container 4b17304ef16fb9ddea8309224bb2ebe4b20cbd7b06430bf2ca3326e5e61a3ac7: Status 404 returned error can't find the container with id 4b17304ef16fb9ddea8309224bb2ebe4b20cbd7b06430bf2ca3326e5e61a3ac7
Sep 09 16:32:30.787236 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:30.787209    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6966-140695943030064"
Sep 09 16:32:30.787471 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:30.787259    2555 scope.go:117] "RemoveContainer" containerID="051905cb901638da9400b1196d53ed29fa7c056a106bafe1d942e0544a425b83"
Sep 09 16:32:30.792060 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:30.792024    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd" event={"ID":"1bf1cf22-47e8-4729-9049-ff4c40bb6e8c","Type":"ContainerStarted","Data":"07e6117fe75a707afd11f6477c98739c75a07febeb15325bda71cd8bb7d01c5d"}
Sep 09 16:32:30.793331 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:30.793284    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr" event={"ID":"da1cef8f-f790-4448-b049-4912347443ff","Type":"ContainerStarted","Data":"7e5001402858bc72f0dd7ed963d0b0741725ffbdb2b867a9c291b1f95388dbf0"}
Sep 09 16:32:30.794492 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:30.794464    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6966-140695943030064" event={"ID":"f612c4f0-e37d-43d3-96cc-6e8e2716be43","Type":"ContainerStarted","Data":"4b17304ef16fb9ddea8309224bb2ebe4b20cbd7b06430bf2ca3326e5e61a3ac7"}
Sep 09 16:32:31.802808 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:31.802779    2555 generic.go:356] "Generic (PLEG): container finished" podID="f612c4f0-e37d-43d3-96cc-6e8e2716be43" containerID="bc0088711b23a35d21997d9436a0ae4d59547447966e6e6edd8f7f7c834ca97a" exitCode=0
Sep 09 16:32:31.803175 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:31.802852    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6966-140695943030064" event={"ID":"f612c4f0-e37d-43d3-96cc-6e8e2716be43","Type":"ContainerDied","Data":"bc0088711b23a35d21997d9436a0ae4d59547447966e6e6edd8f7f7c834ca97a"}
Sep 09 16:32:32.946882 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:32.946856    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6966-140695943030064"
Sep 09 16:32:32.966955 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:32.966921    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-6966-140695943030064_f612c4f0-e37d-43d3-96cc-6e8e2716be43/test-curl-6966-140695943030064/0.log"
Sep 09 16:32:32.990239 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:32.990204    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6966-140695943030064"]
Sep 09 16:32:32.992426 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:32.992402    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6966-140695943030064"]
Sep 09 16:32:32.995410 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:32.995386    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwgdd\" (UniqueName: \"kubernetes.io/projected/f612c4f0-e37d-43d3-96cc-6e8e2716be43-kube-api-access-mwgdd\") pod \"f612c4f0-e37d-43d3-96cc-6e8e2716be43\" (UID: \"f612c4f0-e37d-43d3-96cc-6e8e2716be43\") "
Sep 09 16:32:32.997466 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:32.997436    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f612c4f0-e37d-43d3-96cc-6e8e2716be43-kube-api-access-mwgdd" (OuterVolumeSpecName: "kube-api-access-mwgdd") pod "f612c4f0-e37d-43d3-96cc-6e8e2716be43" (UID: "f612c4f0-e37d-43d3-96cc-6e8e2716be43"). InnerVolumeSpecName "kube-api-access-mwgdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:32:33.096736 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:33.096706    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwgdd\" (UniqueName: \"kubernetes.io/projected/f612c4f0-e37d-43d3-96cc-6e8e2716be43-kube-api-access-mwgdd\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:32:33.817987 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:33.817947    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr" event={"ID":"da1cef8f-f790-4448-b049-4912347443ff","Type":"ContainerStarted","Data":"cb25d84828e257eb80256e28f8ac0613b8d178728f1123803079d5c19b206d40"}
Sep 09 16:32:33.818171 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:33.818022    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"
Sep 09 16:32:33.819513 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:33.819492    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b17304ef16fb9ddea8309224bb2ebe4b20cbd7b06430bf2ca3326e5e61a3ac7"
Sep 09 16:32:33.819626 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:33.819517    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6966-140695943030064"
Sep 09 16:32:33.838803 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:33.838745    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr" podStartSLOduration=2.810403601 podStartE2EDuration="5.838728064s" totalImagesPullingTime="3.028324463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:32:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:32:33.837572073 +0000 UTC m=+1119.667780146" watchObservedRunningTime="2026-09-09 16:32:33.838728064 +0000 UTC m=+1119.668936149"
Sep 09 16:32:33.839656 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:32:33.839617    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6966-140695943030064\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="f612c4f0-e37d-43d3-96cc-6e8e2716be43" pod="opendatahub/test-curl-6966-140695943030064"
Sep 09 16:32:33.902382 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:33.902344    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"]
Sep 09 16:32:33.903127 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:33.903108    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f612c4f0-e37d-43d3-96cc-6e8e2716be43" containerName="test-curl-6966-140695943030064"
Sep 09 16:32:33.903228 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:33.903218    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f612c4f0-e37d-43d3-96cc-6e8e2716be43" containerName="test-curl-6966-140695943030064"
Sep 09 16:32:33.903444 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:33.903433    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f612c4f0-e37d-43d3-96cc-6e8e2716be43" containerName="test-curl-6966-140695943030064"
Sep 09 16:32:33.906822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:33.906800    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:33.909783 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:33.909552    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-05e496-openshift-default-dockercfg-x5rlr\""
Sep 09 16:32:33.909783 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:32:33.909613    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6966-140695943030064\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="f612c4f0-e37d-43d3-96cc-6e8e2716be43" pod="opendatahub/test-curl-6966-140695943030064"
Sep 09 16:32:33.916058 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:33.916024    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"]
Sep 09 16:32:34.006173 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.005806    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-workload-socket\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.006173 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.005879    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwrvz\" (UniqueName: \"kubernetes.io/projected/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-kube-api-access-nwrvz\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.006173 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.005926    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istio-data\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.006173 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.005954    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istio-token\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.006173 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.005999    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-workload-certs\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.006173 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.006027    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-credential-socket\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.006173 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.006059    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istio-envoy\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.006173 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.006098    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istiod-ca-cert\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.006173 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.006140    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istio-podinfo\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.107544 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.107442    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istio-podinfo\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.107711 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.107553    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-workload-socket\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.107711 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.107588    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwrvz\" (UniqueName: \"kubernetes.io/projected/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-kube-api-access-nwrvz\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.107711 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.107630    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istio-data\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.107711 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.107655    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istio-token\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.107711 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.107684    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-workload-certs\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.108011 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.107714    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-credential-socket\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.108011 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.107746    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istio-envoy\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.108011 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.107785    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istiod-ca-cert\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.108171 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.108040    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-workload-socket\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.108171 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.108091    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istio-data\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.108354 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.108287    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-workload-certs\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.108438 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.108364    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-credential-socket\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.108571 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.108543    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istiod-ca-cert\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.110342 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.110283    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istio-podinfo\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.110742 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.110721    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istio-envoy\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.117155 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.117127    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istio-token\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.118244 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.118202    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwrvz\" (UniqueName: \"kubernetes.io/projected/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-kube-api-access-nwrvz\") pod \"e2e-autores-05e496-openshift-default-754fbb964d-2wb5t\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") " pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.225406 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.225369    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:34.520186 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.520151    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"]
Sep 09 16:32:34.594899 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:32:34.594867    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89c30e4_ab2a_4026_ace4_d7973e8ef3fe.slice/crio-d2db09ea1f9264f26e622fd61dff93d1facc05a52a5145ed4031f90332105a40 WatchSource:0}: Error finding container d2db09ea1f9264f26e622fd61dff93d1facc05a52a5145ed4031f90332105a40: Status 404 returned error can't find the container with id d2db09ea1f9264f26e622fd61dff93d1facc05a52a5145ed4031f90332105a40
Sep 09 16:32:34.597135 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.597099    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:32:34.597224 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.597168    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:32:34.597224 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.597198    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:32:34.607630 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:32:34.607600    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6966-140695943030064\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="f612c4f0-e37d-43d3-96cc-6e8e2716be43" pod="opendatahub/test-curl-6966-140695943030064"
Sep 09 16:32:34.609490 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.609468    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f612c4f0-e37d-43d3-96cc-6e8e2716be43" path="/var/lib/kubelet/pods/f612c4f0-e37d-43d3-96cc-6e8e2716be43/volumes"
Sep 09 16:32:34.827986 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.827889    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t" event={"ID":"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe","Type":"ContainerStarted","Data":"e6e71e07fed1abf07332aaf034b7f1e9aa0c785e3b34ca3c13a1f87655beb8ec"}
Sep 09 16:32:34.827986 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.827952    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t" event={"ID":"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe","Type":"ContainerStarted","Data":"d2db09ea1f9264f26e622fd61dff93d1facc05a52a5145ed4031f90332105a40"}
Sep 09 16:32:34.829805 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.829773    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd" event={"ID":"1bf1cf22-47e8-4729-9049-ff4c40bb6e8c","Type":"ContainerStarted","Data":"12bac8ce95c8ef820166dd208f884da0bf63a9f3c80fe67927c0b61f142b85e2"}
Sep 09 16:32:34.829941 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.829924    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd"
Sep 09 16:32:34.866052 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.865993    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t" podStartSLOduration=1.865971442 podStartE2EDuration="1.865971442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:32:34.864305997 +0000 UTC m=+1120.694514054" watchObservedRunningTime="2026-09-09 16:32:34.865971442 +0000 UTC m=+1120.696179501"
Sep 09 16:32:34.888561 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:34.888516    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd" podStartSLOduration=1.321459354 podStartE2EDuration="5.888495939s" totalImagesPullingTime="4.567036585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:32:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:32:34.887097105 +0000 UTC m=+1120.717305164" watchObservedRunningTime="2026-09-09 16:32:34.888495939 +0000 UTC m=+1120.718704039"
Sep 09 16:32:35.226377 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:35.226265    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:35.227903 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:35.227877    2555 patch_prober.go:28] interesting pod/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 09 16:32:35.228014 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:35.227928    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t" podUID="c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 09 16:32:36.229776 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:36.229740    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:36.842387 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:36.842335    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:36.843688 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:36.843659    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:38.379799 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:38.379732    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn"]
Sep 09 16:32:38.383461 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:38.383443    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn"
Sep 09 16:32:38.385909 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:38.385888    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e94c211c-serving-cert\""
Sep 09 16:32:38.393749 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:38.393704    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn"]
Sep 09 16:32:38.470548 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:38.458833    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f400c9f-aea8-4762-bcb6-588d0cd83330-metrics-certs\") pod \"maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn\" (UID: \"7f400c9f-aea8-4762-bcb6-588d0cd83330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn"
Sep 09 16:32:38.470548 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:38.458882    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bqqf\" (UniqueName: \"kubernetes.io/projected/7f400c9f-aea8-4762-bcb6-588d0cd83330-kube-api-access-4bqqf\") pod \"maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn\" (UID: \"7f400c9f-aea8-4762-bcb6-588d0cd83330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn"
Sep 09 16:32:38.470548 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:38.458920    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f400c9f-aea8-4762-bcb6-588d0cd83330-maas-api-tls\") pod \"maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn\" (UID: \"7f400c9f-aea8-4762-bcb6-588d0cd83330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn"
Sep 09 16:32:38.470548 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:38.458984    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f400c9f-aea8-4762-bcb6-588d0cd83330-openshift-service-ca\") pod \"maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn\" (UID: \"7f400c9f-aea8-4762-bcb6-588d0cd83330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn"
Sep 09 16:32:38.559973 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:38.559936    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f400c9f-aea8-4762-bcb6-588d0cd83330-openshift-service-ca\") pod \"maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn\" (UID: \"7f400c9f-aea8-4762-bcb6-588d0cd83330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn"
Sep 09 16:32:38.560406 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:38.560381    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f400c9f-aea8-4762-bcb6-588d0cd83330-metrics-certs\") pod \"maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn\" (UID: \"7f400c9f-aea8-4762-bcb6-588d0cd83330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn"
Sep 09 16:32:38.560565 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:38.560546    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bqqf\" (UniqueName: \"kubernetes.io/projected/7f400c9f-aea8-4762-bcb6-588d0cd83330-kube-api-access-4bqqf\") pod \"maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn\" (UID: \"7f400c9f-aea8-4762-bcb6-588d0cd83330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn"
Sep 09 16:32:38.560713 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:38.560698    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f400c9f-aea8-4762-bcb6-588d0cd83330-maas-api-tls\") pod \"maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn\" (UID: \"7f400c9f-aea8-4762-bcb6-588d0cd83330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn"
Sep 09 16:32:38.562596 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:38.562574    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f400c9f-aea8-4762-bcb6-588d0cd83330-openshift-service-ca\") pod \"maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn\" (UID: \"7f400c9f-aea8-4762-bcb6-588d0cd83330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn"
Sep 09 16:32:38.567531 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:38.567508    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f400c9f-aea8-4762-bcb6-588d0cd83330-maas-api-tls\") pod \"maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn\" (UID: \"7f400c9f-aea8-4762-bcb6-588d0cd83330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn"
Sep 09 16:32:38.572356 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:38.567780    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f400c9f-aea8-4762-bcb6-588d0cd83330-metrics-certs\") pod \"maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn\" (UID: \"7f400c9f-aea8-4762-bcb6-588d0cd83330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn"
Sep 09 16:32:38.572356 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:38.572060    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bqqf\" (UniqueName: \"kubernetes.io/projected/7f400c9f-aea8-4762-bcb6-588d0cd83330-kube-api-access-4bqqf\") pod \"maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn\" (UID: \"7f400c9f-aea8-4762-bcb6-588d0cd83330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn"
Sep 09 16:32:38.695531 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:38.695448    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn"
Sep 09 16:32:38.992617 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:38.992579    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn"]
Sep 09 16:32:38.995106 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:32:38.995072    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f400c9f_aea8_4762_bcb6_588d0cd83330.slice/crio-bb2e0654af2ef94666affb73b890306f27509e7ad5b0c19d9a9ce3a1b4cc8643 WatchSource:0}: Error finding container bb2e0654af2ef94666affb73b890306f27509e7ad5b0c19d9a9ce3a1b4cc8643: Status 404 returned error can't find the container with id bb2e0654af2ef94666affb73b890306f27509e7ad5b0c19d9a9ce3a1b4cc8643
Sep 09 16:32:39.329140 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:39.329056    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"]
Sep 09 16:32:39.334277 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:39.334257    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"
Sep 09 16:32:39.336982 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:39.336961    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-05e496-serving-cert\""
Sep 09 16:32:39.361319 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:39.354730    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"]
Sep 09 16:32:39.479400 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:39.479371    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c3a7510-7aa7-422c-bd35-654bcdda4167-metrics-certs\") pod \"maas-api-e2e-autores-05e496-849d8c7ffc-m875g\" (UID: \"1c3a7510-7aa7-422c-bd35-654bcdda4167\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"
Sep 09 16:32:39.479756 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:39.479506    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c3a7510-7aa7-422c-bd35-654bcdda4167-openshift-service-ca\") pod \"maas-api-e2e-autores-05e496-849d8c7ffc-m875g\" (UID: \"1c3a7510-7aa7-422c-bd35-654bcdda4167\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"
Sep 09 16:32:39.479756 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:39.479592    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c3a7510-7aa7-422c-bd35-654bcdda4167-maas-api-tls\") pod \"maas-api-e2e-autores-05e496-849d8c7ffc-m875g\" (UID: \"1c3a7510-7aa7-422c-bd35-654bcdda4167\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"
Sep 09 16:32:39.479756 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:39.479672    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n67j\" (UniqueName: \"kubernetes.io/projected/1c3a7510-7aa7-422c-bd35-654bcdda4167-kube-api-access-2n67j\") pod \"maas-api-e2e-autores-05e496-849d8c7ffc-m875g\" (UID: \"1c3a7510-7aa7-422c-bd35-654bcdda4167\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"
Sep 09 16:32:39.580710 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:39.580628    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c3a7510-7aa7-422c-bd35-654bcdda4167-maas-api-tls\") pod \"maas-api-e2e-autores-05e496-849d8c7ffc-m875g\" (UID: \"1c3a7510-7aa7-422c-bd35-654bcdda4167\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"
Sep 09 16:32:39.580845 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:32:39.580794    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-05e496-serving-cert: secret "maas-api-e2e-autores-05e496-serving-cert" not found
Sep 09 16:32:39.580887 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:32:39.580875    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c3a7510-7aa7-422c-bd35-654bcdda4167-maas-api-tls podName:1c3a7510-7aa7-422c-bd35-654bcdda4167 nodeName:}" failed. No retries permitted until 2026-09-09 16:32:40.080853981 +0000 UTC m=+1125.911062022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1c3a7510-7aa7-422c-bd35-654bcdda4167-maas-api-tls") pod "maas-api-e2e-autores-05e496-849d8c7ffc-m875g" (UID: "1c3a7510-7aa7-422c-bd35-654bcdda4167") : secret "maas-api-e2e-autores-05e496-serving-cert" not found
Sep 09 16:32:39.581241 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:39.581201    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n67j\" (UniqueName: \"kubernetes.io/projected/1c3a7510-7aa7-422c-bd35-654bcdda4167-kube-api-access-2n67j\") pod \"maas-api-e2e-autores-05e496-849d8c7ffc-m875g\" (UID: \"1c3a7510-7aa7-422c-bd35-654bcdda4167\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"
Sep 09 16:32:39.581557 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:39.581539    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c3a7510-7aa7-422c-bd35-654bcdda4167-metrics-certs\") pod \"maas-api-e2e-autores-05e496-849d8c7ffc-m875g\" (UID: \"1c3a7510-7aa7-422c-bd35-654bcdda4167\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"
Sep 09 16:32:39.581713 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:39.581696    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c3a7510-7aa7-422c-bd35-654bcdda4167-openshift-service-ca\") pod \"maas-api-e2e-autores-05e496-849d8c7ffc-m875g\" (UID: \"1c3a7510-7aa7-422c-bd35-654bcdda4167\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"
Sep 09 16:32:39.582713 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:39.582691    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c3a7510-7aa7-422c-bd35-654bcdda4167-openshift-service-ca\") pod \"maas-api-e2e-autores-05e496-849d8c7ffc-m875g\" (UID: \"1c3a7510-7aa7-422c-bd35-654bcdda4167\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"
Sep 09 16:32:39.584008 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:39.583985    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c3a7510-7aa7-422c-bd35-654bcdda4167-metrics-certs\") pod \"maas-api-e2e-autores-05e496-849d8c7ffc-m875g\" (UID: \"1c3a7510-7aa7-422c-bd35-654bcdda4167\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"
Sep 09 16:32:39.589892 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:39.589873    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n67j\" (UniqueName: \"kubernetes.io/projected/1c3a7510-7aa7-422c-bd35-654bcdda4167-kube-api-access-2n67j\") pod \"maas-api-e2e-autores-05e496-849d8c7ffc-m875g\" (UID: \"1c3a7510-7aa7-422c-bd35-654bcdda4167\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"
Sep 09 16:32:39.835816 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:39.835728    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"
Sep 09 16:32:39.881365 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:39.881324    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn" event={"ID":"7f400c9f-aea8-4762-bcb6-588d0cd83330","Type":"ContainerStarted","Data":"afe7a6cc9795e75c2a04f5945460f3e7504afbe505619d5edfd244ae9eccedf4"}
Sep 09 16:32:39.881550 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:39.881391    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn" event={"ID":"7f400c9f-aea8-4762-bcb6-588d0cd83330","Type":"ContainerStarted","Data":"bb2e0654af2ef94666affb73b890306f27509e7ad5b0c19d9a9ce3a1b4cc8643"}
Sep 09 16:32:39.881632 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:39.881567    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn"
Sep 09 16:32:39.900678 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:39.900621    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn" podStartSLOduration=1.483550903 podStartE2EDuration="1.900602575s" totalImagesPullingTime="417.051672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:32:39.896463328 +0000 UTC m=+1125.726671389" watchObservedRunningTime="2026-09-09 16:32:39.900602575 +0000 UTC m=+1125.730810635"
Sep 09 16:32:40.088361 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:40.087584    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c3a7510-7aa7-422c-bd35-654bcdda4167-maas-api-tls\") pod \"maas-api-e2e-autores-05e496-849d8c7ffc-m875g\" (UID: \"1c3a7510-7aa7-422c-bd35-654bcdda4167\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"
Sep 09 16:32:40.088361 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:32:40.087769    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-05e496-serving-cert: secret "maas-api-e2e-autores-05e496-serving-cert" not found
Sep 09 16:32:40.088361 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:32:40.087851    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c3a7510-7aa7-422c-bd35-654bcdda4167-maas-api-tls podName:1c3a7510-7aa7-422c-bd35-654bcdda4167 nodeName:}" failed. No retries permitted until 2026-09-09 16:32:41.087830963 +0000 UTC m=+1126.918039015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1c3a7510-7aa7-422c-bd35-654bcdda4167-maas-api-tls") pod "maas-api-e2e-autores-05e496-849d8c7ffc-m875g" (UID: "1c3a7510-7aa7-422c-bd35-654bcdda4167") : secret "maas-api-e2e-autores-05e496-serving-cert" not found
Sep 09 16:32:41.100608 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:41.100570    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c3a7510-7aa7-422c-bd35-654bcdda4167-maas-api-tls\") pod \"maas-api-e2e-autores-05e496-849d8c7ffc-m875g\" (UID: \"1c3a7510-7aa7-422c-bd35-654bcdda4167\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"
Sep 09 16:32:41.103125 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:41.103095    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c3a7510-7aa7-422c-bd35-654bcdda4167-maas-api-tls\") pod \"maas-api-e2e-autores-05e496-849d8c7ffc-m875g\" (UID: \"1c3a7510-7aa7-422c-bd35-654bcdda4167\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"
Sep 09 16:32:41.153268 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:41.153229    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"
Sep 09 16:32:41.440251 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:41.440223    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"]
Sep 09 16:32:41.450314 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:32:41.442695    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3a7510_7aa7_422c_bd35_654bcdda4167.slice/crio-03daad14e6f542ce8e56025c69973a8cf073b4850186b2cc2aa8f89ae9e61650 WatchSource:0}: Error finding container 03daad14e6f542ce8e56025c69973a8cf073b4850186b2cc2aa8f89ae9e61650: Status 404 returned error can't find the container with id 03daad14e6f542ce8e56025c69973a8cf073b4850186b2cc2aa8f89ae9e61650
Sep 09 16:32:41.894548 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:41.894514    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g" event={"ID":"1c3a7510-7aa7-422c-bd35-654bcdda4167","Type":"ContainerStarted","Data":"03daad14e6f542ce8e56025c69973a8cf073b4850186b2cc2aa8f89ae9e61650"}
Sep 09 16:32:42.901921 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:42.901878    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g" event={"ID":"1c3a7510-7aa7-422c-bd35-654bcdda4167","Type":"ContainerStarted","Data":"5ea4e8b8d5236b1432293d821c8411214c8977a0f87784fc13a477ec58f7af6c"}
Sep 09 16:32:42.902377 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:42.901974    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"
Sep 09 16:32:42.919194 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:42.919151    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g" podStartSLOduration=3.4882663369999998 podStartE2EDuration="3.919135776s" totalImagesPullingTime="430.869439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:32:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:32:42.917221188 +0000 UTC m=+1128.747429248" watchObservedRunningTime="2026-09-09 16:32:42.919135776 +0000 UTC m=+1128.749343834"
Sep 09 16:32:45.199471 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:45.199435    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:32:45.837371 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:45.837337    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd"
Sep 09 16:32:45.893054 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:45.893024    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn"
Sep 09 16:32:46.928379 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:46.928338    2555 generic.go:356] "Generic (PLEG): container finished" podID="47bc83af-995c-4e42-bc90-29961e944677" containerID="c358c8ff041a4b99cdb0489577849c054d6a7f5ec90401c27c0c25da76c52ee8" exitCode=0
Sep 09 16:32:46.928790 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:46.928414    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt" event={"ID":"47bc83af-995c-4e42-bc90-29961e944677","Type":"ContainerDied","Data":"c358c8ff041a4b99cdb0489577849c054d6a7f5ec90401c27c0c25da76c52ee8"}
Sep 09 16:32:46.928836 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:46.928819    2555 scope.go:117] "RemoveContainer" containerID="c358c8ff041a4b99cdb0489577849c054d6a7f5ec90401c27c0c25da76c52ee8"
Sep 09 16:32:46.929964 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:46.929932    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:32:46.930022 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:46.930006    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:32:46.930057 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:46.930035    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:32:47.935992 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:47.935955    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt" event={"ID":"47bc83af-995c-4e42-bc90-29961e944677","Type":"ContainerStarted","Data":"ac8380b7bbcb54e200cf1135a836275faca688c52dfccf326106870a3005d758"}
Sep 09 16:32:48.113670 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:48.113638    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:48.113848 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:48.113805    2555 patch_prober.go:28] interesting pod/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 09 16:32:48.113898 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:48.113850    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt" podUID="47bc83af-995c-4e42-bc90-29961e944677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 09 16:32:48.912942 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:48.912907    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"
Sep 09 16:32:49.113563 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:49.113525    2555 patch_prober.go:28] interesting pod/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 09 16:32:49.113947 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:49.113595    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt" podUID="47bc83af-995c-4e42-bc90-29961e944677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 09 16:32:50.113778 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:50.113745    2555 patch_prober.go:28] interesting pod/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 09 16:32:50.114138 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:50.113801    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt" podUID="47bc83af-995c-4e42-bc90-29961e944677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 09 16:32:51.113206 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:51.113163    2555 patch_prober.go:28] interesting pod/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 09 16:32:51.113481 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:51.113450    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt" podUID="47bc83af-995c-4e42-bc90-29961e944677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 09 16:32:52.113422 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:52.113382    2555 patch_prober.go:28] interesting pod/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 09 16:32:52.113794 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:52.113439    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt" podUID="47bc83af-995c-4e42-bc90-29961e944677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 09 16:32:53.112911 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:53.112868    2555 patch_prober.go:28] interesting pod/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 09 16:32:53.113084 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:53.112944    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt" podUID="47bc83af-995c-4e42-bc90-29961e944677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 09 16:32:54.113356 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:54.113308    2555 patch_prober.go:28] interesting pod/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 09 16:32:54.113736 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:54.113383    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt" podUID="47bc83af-995c-4e42-bc90-29961e944677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 09 16:32:55.116976 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:55.116893    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:55.117425 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:55.117408    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:55.118030 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:55.118007    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:32:57.047470 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:57.047430    2555 generic.go:356] "Generic (PLEG): container finished" podID="c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" containerID="e6e71e07fed1abf07332aaf034b7f1e9aa0c785e3b34ca3c13a1f87655beb8ec" exitCode=0
Sep 09 16:32:57.047997 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:57.047511    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t" event={"ID":"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe","Type":"ContainerDied","Data":"e6e71e07fed1abf07332aaf034b7f1e9aa0c785e3b34ca3c13a1f87655beb8ec"}
Sep 09 16:32:57.048131 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:57.048113    2555 scope.go:117] "RemoveContainer" containerID="e6e71e07fed1abf07332aaf034b7f1e9aa0c785e3b34ca3c13a1f87655beb8ec"
Sep 09 16:32:57.049147 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:57.049120    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:32:57.049226 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:57.049184    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:32:57.049226 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:57.049217    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:32:57.834282 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:57.834237    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"]
Sep 09 16:32:57.837070 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:57.837038    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"
Sep 09 16:32:57.839819 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:57.839792    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-34141f-serving-cert\""
Sep 09 16:32:57.850084 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:57.850045    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"]
Sep 09 16:32:57.872314 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:57.871448    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c83958-db6b-48cb-9468-7ea61b946a3a-metrics-certs\") pod \"maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m\" (UID: \"90c83958-db6b-48cb-9468-7ea61b946a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"
Sep 09 16:32:57.872314 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:57.871494    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2dnw\" (UniqueName: \"kubernetes.io/projected/90c83958-db6b-48cb-9468-7ea61b946a3a-kube-api-access-p2dnw\") pod \"maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m\" (UID: \"90c83958-db6b-48cb-9468-7ea61b946a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"
Sep 09 16:32:57.872314 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:57.871548    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c83958-db6b-48cb-9468-7ea61b946a3a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m\" (UID: \"90c83958-db6b-48cb-9468-7ea61b946a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"
Sep 09 16:32:57.872314 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:57.871580    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90c83958-db6b-48cb-9468-7ea61b946a3a-maas-api-tls\") pod \"maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m\" (UID: \"90c83958-db6b-48cb-9468-7ea61b946a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"
Sep 09 16:32:57.973908 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:57.973865    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c83958-db6b-48cb-9468-7ea61b946a3a-metrics-certs\") pod \"maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m\" (UID: \"90c83958-db6b-48cb-9468-7ea61b946a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"
Sep 09 16:32:57.974135 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:57.973919    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2dnw\" (UniqueName: \"kubernetes.io/projected/90c83958-db6b-48cb-9468-7ea61b946a3a-kube-api-access-p2dnw\") pod \"maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m\" (UID: \"90c83958-db6b-48cb-9468-7ea61b946a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"
Sep 09 16:32:57.974135 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:57.973978    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c83958-db6b-48cb-9468-7ea61b946a3a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m\" (UID: \"90c83958-db6b-48cb-9468-7ea61b946a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"
Sep 09 16:32:57.974135 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:57.974014    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90c83958-db6b-48cb-9468-7ea61b946a3a-maas-api-tls\") pod \"maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m\" (UID: \"90c83958-db6b-48cb-9468-7ea61b946a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"
Sep 09 16:32:57.974342 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:32:57.974199    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-serving-cert: secret "maas-api-e2e-inf-b-34141f-serving-cert" not found
Sep 09 16:32:57.974342 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:32:57.974265    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90c83958-db6b-48cb-9468-7ea61b946a3a-maas-api-tls podName:90c83958-db6b-48cb-9468-7ea61b946a3a nodeName:}" failed. No retries permitted until 2026-09-09 16:32:58.474242145 +0000 UTC m=+1144.304450205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/90c83958-db6b-48cb-9468-7ea61b946a3a-maas-api-tls") pod "maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m" (UID: "90c83958-db6b-48cb-9468-7ea61b946a3a") : secret "maas-api-e2e-inf-b-34141f-serving-cert" not found
Sep 09 16:32:57.975207 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:57.975178    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c83958-db6b-48cb-9468-7ea61b946a3a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m\" (UID: \"90c83958-db6b-48cb-9468-7ea61b946a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"
Sep 09 16:32:57.977848 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:57.977821    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c83958-db6b-48cb-9468-7ea61b946a3a-metrics-certs\") pod \"maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m\" (UID: \"90c83958-db6b-48cb-9468-7ea61b946a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"
Sep 09 16:32:57.987728 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:57.986563    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2dnw\" (UniqueName: \"kubernetes.io/projected/90c83958-db6b-48cb-9468-7ea61b946a3a-kube-api-access-p2dnw\") pod \"maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m\" (UID: \"90c83958-db6b-48cb-9468-7ea61b946a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"
Sep 09 16:32:58.062371 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:58.062318    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t" event={"ID":"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe","Type":"ContainerStarted","Data":"201fd0a4106a35c4c5c0d7b293d78922263af86564ba906e7019f6bbcccfb3aa"}
Sep 09 16:32:58.225936 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:58.225843    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:32:58.226087 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:58.226034    2555 patch_prober.go:28] interesting pod/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 09 16:32:58.226130 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:58.226079    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t" podUID="c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 09 16:32:58.480746 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:58.480657    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90c83958-db6b-48cb-9468-7ea61b946a3a-maas-api-tls\") pod \"maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m\" (UID: \"90c83958-db6b-48cb-9468-7ea61b946a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"
Sep 09 16:32:58.482646 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:58.482623    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90c83958-db6b-48cb-9468-7ea61b946a3a-maas-api-tls\") pod \"maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m\" (UID: \"90c83958-db6b-48cb-9468-7ea61b946a3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"
Sep 09 16:32:58.771941 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:58.771899    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"
Sep 09 16:32:59.053830 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:32:59.053784    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c83958_db6b_48cb_9468_7ea61b946a3a.slice/crio-e8a67acaf6143c566c4775331f9bb72598c6875de19b1bc4725fb192bc21fe8b WatchSource:0}: Error finding container e8a67acaf6143c566c4775331f9bb72598c6875de19b1bc4725fb192bc21fe8b: Status 404 returned error can't find the container with id e8a67acaf6143c566c4775331f9bb72598c6875de19b1bc4725fb192bc21fe8b
Sep 09 16:32:59.056059 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:59.056031    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"]
Sep 09 16:32:59.075795 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:59.075757    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m" event={"ID":"90c83958-db6b-48cb-9468-7ea61b946a3a","Type":"ContainerStarted","Data":"e8a67acaf6143c566c4775331f9bb72598c6875de19b1bc4725fb192bc21fe8b"}
Sep 09 16:32:59.226149 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:59.226114    2555 patch_prober.go:28] interesting pod/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 09 16:32:59.226360 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:32:59.226173    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t" podUID="c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 09 16:33:00.082996 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:00.082960    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m" event={"ID":"90c83958-db6b-48cb-9468-7ea61b946a3a","Type":"ContainerStarted","Data":"1471f6e6b8d479b36c4040359074c172f453dd4b180a14b0e132dff631b74a3b"}
Sep 09 16:33:00.083409 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:00.083078    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"
Sep 09 16:33:00.102193 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:00.102141    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m" podStartSLOduration=2.607641395 podStartE2EDuration="3.102125733s" totalImagesPullingTime="494.484338ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:33:00.100459899 +0000 UTC m=+1145.930667958" watchObservedRunningTime="2026-09-09 16:33:00.102125733 +0000 UTC m=+1145.932333849"
Sep 09 16:33:00.226042 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:00.226003    2555 patch_prober.go:28] interesting pod/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 09 16:33:00.226242 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:00.226067    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t" podUID="c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 09 16:33:01.226621 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:01.226581    2555 patch_prober.go:28] interesting pod/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 09 16:33:01.227037 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:01.226640    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t" podUID="c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 09 16:33:02.226645 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:02.226601    2555 patch_prober.go:28] interesting pod/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 09 16:33:02.227034 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:02.226683    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t" podUID="c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 09 16:33:03.267700 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:03.267665    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:33:03.268149 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:03.268104    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:33:03.270463 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:03.270439    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:33:06.094473 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:06.094444    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"
Sep 09 16:33:06.875155 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:06.875122    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn"]
Sep 09 16:33:06.875399 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:06.875373    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn" podUID="7f400c9f-aea8-4762-bcb6-588d0cd83330" containerName="maas-api" containerID="cri-o://afe7a6cc9795e75c2a04f5945460f3e7504afbe505619d5edfd244ae9eccedf4" gracePeriod=30
Sep 09 16:33:07.116174 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:07.116152    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn"
Sep 09 16:33:07.127866 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:07.127809    2555 generic.go:356] "Generic (PLEG): container finished" podID="7f400c9f-aea8-4762-bcb6-588d0cd83330" containerID="afe7a6cc9795e75c2a04f5945460f3e7504afbe505619d5edfd244ae9eccedf4" exitCode=0
Sep 09 16:33:07.127866 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:07.127849    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn"
Sep 09 16:33:07.128002 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:07.127888    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn" event={"ID":"7f400c9f-aea8-4762-bcb6-588d0cd83330","Type":"ContainerDied","Data":"afe7a6cc9795e75c2a04f5945460f3e7504afbe505619d5edfd244ae9eccedf4"}
Sep 09 16:33:07.128002 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:07.127937    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn" event={"ID":"7f400c9f-aea8-4762-bcb6-588d0cd83330","Type":"ContainerDied","Data":"bb2e0654af2ef94666affb73b890306f27509e7ad5b0c19d9a9ce3a1b4cc8643"}
Sep 09 16:33:07.128002 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:07.127961    2555 scope.go:117] "RemoveContainer" containerID="afe7a6cc9795e75c2a04f5945460f3e7504afbe505619d5edfd244ae9eccedf4"
Sep 09 16:33:07.149361 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:07.144039    2555 scope.go:117] "RemoveContainer" containerID="afe7a6cc9795e75c2a04f5945460f3e7504afbe505619d5edfd244ae9eccedf4"
Sep 09 16:33:07.149361 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:33:07.144461    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe7a6cc9795e75c2a04f5945460f3e7504afbe505619d5edfd244ae9eccedf4\": container with ID starting with afe7a6cc9795e75c2a04f5945460f3e7504afbe505619d5edfd244ae9eccedf4 not found: ID does not exist" containerID="afe7a6cc9795e75c2a04f5945460f3e7504afbe505619d5edfd244ae9eccedf4"
Sep 09 16:33:07.149361 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:07.144496    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe7a6cc9795e75c2a04f5945460f3e7504afbe505619d5edfd244ae9eccedf4"} err="failed to get container status \"afe7a6cc9795e75c2a04f5945460f3e7504afbe505619d5edfd244ae9eccedf4\": rpc error: code = NotFound desc = could not find container \"afe7a6cc9795e75c2a04f5945460f3e7504afbe505619d5edfd244ae9eccedf4\": container with ID starting with afe7a6cc9795e75c2a04f5945460f3e7504afbe505619d5edfd244ae9eccedf4 not found: ID does not exist"
Sep 09 16:33:07.163165 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:07.162812    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f400c9f-aea8-4762-bcb6-588d0cd83330-metrics-certs\") pod \"7f400c9f-aea8-4762-bcb6-588d0cd83330\" (UID: \"7f400c9f-aea8-4762-bcb6-588d0cd83330\") "
Sep 09 16:33:07.163165 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:07.162900    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f400c9f-aea8-4762-bcb6-588d0cd83330-maas-api-tls\") pod \"7f400c9f-aea8-4762-bcb6-588d0cd83330\" (UID: \"7f400c9f-aea8-4762-bcb6-588d0cd83330\") "
Sep 09 16:33:07.163165 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:07.163012    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f400c9f-aea8-4762-bcb6-588d0cd83330-openshift-service-ca\") pod \"7f400c9f-aea8-4762-bcb6-588d0cd83330\" (UID: \"7f400c9f-aea8-4762-bcb6-588d0cd83330\") "
Sep 09 16:33:07.163165 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:07.163046    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bqqf\" (UniqueName: \"kubernetes.io/projected/7f400c9f-aea8-4762-bcb6-588d0cd83330-kube-api-access-4bqqf\") pod \"7f400c9f-aea8-4762-bcb6-588d0cd83330\" (UID: \"7f400c9f-aea8-4762-bcb6-588d0cd83330\") "
Sep 09 16:33:07.165439 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:07.165404    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f400c9f-aea8-4762-bcb6-588d0cd83330-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7f400c9f-aea8-4762-bcb6-588d0cd83330" (UID: "7f400c9f-aea8-4762-bcb6-588d0cd83330"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:33:07.165991 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:07.165954    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f400c9f-aea8-4762-bcb6-588d0cd83330-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7f400c9f-aea8-4762-bcb6-588d0cd83330" (UID: "7f400c9f-aea8-4762-bcb6-588d0cd83330"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:33:07.169740 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:07.169668    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f400c9f-aea8-4762-bcb6-588d0cd83330-kube-api-access-4bqqf" (OuterVolumeSpecName: "kube-api-access-4bqqf") pod "7f400c9f-aea8-4762-bcb6-588d0cd83330" (UID: "7f400c9f-aea8-4762-bcb6-588d0cd83330"). InnerVolumeSpecName "kube-api-access-4bqqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:33:07.170809 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:07.170782    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f400c9f-aea8-4762-bcb6-588d0cd83330-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7f400c9f-aea8-4762-bcb6-588d0cd83330" (UID: "7f400c9f-aea8-4762-bcb6-588d0cd83330"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:33:07.266280 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:07.266246    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f400c9f-aea8-4762-bcb6-588d0cd83330-openshift-service-ca\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:33:07.266280 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:07.266274    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bqqf\" (UniqueName: \"kubernetes.io/projected/7f400c9f-aea8-4762-bcb6-588d0cd83330-kube-api-access-4bqqf\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:33:07.266280 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:07.266285    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f400c9f-aea8-4762-bcb6-588d0cd83330-metrics-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:33:07.266530 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:07.266318    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f400c9f-aea8-4762-bcb6-588d0cd83330-maas-api-tls\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:33:07.465933 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:07.465904    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn"]
Sep 09 16:33:07.479394 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:07.479366    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e94c211c-69bd7469f7-25gjn"]
Sep 09 16:33:07.612720 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:07.612686    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:33:08.208278 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.208243    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"]
Sep 09 16:33:08.208903 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.208876    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f400c9f-aea8-4762-bcb6-588d0cd83330" containerName="maas-api"
Sep 09 16:33:08.209068 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.209048    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f400c9f-aea8-4762-bcb6-588d0cd83330" containerName="maas-api"
Sep 09 16:33:08.209182 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.209168    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f400c9f-aea8-4762-bcb6-588d0cd83330" containerName="maas-api"
Sep 09 16:33:08.212243 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.212218    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.214988 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.214968    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-9e0f78a3-gw-openshift-default-dockercfg-hv65r\""
Sep 09 16:33:08.226213 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.226123    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"]
Sep 09 16:33:08.275509 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.275476    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istio-envoy\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.275509 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.275517    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9hnm\" (UniqueName: \"kubernetes.io/projected/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-kube-api-access-f9hnm\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.275739 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.275627    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-credential-socket\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.275739 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.275668    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istio-data\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.275739 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.275696    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istiod-ca-cert\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.275739 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.275736    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-workload-socket\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.275925 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.275752    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istio-token\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.275925 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.275776    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istio-podinfo\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.275925 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.275851    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-workload-certs\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.376386 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.376357    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-credential-socket\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.376549 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.376410    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istio-data\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.376634 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.376586    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istiod-ca-cert\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.376696 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.376634    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-workload-socket\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.376696 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.376659    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istio-token\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.376802 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.376702    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istio-podinfo\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.376802 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.376732    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-workload-certs\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.376802 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.376752    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istio-data\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.376802 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.376791    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istio-envoy\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.376999 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.376828    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9hnm\" (UniqueName: \"kubernetes.io/projected/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-kube-api-access-f9hnm\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.376999 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.376957    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-workload-socket\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.377102 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.377069    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-credential-socket\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.377102 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.377090    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-workload-certs\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.377410 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.377381    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istiod-ca-cert\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.379315 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.379270    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istio-podinfo\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.379441 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.379421    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istio-envoy\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.384810 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.384789    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istio-token\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.384889 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.384877    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9hnm\" (UniqueName: \"kubernetes.io/projected/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-kube-api-access-f9hnm\") pod \"e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") " pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.524445 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.524407    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:08.609302 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.608876    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f400c9f-aea8-4762-bcb6-588d0cd83330" path="/var/lib/kubelet/pods/7f400c9f-aea8-4762-bcb6-588d0cd83330/volumes"
Sep 09 16:33:08.824690 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.824660    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"]
Sep 09 16:33:08.826069 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:33:08.826040    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7243d0b_acfe_40f5_9ab9_e03c161a8c90.slice/crio-aae31fd0e89ed1d30bdb6e19f8b24f700ed5b7d8f939106e522ab8a6fb186093 WatchSource:0}: Error finding container aae31fd0e89ed1d30bdb6e19f8b24f700ed5b7d8f939106e522ab8a6fb186093: Status 404 returned error can't find the container with id aae31fd0e89ed1d30bdb6e19f8b24f700ed5b7d8f939106e522ab8a6fb186093
Sep 09 16:33:08.828426 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.828383    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:33:08.828548 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.828474    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:33:08.828548 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:08.828540    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:33:09.143846 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:09.143752    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf" event={"ID":"f7243d0b-acfe-40f5-9ab9-e03c161a8c90","Type":"ContainerStarted","Data":"00f895b50ff6d8cc477f0966c7ac909642cc1b3a01e22d18f8432bd69b2a7950"}
Sep 09 16:33:09.143846 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:09.143814    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf" event={"ID":"f7243d0b-acfe-40f5-9ab9-e03c161a8c90","Type":"ContainerStarted","Data":"aae31fd0e89ed1d30bdb6e19f8b24f700ed5b7d8f939106e522ab8a6fb186093"}
Sep 09 16:33:09.164755 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:09.164700    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf" podStartSLOduration=1.164684902 podStartE2EDuration="1.164684902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:33:09.161959208 +0000 UTC m=+1154.992167265" watchObservedRunningTime="2026-09-09 16:33:09.164684902 +0000 UTC m=+1154.994893026"
Sep 09 16:33:09.525088 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:09.525055    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:09.526634 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:09.526607    2555 patch_prober.go:28] interesting pod/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 09 16:33:09.526784 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:09.526657    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf" podUID="f7243d0b-acfe-40f5-9ab9-e03c161a8c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 09 16:33:10.529609 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:10.529574    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:11.157581 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:11.157547    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:11.159061 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:11.159037    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:26.159450 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:26.159367    2555 patch_prober.go:28] interesting pod/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 09 16:33:26.159450 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:26.159429    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf" podUID="f7243d0b-acfe-40f5-9ab9-e03c161a8c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 09 16:33:26.258715 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:26.258680    2555 generic.go:356] "Generic (PLEG): container finished" podID="f7243d0b-acfe-40f5-9ab9-e03c161a8c90" containerID="00f895b50ff6d8cc477f0966c7ac909642cc1b3a01e22d18f8432bd69b2a7950" exitCode=0
Sep 09 16:33:26.258877 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:26.258757    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf" event={"ID":"f7243d0b-acfe-40f5-9ab9-e03c161a8c90","Type":"ContainerDied","Data":"00f895b50ff6d8cc477f0966c7ac909642cc1b3a01e22d18f8432bd69b2a7950"}
Sep 09 16:33:26.259179 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:26.259165    2555 scope.go:117] "RemoveContainer" containerID="00f895b50ff6d8cc477f0966c7ac909642cc1b3a01e22d18f8432bd69b2a7950"
Sep 09 16:33:26.260355 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:26.260318    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:33:26.260474 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:26.260453    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:33:26.260528 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:26.260510    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:33:26.303623 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:26.303591    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:33:27.272512 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:27.272476    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf" event={"ID":"f7243d0b-acfe-40f5-9ab9-e03c161a8c90","Type":"ContainerStarted","Data":"88c1bfd3f6c91e43ebddfbcb949533bd16c9a6f8263ef2e264d5ab67a41b9a00"}
Sep 09 16:33:27.525721 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:27.525619    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:27.525886 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:27.525841    2555 patch_prober.go:28] interesting pod/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 09 16:33:27.525997 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:27.525890    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf" podUID="f7243d0b-acfe-40f5-9ab9-e03c161a8c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 09 16:33:28.525224 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:28.525179    2555 patch_prober.go:28] interesting pod/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 09 16:33:28.525662 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:28.525238    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf" podUID="f7243d0b-acfe-40f5-9ab9-e03c161a8c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 09 16:33:29.525227 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:29.525191    2555 patch_prober.go:28] interesting pod/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 09 16:33:29.525687 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:29.525250    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf" podUID="f7243d0b-acfe-40f5-9ab9-e03c161a8c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 09 16:33:30.525393 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:30.525352    2555 patch_prober.go:28] interesting pod/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 09 16:33:30.525789 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:30.525416    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf" podUID="f7243d0b-acfe-40f5-9ab9-e03c161a8c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 09 16:33:31.525127 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:31.525078    2555 patch_prober.go:28] interesting pod/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 09 16:33:31.525333 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:31.525156    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf" podUID="f7243d0b-acfe-40f5-9ab9-e03c161a8c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 09 16:33:32.528878 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:32.528841    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:32.529356 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:32.529231    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:32.529914 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:32.529895    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:42.717090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:42.717050    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"]
Sep 09 16:33:42.717566 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:42.717329    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf" podUID="f7243d0b-acfe-40f5-9ab9-e03c161a8c90" containerName="istio-proxy" containerID="cri-o://88c1bfd3f6c91e43ebddfbcb949533bd16c9a6f8263ef2e264d5ab67a41b9a00" gracePeriod=30
Sep 09 16:33:47.250210 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:47.250173    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:33:47.531542 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:47.531455    2555 patch_prober.go:28] interesting pod/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 16:33:47.531715 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:47.531533    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf" podUID="f7243d0b-acfe-40f5-9ab9-e03c161a8c90" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 16:33:47.979305 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:47.979266    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:48.075016 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.074981    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istio-token\") pod \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") "
Sep 09 16:33:48.075016 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.075019    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istiod-ca-cert\") pod \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") "
Sep 09 16:33:48.075271 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.075048    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-workload-socket\") pod \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") "
Sep 09 16:33:48.075271 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.075075    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istio-podinfo\") pod \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") "
Sep 09 16:33:48.075271 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.075101    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istio-envoy\") pod \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") "
Sep 09 16:33:48.075271 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.075128    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-workload-certs\") pod \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") "
Sep 09 16:33:48.075271 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.075156    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istio-data\") pod \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") "
Sep 09 16:33:48.075271 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.075266    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-credential-socket\") pod \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") "
Sep 09 16:33:48.075607 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.075318    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9hnm\" (UniqueName: \"kubernetes.io/projected/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-kube-api-access-f9hnm\") pod \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\" (UID: \"f7243d0b-acfe-40f5-9ab9-e03c161a8c90\") "
Sep 09 16:33:48.075607 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.075352    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f7243d0b-acfe-40f5-9ab9-e03c161a8c90" (UID: "f7243d0b-acfe-40f5-9ab9-e03c161a8c90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:33:48.075607 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.075510    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f7243d0b-acfe-40f5-9ab9-e03c161a8c90" (UID: "f7243d0b-acfe-40f5-9ab9-e03c161a8c90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:33:48.075607 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.075534    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f7243d0b-acfe-40f5-9ab9-e03c161a8c90" (UID: "f7243d0b-acfe-40f5-9ab9-e03c161a8c90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:33:48.075830 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.075677    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-credential-socket\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:33:48.075830 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.075697    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-workload-socket\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:33:48.075830 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.075711    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-workload-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:33:48.075830 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.075707    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f7243d0b-acfe-40f5-9ab9-e03c161a8c90" (UID: "f7243d0b-acfe-40f5-9ab9-e03c161a8c90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:33:48.076005 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.075913    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istio-data" (OuterVolumeSpecName: "istio-data") pod "f7243d0b-acfe-40f5-9ab9-e03c161a8c90" (UID: "f7243d0b-acfe-40f5-9ab9-e03c161a8c90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:33:48.077577 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.077551    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f7243d0b-acfe-40f5-9ab9-e03c161a8c90" (UID: "f7243d0b-acfe-40f5-9ab9-e03c161a8c90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:33:48.077837 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.077815    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f7243d0b-acfe-40f5-9ab9-e03c161a8c90" (UID: "f7243d0b-acfe-40f5-9ab9-e03c161a8c90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 16:33:48.077913 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.077838    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-kube-api-access-f9hnm" (OuterVolumeSpecName: "kube-api-access-f9hnm") pod "f7243d0b-acfe-40f5-9ab9-e03c161a8c90" (UID: "f7243d0b-acfe-40f5-9ab9-e03c161a8c90"). InnerVolumeSpecName "kube-api-access-f9hnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:33:48.078123 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.078095    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istio-token" (OuterVolumeSpecName: "istio-token") pod "f7243d0b-acfe-40f5-9ab9-e03c161a8c90" (UID: "f7243d0b-acfe-40f5-9ab9-e03c161a8c90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:33:48.164554 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.164470    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427"]
Sep 09 16:33:48.164971 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.164952    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7243d0b-acfe-40f5-9ab9-e03c161a8c90" containerName="istio-proxy"
Sep 09 16:33:48.165063 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.164974    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7243d0b-acfe-40f5-9ab9-e03c161a8c90" containerName="istio-proxy"
Sep 09 16:33:48.165063 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.164996    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7243d0b-acfe-40f5-9ab9-e03c161a8c90" containerName="istio-proxy"
Sep 09 16:33:48.165063 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.165006    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7243d0b-acfe-40f5-9ab9-e03c161a8c90" containerName="istio-proxy"
Sep 09 16:33:48.165209 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.165122    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7243d0b-acfe-40f5-9ab9-e03c161a8c90" containerName="istio-proxy"
Sep 09 16:33:48.165209 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.165136    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7243d0b-acfe-40f5-9ab9-e03c161a8c90" containerName="istio-proxy"
Sep 09 16:33:48.167523 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.167501    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427"
Sep 09 16:33:48.170433 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.170410    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-49eb9be1-serving-cert\""
Sep 09 16:33:48.176745 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.176675    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427"]
Sep 09 16:33:48.177407 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.177276    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istio-podinfo\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:33:48.177407 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.177334    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istio-envoy\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:33:48.177407 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.177349    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istio-data\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:33:48.177407 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.177361    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9hnm\" (UniqueName: \"kubernetes.io/projected/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-kube-api-access-f9hnm\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:33:48.177407 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.177375    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istio-token\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:33:48.177407 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.177386    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7243d0b-acfe-40f5-9ab9-e03c161a8c90-istiod-ca-cert\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:33:48.278586 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.278549    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25b8726-066a-408a-908e-2ceddf777165-maas-api-tls\") pod \"maas-api-e2e-ait-49eb9be1-5bd9858644-kf427\" (UID: \"b25b8726-066a-408a-908e-2ceddf777165\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427"
Sep 09 16:33:48.278998 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.278703    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25b8726-066a-408a-908e-2ceddf777165-openshift-service-ca\") pod \"maas-api-e2e-ait-49eb9be1-5bd9858644-kf427\" (UID: \"b25b8726-066a-408a-908e-2ceddf777165\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427"
Sep 09 16:33:48.278998 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.278728    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln4sp\" (UniqueName: \"kubernetes.io/projected/b25b8726-066a-408a-908e-2ceddf777165-kube-api-access-ln4sp\") pod \"maas-api-e2e-ait-49eb9be1-5bd9858644-kf427\" (UID: \"b25b8726-066a-408a-908e-2ceddf777165\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427"
Sep 09 16:33:48.278998 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.278747    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b25b8726-066a-408a-908e-2ceddf777165-metrics-certs\") pod \"maas-api-e2e-ait-49eb9be1-5bd9858644-kf427\" (UID: \"b25b8726-066a-408a-908e-2ceddf777165\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427"
Sep 09 16:33:48.379619 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.379579    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25b8726-066a-408a-908e-2ceddf777165-openshift-service-ca\") pod \"maas-api-e2e-ait-49eb9be1-5bd9858644-kf427\" (UID: \"b25b8726-066a-408a-908e-2ceddf777165\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427"
Sep 09 16:33:48.379619 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.379624    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln4sp\" (UniqueName: \"kubernetes.io/projected/b25b8726-066a-408a-908e-2ceddf777165-kube-api-access-ln4sp\") pod \"maas-api-e2e-ait-49eb9be1-5bd9858644-kf427\" (UID: \"b25b8726-066a-408a-908e-2ceddf777165\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427"
Sep 09 16:33:48.379891 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.379648    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b25b8726-066a-408a-908e-2ceddf777165-metrics-certs\") pod \"maas-api-e2e-ait-49eb9be1-5bd9858644-kf427\" (UID: \"b25b8726-066a-408a-908e-2ceddf777165\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427"
Sep 09 16:33:48.379891 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.379684    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25b8726-066a-408a-908e-2ceddf777165-maas-api-tls\") pod \"maas-api-e2e-ait-49eb9be1-5bd9858644-kf427\" (UID: \"b25b8726-066a-408a-908e-2ceddf777165\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427"
Sep 09 16:33:48.380797 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.380766    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25b8726-066a-408a-908e-2ceddf777165-openshift-service-ca\") pod \"maas-api-e2e-ait-49eb9be1-5bd9858644-kf427\" (UID: \"b25b8726-066a-408a-908e-2ceddf777165\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427"
Sep 09 16:33:48.382946 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.382918    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25b8726-066a-408a-908e-2ceddf777165-maas-api-tls\") pod \"maas-api-e2e-ait-49eb9be1-5bd9858644-kf427\" (UID: \"b25b8726-066a-408a-908e-2ceddf777165\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427"
Sep 09 16:33:48.383126 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.383102    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b25b8726-066a-408a-908e-2ceddf777165-metrics-certs\") pod \"maas-api-e2e-ait-49eb9be1-5bd9858644-kf427\" (UID: \"b25b8726-066a-408a-908e-2ceddf777165\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427"
Sep 09 16:33:48.387588 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.387562    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln4sp\" (UniqueName: \"kubernetes.io/projected/b25b8726-066a-408a-908e-2ceddf777165-kube-api-access-ln4sp\") pod \"maas-api-e2e-ait-49eb9be1-5bd9858644-kf427\" (UID: \"b25b8726-066a-408a-908e-2ceddf777165\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427"
Sep 09 16:33:48.409553 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.409508    2555 generic.go:356] "Generic (PLEG): container finished" podID="f7243d0b-acfe-40f5-9ab9-e03c161a8c90" containerID="88c1bfd3f6c91e43ebddfbcb949533bd16c9a6f8263ef2e264d5ab67a41b9a00" exitCode=0
Sep 09 16:33:48.409721 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.409576    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"
Sep 09 16:33:48.409721 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.409573    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf" event={"ID":"f7243d0b-acfe-40f5-9ab9-e03c161a8c90","Type":"ContainerDied","Data":"88c1bfd3f6c91e43ebddfbcb949533bd16c9a6f8263ef2e264d5ab67a41b9a00"}
Sep 09 16:33:48.409812 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.409727    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf" event={"ID":"f7243d0b-acfe-40f5-9ab9-e03c161a8c90","Type":"ContainerDied","Data":"aae31fd0e89ed1d30bdb6e19f8b24f700ed5b7d8f939106e522ab8a6fb186093"}
Sep 09 16:33:48.409812 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.409765    2555 scope.go:117] "RemoveContainer" containerID="88c1bfd3f6c91e43ebddfbcb949533bd16c9a6f8263ef2e264d5ab67a41b9a00"
Sep 09 16:33:48.419862 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.419843    2555 scope.go:117] "RemoveContainer" containerID="00f895b50ff6d8cc477f0966c7ac909642cc1b3a01e22d18f8432bd69b2a7950"
Sep 09 16:33:48.428820 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.428794    2555 scope.go:117] "RemoveContainer" containerID="88c1bfd3f6c91e43ebddfbcb949533bd16c9a6f8263ef2e264d5ab67a41b9a00"
Sep 09 16:33:48.429279 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:33:48.429255    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88c1bfd3f6c91e43ebddfbcb949533bd16c9a6f8263ef2e264d5ab67a41b9a00\": container with ID starting with 88c1bfd3f6c91e43ebddfbcb949533bd16c9a6f8263ef2e264d5ab67a41b9a00 not found: ID does not exist" containerID="88c1bfd3f6c91e43ebddfbcb949533bd16c9a6f8263ef2e264d5ab67a41b9a00"
Sep 09 16:33:48.429378 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.429308    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c1bfd3f6c91e43ebddfbcb949533bd16c9a6f8263ef2e264d5ab67a41b9a00"} err="failed to get container status \"88c1bfd3f6c91e43ebddfbcb949533bd16c9a6f8263ef2e264d5ab67a41b9a00\": rpc error: code = NotFound desc = could not find container \"88c1bfd3f6c91e43ebddfbcb949533bd16c9a6f8263ef2e264d5ab67a41b9a00\": container with ID starting with 88c1bfd3f6c91e43ebddfbcb949533bd16c9a6f8263ef2e264d5ab67a41b9a00 not found: ID does not exist"
Sep 09 16:33:48.429378 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.429333    2555 scope.go:117] "RemoveContainer" containerID="00f895b50ff6d8cc477f0966c7ac909642cc1b3a01e22d18f8432bd69b2a7950"
Sep 09 16:33:48.429632 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:33:48.429609    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00f895b50ff6d8cc477f0966c7ac909642cc1b3a01e22d18f8432bd69b2a7950\": container with ID starting with 00f895b50ff6d8cc477f0966c7ac909642cc1b3a01e22d18f8432bd69b2a7950 not found: ID does not exist" containerID="00f895b50ff6d8cc477f0966c7ac909642cc1b3a01e22d18f8432bd69b2a7950"
Sep 09 16:33:48.429695 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.429640    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00f895b50ff6d8cc477f0966c7ac909642cc1b3a01e22d18f8432bd69b2a7950"} err="failed to get container status \"00f895b50ff6d8cc477f0966c7ac909642cc1b3a01e22d18f8432bd69b2a7950\": rpc error: code = NotFound desc = could not find container \"00f895b50ff6d8cc477f0966c7ac909642cc1b3a01e22d18f8432bd69b2a7950\": container with ID starting with 00f895b50ff6d8cc477f0966c7ac909642cc1b3a01e22d18f8432bd69b2a7950 not found: ID does not exist"
Sep 09 16:33:48.432147 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.432079    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"]
Sep 09 16:33:48.434666 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.434645    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-9e0f78a3-gw-openshift-default-76546779d4-qldmf"]
Sep 09 16:33:48.487717 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.487669    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427"
Sep 09 16:33:48.612940 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.612839    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7243d0b-acfe-40f5-9ab9-e03c161a8c90" path="/var/lib/kubelet/pods/f7243d0b-acfe-40f5-9ab9-e03c161a8c90/volumes"
Sep 09 16:33:48.782622 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:48.782440    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427"]
Sep 09 16:33:48.783498 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:33:48.783400    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25b8726_066a_408a_908e_2ceddf777165.slice/crio-7c67f7ac0fd26cb1291d2a26ce30a6b94cc03f850eecf98383c791bdb8cbe7fe WatchSource:0}: Error finding container 7c67f7ac0fd26cb1291d2a26ce30a6b94cc03f850eecf98383c791bdb8cbe7fe: Status 404 returned error can't find the container with id 7c67f7ac0fd26cb1291d2a26ce30a6b94cc03f850eecf98383c791bdb8cbe7fe
Sep 09 16:33:49.419319 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:49.419262    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427" event={"ID":"b25b8726-066a-408a-908e-2ceddf777165","Type":"ContainerStarted","Data":"68b7d4fc384a4c61fe4cf2139e739dcd0e9413cc8d194554f467e858de98f15d"}
Sep 09 16:33:49.419701 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:49.419334    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427" event={"ID":"b25b8726-066a-408a-908e-2ceddf777165","Type":"ContainerStarted","Data":"7c67f7ac0fd26cb1291d2a26ce30a6b94cc03f850eecf98383c791bdb8cbe7fe"}
Sep 09 16:33:49.419701 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:49.419357    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427"
Sep 09 16:33:49.437191 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:49.437137    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427" podStartSLOduration=1.010197628 podStartE2EDuration="1.437120808s" totalImagesPullingTime="426.92318ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:33:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:33:49.436744257 +0000 UTC m=+1195.266952318" watchObservedRunningTime="2026-09-09 16:33:49.437120808 +0000 UTC m=+1195.267328880"
Sep 09 16:33:54.627132 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:54.627100    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgcjl_561d9dd7-04c3-42dc-bcf6-2c92fd61706b/console-operator/1.log"
Sep 09 16:33:54.630107 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:54.630085    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgcjl_561d9dd7-04c3-42dc-bcf6-2c92fd61706b/console-operator/1.log"
Sep 09 16:33:55.430750 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:33:55.430719    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427"
Sep 09 16:34:05.099406 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:05.099369    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:34:17.630751 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:17.630706    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427"]
Sep 09 16:34:17.631205 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:17.630959    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427" podUID="b25b8726-066a-408a-908e-2ceddf777165" containerName="maas-api" containerID="cri-o://68b7d4fc384a4c61fe4cf2139e739dcd0e9413cc8d194554f467e858de98f15d" gracePeriod=30
Sep 09 16:34:17.886544 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:17.886486    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427"
Sep 09 16:34:17.969990 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:17.969958    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25b8726-066a-408a-908e-2ceddf777165-openshift-service-ca\") pod \"b25b8726-066a-408a-908e-2ceddf777165\" (UID: \"b25b8726-066a-408a-908e-2ceddf777165\") "
Sep 09 16:34:17.970189 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:17.970028    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25b8726-066a-408a-908e-2ceddf777165-maas-api-tls\") pod \"b25b8726-066a-408a-908e-2ceddf777165\" (UID: \"b25b8726-066a-408a-908e-2ceddf777165\") "
Sep 09 16:34:17.970189 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:17.970089    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ln4sp\" (UniqueName: \"kubernetes.io/projected/b25b8726-066a-408a-908e-2ceddf777165-kube-api-access-ln4sp\") pod \"b25b8726-066a-408a-908e-2ceddf777165\" (UID: \"b25b8726-066a-408a-908e-2ceddf777165\") "
Sep 09 16:34:17.970189 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:17.970172    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b25b8726-066a-408a-908e-2ceddf777165-metrics-certs\") pod \"b25b8726-066a-408a-908e-2ceddf777165\" (UID: \"b25b8726-066a-408a-908e-2ceddf777165\") "
Sep 09 16:34:17.970436 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:17.970410    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b25b8726-066a-408a-908e-2ceddf777165-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b25b8726-066a-408a-908e-2ceddf777165" (UID: "b25b8726-066a-408a-908e-2ceddf777165"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:34:17.972263 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:17.972231    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b25b8726-066a-408a-908e-2ceddf777165-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b25b8726-066a-408a-908e-2ceddf777165" (UID: "b25b8726-066a-408a-908e-2ceddf777165"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:34:17.972446 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:17.972280    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25b8726-066a-408a-908e-2ceddf777165-kube-api-access-ln4sp" (OuterVolumeSpecName: "kube-api-access-ln4sp") pod "b25b8726-066a-408a-908e-2ceddf777165" (UID: "b25b8726-066a-408a-908e-2ceddf777165"). InnerVolumeSpecName "kube-api-access-ln4sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:34:17.972446 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:17.972380    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b25b8726-066a-408a-908e-2ceddf777165-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b25b8726-066a-408a-908e-2ceddf777165" (UID: "b25b8726-066a-408a-908e-2ceddf777165"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:34:18.071654 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:18.071617    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln4sp\" (UniqueName: \"kubernetes.io/projected/b25b8726-066a-408a-908e-2ceddf777165-kube-api-access-ln4sp\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:34:18.071654 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:18.071651    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b25b8726-066a-408a-908e-2ceddf777165-metrics-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:34:18.071654 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:18.071661    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25b8726-066a-408a-908e-2ceddf777165-openshift-service-ca\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:34:18.071654 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:18.071670    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25b8726-066a-408a-908e-2ceddf777165-maas-api-tls\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:34:18.591320 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:18.591270    2555 generic.go:356] "Generic (PLEG): container finished" podID="b25b8726-066a-408a-908e-2ceddf777165" containerID="68b7d4fc384a4c61fe4cf2139e739dcd0e9413cc8d194554f467e858de98f15d" exitCode=0
Sep 09 16:34:18.591521 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:18.591341    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427"
Sep 09 16:34:18.591521 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:18.591367    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427" event={"ID":"b25b8726-066a-408a-908e-2ceddf777165","Type":"ContainerDied","Data":"68b7d4fc384a4c61fe4cf2139e739dcd0e9413cc8d194554f467e858de98f15d"}
Sep 09 16:34:18.591521 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:18.591452    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427" event={"ID":"b25b8726-066a-408a-908e-2ceddf777165","Type":"ContainerDied","Data":"7c67f7ac0fd26cb1291d2a26ce30a6b94cc03f850eecf98383c791bdb8cbe7fe"}
Sep 09 16:34:18.591521 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:18.591473    2555 scope.go:117] "RemoveContainer" containerID="68b7d4fc384a4c61fe4cf2139e739dcd0e9413cc8d194554f467e858de98f15d"
Sep 09 16:34:18.601404 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:18.601385    2555 scope.go:117] "RemoveContainer" containerID="68b7d4fc384a4c61fe4cf2139e739dcd0e9413cc8d194554f467e858de98f15d"
Sep 09 16:34:18.601711 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:34:18.601685    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68b7d4fc384a4c61fe4cf2139e739dcd0e9413cc8d194554f467e858de98f15d\": container with ID starting with 68b7d4fc384a4c61fe4cf2139e739dcd0e9413cc8d194554f467e858de98f15d not found: ID does not exist" containerID="68b7d4fc384a4c61fe4cf2139e739dcd0e9413cc8d194554f467e858de98f15d"
Sep 09 16:34:18.601782 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:18.601723    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b7d4fc384a4c61fe4cf2139e739dcd0e9413cc8d194554f467e858de98f15d"} err="failed to get container status \"68b7d4fc384a4c61fe4cf2139e739dcd0e9413cc8d194554f467e858de98f15d\": rpc error: code = NotFound desc = could not find container \"68b7d4fc384a4c61fe4cf2139e739dcd0e9413cc8d194554f467e858de98f15d\": container with ID starting with 68b7d4fc384a4c61fe4cf2139e739dcd0e9413cc8d194554f467e858de98f15d not found: ID does not exist"
Sep 09 16:34:18.613552 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:18.613530    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427"]
Sep 09 16:34:18.617418 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:18.617399    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49eb9be1-5bd9858644-kf427"]
Sep 09 16:34:19.716935 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.716897    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"]
Sep 09 16:34:19.717548 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.717527    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b25b8726-066a-408a-908e-2ceddf777165" containerName="maas-api"
Sep 09 16:34:19.717602 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.717553    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b25b8726-066a-408a-908e-2ceddf777165" containerName="maas-api"
Sep 09 16:34:19.717668 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.717656    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b25b8726-066a-408a-908e-2ceddf777165" containerName="maas-api"
Sep 09 16:34:19.720672 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.720653    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.723640 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.723455    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-20ed533b-openshift-default-dockercfg-vvcrj\""
Sep 09 16:34:19.732976 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.732950    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"]
Sep 09 16:34:19.787156 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.787125    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-istio-envoy\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.787156 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.787161    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0089347f-76b8-43a7-9dab-0def01e053e5-istiod-ca-cert\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.787424 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.787187    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-istio-data\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.787424 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.787230    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0089347f-76b8-43a7-9dab-0def01e053e5-istio-podinfo\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.787424 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.787276    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-credential-socket\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.787424 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.787328    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-workload-certs\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.787424 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.787380    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-workload-socket\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.787424 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.787396    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2lvb\" (UniqueName: \"kubernetes.io/projected/0089347f-76b8-43a7-9dab-0def01e053e5-kube-api-access-z2lvb\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.787424 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.787426    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0089347f-76b8-43a7-9dab-0def01e053e5-istio-token\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.888194 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.888142    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-credential-socket\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.888481 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.888197    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-workload-certs\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.888481 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.888252    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-workload-socket\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.888481 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.888283    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2lvb\" (UniqueName: \"kubernetes.io/projected/0089347f-76b8-43a7-9dab-0def01e053e5-kube-api-access-z2lvb\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.888481 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.888376    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0089347f-76b8-43a7-9dab-0def01e053e5-istio-token\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.888481 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.888434    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-istio-envoy\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.888481 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.888460    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0089347f-76b8-43a7-9dab-0def01e053e5-istiod-ca-cert\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.888846 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.888513    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-istio-data\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.888846 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.888541    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0089347f-76b8-43a7-9dab-0def01e053e5-istio-podinfo\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.888846 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.888624    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-credential-socket\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.889073 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.888897    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-workload-socket\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.889073 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.888935    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-workload-certs\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.889194 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.889091    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-istio-data\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.889389 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.889372    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0089347f-76b8-43a7-9dab-0def01e053e5-istiod-ca-cert\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.891017 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.890994    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-istio-envoy\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.891223 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.891207    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0089347f-76b8-43a7-9dab-0def01e053e5-istio-podinfo\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.896202 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.896177    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0089347f-76b8-43a7-9dab-0def01e053e5-istio-token\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:19.896411 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:19.896395    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2lvb\" (UniqueName: \"kubernetes.io/projected/0089347f-76b8-43a7-9dab-0def01e053e5-kube-api-access-z2lvb\") pod \"e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") " pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:20.036040 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:20.035997    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:20.313571 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:20.313545    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"]
Sep 09 16:34:20.315471 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:34:20.315441    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0089347f_76b8_43a7_9dab_0def01e053e5.slice/crio-bffd40aa0d8406ec1cb31cb8ac922d9028f5228f0e5410b37ded299742692da4 WatchSource:0}: Error finding container bffd40aa0d8406ec1cb31cb8ac922d9028f5228f0e5410b37ded299742692da4: Status 404 returned error can't find the container with id bffd40aa0d8406ec1cb31cb8ac922d9028f5228f0e5410b37ded299742692da4
Sep 09 16:34:20.317978 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:20.317936    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:34:20.318057 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:20.318036    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:34:20.318099 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:20.318081    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:34:20.611504 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:20.609804    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b25b8726-066a-408a-908e-2ceddf777165" path="/var/lib/kubelet/pods/b25b8726-066a-408a-908e-2ceddf777165/volumes"
Sep 09 16:34:20.611504 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:20.610212    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2" event={"ID":"0089347f-76b8-43a7-9dab-0def01e053e5","Type":"ContainerStarted","Data":"4b85145374fc462830b14a67eb3dea3c2fd3913ec35acdbcc17bfcdb72cce05c"}
Sep 09 16:34:20.611504 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:20.610260    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2" event={"ID":"0089347f-76b8-43a7-9dab-0def01e053e5","Type":"ContainerStarted","Data":"bffd40aa0d8406ec1cb31cb8ac922d9028f5228f0e5410b37ded299742692da4"}
Sep 09 16:34:20.632079 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:20.632031    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2" podStartSLOduration=1.632017643 podStartE2EDuration="1.632017643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:34:20.629780978 +0000 UTC m=+1226.459989038" watchObservedRunningTime="2026-09-09 16:34:20.632017643 +0000 UTC m=+1226.462225701"
Sep 09 16:34:21.037115 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:21.037075    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:21.042452 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:21.042424    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:21.487923 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:21.487831    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:34:21.613288 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:21.613256    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:21.614597 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:21.614577    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:24.607971 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:24.607939    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s"]
Sep 09 16:34:24.611355 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:24.611332    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s"
Sep 09 16:34:24.613944 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:24.613919    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-20ed533b-dockercfg-ssx6b\""
Sep 09 16:34:24.614077 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:24.613919    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-20ed533b\""
Sep 09 16:34:24.615380 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:24.615356    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s"]
Sep 09 16:34:24.733555 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:24.733522    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfqm9\" (UniqueName: \"kubernetes.io/projected/c22ca34f-0971-4d10-ae1a-bee3cfbf3452-kube-api-access-kfqm9\") pod \"payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s\" (UID: \"c22ca34f-0971-4d10-ae1a-bee3cfbf3452\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s"
Sep 09 16:34:24.733727 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:24.733564    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22ca34f-0971-4d10-ae1a-bee3cfbf3452-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s\" (UID: \"c22ca34f-0971-4d10-ae1a-bee3cfbf3452\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s"
Sep 09 16:34:24.834261 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:24.834233    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfqm9\" (UniqueName: \"kubernetes.io/projected/c22ca34f-0971-4d10-ae1a-bee3cfbf3452-kube-api-access-kfqm9\") pod \"payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s\" (UID: \"c22ca34f-0971-4d10-ae1a-bee3cfbf3452\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s"
Sep 09 16:34:24.834444 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:24.834270    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22ca34f-0971-4d10-ae1a-bee3cfbf3452-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s\" (UID: \"c22ca34f-0971-4d10-ae1a-bee3cfbf3452\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s"
Sep 09 16:34:24.835028 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:24.835007    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22ca34f-0971-4d10-ae1a-bee3cfbf3452-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s\" (UID: \"c22ca34f-0971-4d10-ae1a-bee3cfbf3452\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s"
Sep 09 16:34:24.841848 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:24.841821    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfqm9\" (UniqueName: \"kubernetes.io/projected/c22ca34f-0971-4d10-ae1a-bee3cfbf3452-kube-api-access-kfqm9\") pod \"payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s\" (UID: \"c22ca34f-0971-4d10-ae1a-bee3cfbf3452\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s"
Sep 09 16:34:24.923983 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:24.923903    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s"
Sep 09 16:34:25.194580 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:25.194550    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s"]
Sep 09 16:34:25.196089 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:34:25.196065    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22ca34f_0971_4d10_ae1a_bee3cfbf3452.slice/crio-e7eb8df02aa66d76a5a2d936d1dc19a2b77d32cf15a66a0db952953cd4a86829 WatchSource:0}: Error finding container e7eb8df02aa66d76a5a2d936d1dc19a2b77d32cf15a66a0db952953cd4a86829: Status 404 returned error can't find the container with id e7eb8df02aa66d76a5a2d936d1dc19a2b77d32cf15a66a0db952953cd4a86829
Sep 09 16:34:25.637240 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:25.637205    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s" event={"ID":"c22ca34f-0971-4d10-ae1a-bee3cfbf3452","Type":"ContainerStarted","Data":"dcbb0bcf96093fcee2b068eca8ff0465287a0ac5801ed9b1df245825fbf099f1"}
Sep 09 16:34:25.637659 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:25.637259    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s" event={"ID":"c22ca34f-0971-4d10-ae1a-bee3cfbf3452","Type":"ContainerStarted","Data":"e7eb8df02aa66d76a5a2d936d1dc19a2b77d32cf15a66a0db952953cd4a86829"}
Sep 09 16:34:25.637659 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:25.637365    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s"
Sep 09 16:34:25.652455 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:25.652406    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s" podStartSLOduration=1.652392095 podStartE2EDuration="1.652392095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:34:25.651656123 +0000 UTC m=+1231.481864205" watchObservedRunningTime="2026-09-09 16:34:25.652392095 +0000 UTC m=+1231.482600153"
Sep 09 16:34:36.648026 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:36.647927    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s"
Sep 09 16:34:37.394224 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.394182    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"]
Sep 09 16:34:37.415098 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.415056    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"]
Sep 09 16:34:37.415312 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.415199    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:34:37.418015 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.417979    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-05e496\"/\"kube-root-ca.crt\""
Sep 09 16:34:37.419730 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.419472    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-05e496\"/\"default-dockercfg-srl4d\""
Sep 09 16:34:37.419730 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.419532    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-05e496\"/\"openshift-service-ca.crt\""
Sep 09 16:34:37.420368 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.420076    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-05e496\"/\"autores-model-05e496-kserve-self-signed-certs\""
Sep 09 16:34:37.456763 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.456730    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991e3c32-3fc6-4288-b22d-100e6ac79fc0-tmp-dir\") pod \"autores-model-05e496-kserve-56db6979-tzbh6\" (UID: \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\") " pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:34:37.456918 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.456774    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/991e3c32-3fc6-4288-b22d-100e6ac79fc0-tls-certs\") pod \"autores-model-05e496-kserve-56db6979-tzbh6\" (UID: \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\") " pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:34:37.456918 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.456816    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/991e3c32-3fc6-4288-b22d-100e6ac79fc0-home\") pod \"autores-model-05e496-kserve-56db6979-tzbh6\" (UID: \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\") " pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:34:37.456918 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.456841    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/991e3c32-3fc6-4288-b22d-100e6ac79fc0-dshm\") pod \"autores-model-05e496-kserve-56db6979-tzbh6\" (UID: \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\") " pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:34:37.456918 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.456871    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/991e3c32-3fc6-4288-b22d-100e6ac79fc0-model-cache\") pod \"autores-model-05e496-kserve-56db6979-tzbh6\" (UID: \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\") " pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:34:37.457063 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.456995    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp8tk\" (UniqueName: \"kubernetes.io/projected/991e3c32-3fc6-4288-b22d-100e6ac79fc0-kube-api-access-cp8tk\") pod \"autores-model-05e496-kserve-56db6979-tzbh6\" (UID: \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\") " pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:34:37.558215 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.558175    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/991e3c32-3fc6-4288-b22d-100e6ac79fc0-model-cache\") pod \"autores-model-05e496-kserve-56db6979-tzbh6\" (UID: \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\") " pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:34:37.558438 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.558277    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp8tk\" (UniqueName: \"kubernetes.io/projected/991e3c32-3fc6-4288-b22d-100e6ac79fc0-kube-api-access-cp8tk\") pod \"autores-model-05e496-kserve-56db6979-tzbh6\" (UID: \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\") " pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:34:37.558438 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.558356    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991e3c32-3fc6-4288-b22d-100e6ac79fc0-tmp-dir\") pod \"autores-model-05e496-kserve-56db6979-tzbh6\" (UID: \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\") " pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:34:37.558438 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.558391    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/991e3c32-3fc6-4288-b22d-100e6ac79fc0-tls-certs\") pod \"autores-model-05e496-kserve-56db6979-tzbh6\" (UID: \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\") " pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:34:37.558438 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.558426    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/991e3c32-3fc6-4288-b22d-100e6ac79fc0-home\") pod \"autores-model-05e496-kserve-56db6979-tzbh6\" (UID: \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\") " pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:34:37.558681 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.558457    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/991e3c32-3fc6-4288-b22d-100e6ac79fc0-dshm\") pod \"autores-model-05e496-kserve-56db6979-tzbh6\" (UID: \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\") " pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:34:37.558757 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.558731    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/991e3c32-3fc6-4288-b22d-100e6ac79fc0-model-cache\") pod \"autores-model-05e496-kserve-56db6979-tzbh6\" (UID: \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\") " pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:34:37.558825 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.558759    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991e3c32-3fc6-4288-b22d-100e6ac79fc0-tmp-dir\") pod \"autores-model-05e496-kserve-56db6979-tzbh6\" (UID: \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\") " pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:34:37.558871 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.558830    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/991e3c32-3fc6-4288-b22d-100e6ac79fc0-home\") pod \"autores-model-05e496-kserve-56db6979-tzbh6\" (UID: \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\") " pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:34:37.560917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.560886    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/991e3c32-3fc6-4288-b22d-100e6ac79fc0-dshm\") pod \"autores-model-05e496-kserve-56db6979-tzbh6\" (UID: \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\") " pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:34:37.561199 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.561177    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/991e3c32-3fc6-4288-b22d-100e6ac79fc0-tls-certs\") pod \"autores-model-05e496-kserve-56db6979-tzbh6\" (UID: \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\") " pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:34:37.566463 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.566431    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp8tk\" (UniqueName: \"kubernetes.io/projected/991e3c32-3fc6-4288-b22d-100e6ac79fc0-kube-api-access-cp8tk\") pod \"autores-model-05e496-kserve-56db6979-tzbh6\" (UID: \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\") " pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:34:37.656544 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.656450    2555 patch_prober.go:28] interesting pod/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 16:34:37.656544 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.656524    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2" podUID="0089347f-76b8-43a7-9dab-0def01e053e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 16:34:37.727985 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:37.727939    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:34:38.005364 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:38.005326    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"]
Sep 09 16:34:38.006472 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:34:38.006430    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod991e3c32_3fc6_4288_b22d_100e6ac79fc0.slice/crio-770e3158c2348e09116d435f8a5ae4d9fa09cc42d270038993279e60db4d5354 WatchSource:0}: Error finding container 770e3158c2348e09116d435f8a5ae4d9fa09cc42d270038993279e60db4d5354: Status 404 returned error can't find the container with id 770e3158c2348e09116d435f8a5ae4d9fa09cc42d270038993279e60db4d5354
Sep 09 16:34:38.100089 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:38.100050    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:34:38.724455 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:38.724413    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6" event={"ID":"991e3c32-3fc6-4288-b22d-100e6ac79fc0","Type":"ContainerStarted","Data":"3b502bc80d57c849013722ca96828d8f7debe9bb7fa19b4670b6fd5c5b8d4c26"}
Sep 09 16:34:38.724845 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:38.724465    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6" event={"ID":"991e3c32-3fc6-4288-b22d-100e6ac79fc0","Type":"ContainerStarted","Data":"770e3158c2348e09116d435f8a5ae4d9fa09cc42d270038993279e60db4d5354"}
Sep 09 16:34:38.745150 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:38.745072    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6" podStartSLOduration=1.7450464700000001 podStartE2EDuration="1.74504647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:34:38.740395998 +0000 UTC m=+1244.570604059" watchObservedRunningTime="2026-09-09 16:34:38.74504647 +0000 UTC m=+1244.575254563"
Sep 09 16:34:39.731921 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:39.731886    2555 generic.go:356] "Generic (PLEG): container finished" podID="0089347f-76b8-43a7-9dab-0def01e053e5" containerID="4b85145374fc462830b14a67eb3dea3c2fd3913ec35acdbcc17bfcdb72cce05c" exitCode=0
Sep 09 16:34:39.732402 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:39.731961    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2" event={"ID":"0089347f-76b8-43a7-9dab-0def01e053e5","Type":"ContainerDied","Data":"4b85145374fc462830b14a67eb3dea3c2fd3913ec35acdbcc17bfcdb72cce05c"}
Sep 09 16:34:39.732579 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:39.732561    2555 scope.go:117] "RemoveContainer" containerID="4b85145374fc462830b14a67eb3dea3c2fd3913ec35acdbcc17bfcdb72cce05c"
Sep 09 16:34:39.733636 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:39.733603    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:34:39.733704 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:39.733680    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:34:39.733740 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:39.733710    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:34:40.740420 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:40.740379    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2" event={"ID":"0089347f-76b8-43a7-9dab-0def01e053e5","Type":"ContainerStarted","Data":"c1b025892b4766072ed7cab6ad9454acd1cc782859dc1689c882915fc9247260"}
Sep 09 16:34:41.037090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:41.037000    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:41.037246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:41.037220    2555 patch_prober.go:28] interesting pod/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 09 16:34:41.037323 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:41.037271    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2" podUID="0089347f-76b8-43a7-9dab-0def01e053e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 09 16:34:42.036915 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:42.036767    2555 patch_prober.go:28] interesting pod/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 09 16:34:42.036915 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:42.036833    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2" podUID="0089347f-76b8-43a7-9dab-0def01e053e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 09 16:34:43.036772 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:43.036735    2555 patch_prober.go:28] interesting pod/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 09 16:34:43.036956 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:43.036793    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2" podUID="0089347f-76b8-43a7-9dab-0def01e053e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 09 16:34:44.036964 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:44.036926    2555 patch_prober.go:28] interesting pod/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 09 16:34:44.037384 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:44.036984    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2" podUID="0089347f-76b8-43a7-9dab-0def01e053e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 09 16:34:45.037177 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:45.037133    2555 patch_prober.go:28] interesting pod/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 09 16:34:45.037624 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:45.037202    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2" podUID="0089347f-76b8-43a7-9dab-0def01e053e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 09 16:34:46.037213 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:46.037171    2555 patch_prober.go:28] interesting pod/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 09 16:34:46.037658 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:46.037239    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2" podUID="0089347f-76b8-43a7-9dab-0def01e053e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 09 16:34:47.040435 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:47.040335    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:47.040884 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:47.040704    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:47.041539 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:47.041509    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:47.728994 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:47.728949    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:34:47.728994 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:47.729009    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:34:47.742967 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:47.742933    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:34:47.818127 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:47.818093    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:34:52.992187 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:52.992142    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:34:53.907715 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:53.907654    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s"]
Sep 09 16:34:53.908247 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:53.907965    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s" podUID="c22ca34f-0971-4d10-ae1a-bee3cfbf3452" containerName="payload-pre-processing" containerID="cri-o://dcbb0bcf96093fcee2b068eca8ff0465287a0ac5801ed9b1df245825fbf099f1" gracePeriod=30
Sep 09 16:34:54.190340 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:54.190272    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s"
Sep 09 16:34:54.210425 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:54.210351    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"]
Sep 09 16:34:54.210987 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:54.210717    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2" podUID="0089347f-76b8-43a7-9dab-0def01e053e5" containerName="istio-proxy" containerID="cri-o://c1b025892b4766072ed7cab6ad9454acd1cc782859dc1689c882915fc9247260" gracePeriod=30
Sep 09 16:34:54.328500 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:54.328464    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22ca34f-0971-4d10-ae1a-bee3cfbf3452-plugins-config-volume\") pod \"c22ca34f-0971-4d10-ae1a-bee3cfbf3452\" (UID: \"c22ca34f-0971-4d10-ae1a-bee3cfbf3452\") "
Sep 09 16:34:54.328703 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:54.328520    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfqm9\" (UniqueName: \"kubernetes.io/projected/c22ca34f-0971-4d10-ae1a-bee3cfbf3452-kube-api-access-kfqm9\") pod \"c22ca34f-0971-4d10-ae1a-bee3cfbf3452\" (UID: \"c22ca34f-0971-4d10-ae1a-bee3cfbf3452\") "
Sep 09 16:34:54.328897 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:54.328867    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22ca34f-0971-4d10-ae1a-bee3cfbf3452-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c22ca34f-0971-4d10-ae1a-bee3cfbf3452" (UID: "c22ca34f-0971-4d10-ae1a-bee3cfbf3452"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:34:54.331038 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:54.331001    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22ca34f-0971-4d10-ae1a-bee3cfbf3452-kube-api-access-kfqm9" (OuterVolumeSpecName: "kube-api-access-kfqm9") pod "c22ca34f-0971-4d10-ae1a-bee3cfbf3452" (UID: "c22ca34f-0971-4d10-ae1a-bee3cfbf3452"). InnerVolumeSpecName "kube-api-access-kfqm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:34:54.429350 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:54.429204    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c22ca34f-0971-4d10-ae1a-bee3cfbf3452-plugins-config-volume\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:34:54.429350 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:54.429245    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfqm9\" (UniqueName: \"kubernetes.io/projected/c22ca34f-0971-4d10-ae1a-bee3cfbf3452-kube-api-access-kfqm9\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:34:54.851182 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:54.851149    2555 generic.go:356] "Generic (PLEG): container finished" podID="c22ca34f-0971-4d10-ae1a-bee3cfbf3452" containerID="dcbb0bcf96093fcee2b068eca8ff0465287a0ac5801ed9b1df245825fbf099f1" exitCode=0
Sep 09 16:34:54.851388 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:54.851201    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s" event={"ID":"c22ca34f-0971-4d10-ae1a-bee3cfbf3452","Type":"ContainerDied","Data":"dcbb0bcf96093fcee2b068eca8ff0465287a0ac5801ed9b1df245825fbf099f1"}
Sep 09 16:34:54.851388 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:54.851211    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s"
Sep 09 16:34:54.851388 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:54.851251    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s" event={"ID":"c22ca34f-0971-4d10-ae1a-bee3cfbf3452","Type":"ContainerDied","Data":"e7eb8df02aa66d76a5a2d936d1dc19a2b77d32cf15a66a0db952953cd4a86829"}
Sep 09 16:34:54.851388 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:54.851277    2555 scope.go:117] "RemoveContainer" containerID="dcbb0bcf96093fcee2b068eca8ff0465287a0ac5801ed9b1df245825fbf099f1"
Sep 09 16:34:54.860854 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:54.860832    2555 scope.go:117] "RemoveContainer" containerID="dcbb0bcf96093fcee2b068eca8ff0465287a0ac5801ed9b1df245825fbf099f1"
Sep 09 16:34:54.861165 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:34:54.861142    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcbb0bcf96093fcee2b068eca8ff0465287a0ac5801ed9b1df245825fbf099f1\": container with ID starting with dcbb0bcf96093fcee2b068eca8ff0465287a0ac5801ed9b1df245825fbf099f1 not found: ID does not exist" containerID="dcbb0bcf96093fcee2b068eca8ff0465287a0ac5801ed9b1df245825fbf099f1"
Sep 09 16:34:54.861237 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:54.861181    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcbb0bcf96093fcee2b068eca8ff0465287a0ac5801ed9b1df245825fbf099f1"} err="failed to get container status \"dcbb0bcf96093fcee2b068eca8ff0465287a0ac5801ed9b1df245825fbf099f1\": rpc error: code = NotFound desc = could not find container \"dcbb0bcf96093fcee2b068eca8ff0465287a0ac5801ed9b1df245825fbf099f1\": container with ID starting with dcbb0bcf96093fcee2b068eca8ff0465287a0ac5801ed9b1df245825fbf099f1 not found: ID does not exist"
Sep 09 16:34:54.868053 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:54.868025    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s"]
Sep 09 16:34:54.873814 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:54.873782    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-20ed533b-65c4757c74-srq9s"]
Sep 09 16:34:56.609950 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:56.609914    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c22ca34f-0971-4d10-ae1a-bee3cfbf3452" path="/var/lib/kubelet/pods/c22ca34f-0971-4d10-ae1a-bee3cfbf3452/volumes"
Sep 09 16:34:59.490769 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.490741    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:59.585375 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.585336    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2lvb\" (UniqueName: \"kubernetes.io/projected/0089347f-76b8-43a7-9dab-0def01e053e5-kube-api-access-z2lvb\") pod \"0089347f-76b8-43a7-9dab-0def01e053e5\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") "
Sep 09 16:34:59.585577 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.585395    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-workload-socket\") pod \"0089347f-76b8-43a7-9dab-0def01e053e5\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") "
Sep 09 16:34:59.585577 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.585432    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-istio-envoy\") pod \"0089347f-76b8-43a7-9dab-0def01e053e5\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") "
Sep 09 16:34:59.585577 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.585456    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0089347f-76b8-43a7-9dab-0def01e053e5-istio-podinfo\") pod \"0089347f-76b8-43a7-9dab-0def01e053e5\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") "
Sep 09 16:34:59.585577 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.585503    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-credential-socket\") pod \"0089347f-76b8-43a7-9dab-0def01e053e5\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") "
Sep 09 16:34:59.585577 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.585556    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-workload-certs\") pod \"0089347f-76b8-43a7-9dab-0def01e053e5\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") "
Sep 09 16:34:59.585860 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.585605    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0089347f-76b8-43a7-9dab-0def01e053e5-istiod-ca-cert\") pod \"0089347f-76b8-43a7-9dab-0def01e053e5\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") "
Sep 09 16:34:59.585860 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.585631    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0089347f-76b8-43a7-9dab-0def01e053e5-istio-token\") pod \"0089347f-76b8-43a7-9dab-0def01e053e5\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") "
Sep 09 16:34:59.585860 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.585684    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-istio-data\") pod \"0089347f-76b8-43a7-9dab-0def01e053e5\" (UID: \"0089347f-76b8-43a7-9dab-0def01e053e5\") "
Sep 09 16:34:59.585860 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.585731    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0089347f-76b8-43a7-9dab-0def01e053e5" (UID: "0089347f-76b8-43a7-9dab-0def01e053e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:34:59.585860 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.585773    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0089347f-76b8-43a7-9dab-0def01e053e5" (UID: "0089347f-76b8-43a7-9dab-0def01e053e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:34:59.586123 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.586027    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-workload-socket\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:34:59.586123 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.586049    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-credential-socket\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:34:59.586123 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.586068    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0089347f-76b8-43a7-9dab-0def01e053e5" (UID: "0089347f-76b8-43a7-9dab-0def01e053e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:34:59.586123 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.586076    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-istio-data" (OuterVolumeSpecName: "istio-data") pod "0089347f-76b8-43a7-9dab-0def01e053e5" (UID: "0089347f-76b8-43a7-9dab-0def01e053e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:34:59.586362 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.586239    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0089347f-76b8-43a7-9dab-0def01e053e5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0089347f-76b8-43a7-9dab-0def01e053e5" (UID: "0089347f-76b8-43a7-9dab-0def01e053e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:34:59.587752 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.587724    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0089347f-76b8-43a7-9dab-0def01e053e5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0089347f-76b8-43a7-9dab-0def01e053e5" (UID: "0089347f-76b8-43a7-9dab-0def01e053e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 16:34:59.587866 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.587805    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0089347f-76b8-43a7-9dab-0def01e053e5-kube-api-access-z2lvb" (OuterVolumeSpecName: "kube-api-access-z2lvb") pod "0089347f-76b8-43a7-9dab-0def01e053e5" (UID: "0089347f-76b8-43a7-9dab-0def01e053e5"). InnerVolumeSpecName "kube-api-access-z2lvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:34:59.587866 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.587819    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0089347f-76b8-43a7-9dab-0def01e053e5-istio-token" (OuterVolumeSpecName: "istio-token") pod "0089347f-76b8-43a7-9dab-0def01e053e5" (UID: "0089347f-76b8-43a7-9dab-0def01e053e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:34:59.588010 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.587992    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0089347f-76b8-43a7-9dab-0def01e053e5" (UID: "0089347f-76b8-43a7-9dab-0def01e053e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:34:59.686589 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.686552    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-workload-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:34:59.686589 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.686583    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0089347f-76b8-43a7-9dab-0def01e053e5-istiod-ca-cert\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:34:59.686589 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.686594    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0089347f-76b8-43a7-9dab-0def01e053e5-istio-token\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:34:59.686839 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.686602    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-istio-data\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:34:59.686839 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.686610    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2lvb\" (UniqueName: \"kubernetes.io/projected/0089347f-76b8-43a7-9dab-0def01e053e5-kube-api-access-z2lvb\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:34:59.686839 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.686618    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0089347f-76b8-43a7-9dab-0def01e053e5-istio-envoy\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:34:59.686839 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.686627    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0089347f-76b8-43a7-9dab-0def01e053e5-istio-podinfo\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:34:59.885809 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.885722    2555 generic.go:356] "Generic (PLEG): container finished" podID="0089347f-76b8-43a7-9dab-0def01e053e5" containerID="c1b025892b4766072ed7cab6ad9454acd1cc782859dc1689c882915fc9247260" exitCode=0
Sep 09 16:34:59.885809 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.885773    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"
Sep 09 16:34:59.885809 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.885775    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2" event={"ID":"0089347f-76b8-43a7-9dab-0def01e053e5","Type":"ContainerDied","Data":"c1b025892b4766072ed7cab6ad9454acd1cc782859dc1689c882915fc9247260"}
Sep 09 16:34:59.886042 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.885832    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2" event={"ID":"0089347f-76b8-43a7-9dab-0def01e053e5","Type":"ContainerDied","Data":"bffd40aa0d8406ec1cb31cb8ac922d9028f5228f0e5410b37ded299742692da4"}
Sep 09 16:34:59.886042 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.885855    2555 scope.go:117] "RemoveContainer" containerID="c1b025892b4766072ed7cab6ad9454acd1cc782859dc1689c882915fc9247260"
Sep 09 16:34:59.896547 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.896527    2555 scope.go:117] "RemoveContainer" containerID="4b85145374fc462830b14a67eb3dea3c2fd3913ec35acdbcc17bfcdb72cce05c"
Sep 09 16:34:59.905102 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.905081    2555 scope.go:117] "RemoveContainer" containerID="c1b025892b4766072ed7cab6ad9454acd1cc782859dc1689c882915fc9247260"
Sep 09 16:34:59.905486 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:34:59.905463    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b025892b4766072ed7cab6ad9454acd1cc782859dc1689c882915fc9247260\": container with ID starting with c1b025892b4766072ed7cab6ad9454acd1cc782859dc1689c882915fc9247260 not found: ID does not exist" containerID="c1b025892b4766072ed7cab6ad9454acd1cc782859dc1689c882915fc9247260"
Sep 09 16:34:59.905611 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.905494    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b025892b4766072ed7cab6ad9454acd1cc782859dc1689c882915fc9247260"} err="failed to get container status \"c1b025892b4766072ed7cab6ad9454acd1cc782859dc1689c882915fc9247260\": rpc error: code = NotFound desc = could not find container \"c1b025892b4766072ed7cab6ad9454acd1cc782859dc1689c882915fc9247260\": container with ID starting with c1b025892b4766072ed7cab6ad9454acd1cc782859dc1689c882915fc9247260 not found: ID does not exist"
Sep 09 16:34:59.905611 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.905513    2555 scope.go:117] "RemoveContainer" containerID="4b85145374fc462830b14a67eb3dea3c2fd3913ec35acdbcc17bfcdb72cce05c"
Sep 09 16:34:59.905827 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:34:59.905803    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b85145374fc462830b14a67eb3dea3c2fd3913ec35acdbcc17bfcdb72cce05c\": container with ID starting with 4b85145374fc462830b14a67eb3dea3c2fd3913ec35acdbcc17bfcdb72cce05c not found: ID does not exist" containerID="4b85145374fc462830b14a67eb3dea3c2fd3913ec35acdbcc17bfcdb72cce05c"
Sep 09 16:34:59.905883 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.905836    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b85145374fc462830b14a67eb3dea3c2fd3913ec35acdbcc17bfcdb72cce05c"} err="failed to get container status \"4b85145374fc462830b14a67eb3dea3c2fd3913ec35acdbcc17bfcdb72cce05c\": rpc error: code = NotFound desc = could not find container \"4b85145374fc462830b14a67eb3dea3c2fd3913ec35acdbcc17bfcdb72cce05c\": container with ID starting with 4b85145374fc462830b14a67eb3dea3c2fd3913ec35acdbcc17bfcdb72cce05c not found: ID does not exist"
Sep 09 16:34:59.908330 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.908310    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"]
Sep 09 16:34:59.912397 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:34:59.912376    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-20ed533b-openshift-default-7b8c94755b-mwxn2"]
Sep 09 16:35:00.615877 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:00.615842    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0089347f-76b8-43a7-9dab-0def01e053e5" path="/var/lib/kubelet/pods/0089347f-76b8-43a7-9dab-0def01e053e5/volumes"
Sep 09 16:35:04.110564 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:04.110525    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:35:18.776010 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:18.775973    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"]
Sep 09 16:35:21.802670 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:21.802631    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"]
Sep 09 16:35:21.803523 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:21.803464    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g" podUID="1c3a7510-7aa7-422c-bd35-654bcdda4167" containerName="maas-api" containerID="cri-o://5ea4e8b8d5236b1432293d821c8411214c8977a0f87784fc13a477ec58f7af6c" gracePeriod=30
Sep 09 16:35:22.022943 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:22.022907    2555 generic.go:356] "Generic (PLEG): container finished" podID="1c3a7510-7aa7-422c-bd35-654bcdda4167" containerID="5ea4e8b8d5236b1432293d821c8411214c8977a0f87784fc13a477ec58f7af6c" exitCode=0
Sep 09 16:35:22.023143 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:22.022999    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g" event={"ID":"1c3a7510-7aa7-422c-bd35-654bcdda4167","Type":"ContainerDied","Data":"5ea4e8b8d5236b1432293d821c8411214c8977a0f87784fc13a477ec58f7af6c"}
Sep 09 16:35:22.062028 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:22.061958    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"
Sep 09 16:35:22.116808 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:22.116767    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c3a7510-7aa7-422c-bd35-654bcdda4167-maas-api-tls\") pod \"1c3a7510-7aa7-422c-bd35-654bcdda4167\" (UID: \"1c3a7510-7aa7-422c-bd35-654bcdda4167\") "
Sep 09 16:35:22.116808 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:22.116819    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2n67j\" (UniqueName: \"kubernetes.io/projected/1c3a7510-7aa7-422c-bd35-654bcdda4167-kube-api-access-2n67j\") pod \"1c3a7510-7aa7-422c-bd35-654bcdda4167\" (UID: \"1c3a7510-7aa7-422c-bd35-654bcdda4167\") "
Sep 09 16:35:22.117083 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:22.116911    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c3a7510-7aa7-422c-bd35-654bcdda4167-metrics-certs\") pod \"1c3a7510-7aa7-422c-bd35-654bcdda4167\" (UID: \"1c3a7510-7aa7-422c-bd35-654bcdda4167\") "
Sep 09 16:35:22.117083 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:22.117030    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c3a7510-7aa7-422c-bd35-654bcdda4167-openshift-service-ca\") pod \"1c3a7510-7aa7-422c-bd35-654bcdda4167\" (UID: \"1c3a7510-7aa7-422c-bd35-654bcdda4167\") "
Sep 09 16:35:22.117809 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:22.117683    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c3a7510-7aa7-422c-bd35-654bcdda4167-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1c3a7510-7aa7-422c-bd35-654bcdda4167" (UID: "1c3a7510-7aa7-422c-bd35-654bcdda4167"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:35:22.119372 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:22.119330    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3a7510-7aa7-422c-bd35-654bcdda4167-kube-api-access-2n67j" (OuterVolumeSpecName: "kube-api-access-2n67j") pod "1c3a7510-7aa7-422c-bd35-654bcdda4167" (UID: "1c3a7510-7aa7-422c-bd35-654bcdda4167"). InnerVolumeSpecName "kube-api-access-2n67j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:35:22.119710 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:22.119613    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c3a7510-7aa7-422c-bd35-654bcdda4167-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1c3a7510-7aa7-422c-bd35-654bcdda4167" (UID: "1c3a7510-7aa7-422c-bd35-654bcdda4167"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:35:22.119973 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:22.119946    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c3a7510-7aa7-422c-bd35-654bcdda4167-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1c3a7510-7aa7-422c-bd35-654bcdda4167" (UID: "1c3a7510-7aa7-422c-bd35-654bcdda4167"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:35:22.218568 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:22.218529    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c3a7510-7aa7-422c-bd35-654bcdda4167-openshift-service-ca\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:35:22.218568 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:22.218564    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c3a7510-7aa7-422c-bd35-654bcdda4167-maas-api-tls\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:35:22.218568 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:22.218574    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n67j\" (UniqueName: \"kubernetes.io/projected/1c3a7510-7aa7-422c-bd35-654bcdda4167-kube-api-access-2n67j\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:35:22.218828 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:22.218583    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c3a7510-7aa7-422c-bd35-654bcdda4167-metrics-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:35:23.030066 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:23.030031    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g" event={"ID":"1c3a7510-7aa7-422c-bd35-654bcdda4167","Type":"ContainerDied","Data":"03daad14e6f542ce8e56025c69973a8cf073b4850186b2cc2aa8f89ae9e61650"}
Sep 09 16:35:23.030514 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:23.030092    2555 scope.go:117] "RemoveContainer" containerID="5ea4e8b8d5236b1432293d821c8411214c8977a0f87784fc13a477ec58f7af6c"
Sep 09 16:35:23.030514 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:23.030039    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"
Sep 09 16:35:23.066311 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:23.066260    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"]
Sep 09 16:35:23.072578 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:23.072551    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-05e496-849d8c7ffc-m875g"]
Sep 09 16:35:23.310877 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:23.310781    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:35:24.609552 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:24.609510    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c3a7510-7aa7-422c-bd35-654bcdda4167" path="/var/lib/kubelet/pods/1c3a7510-7aa7-422c-bd35-654bcdda4167/volumes"
Sep 09 16:35:33.796262 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:33.796207    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6" podUID="991e3c32-3fc6-4288-b22d-100e6ac79fc0" containerName="main" containerID="cri-o://3b502bc80d57c849013722ca96828d8f7debe9bb7fa19b4670b6fd5c5b8d4c26" gracePeriod=45
Sep 09 16:35:34.058900 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.058817    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:35:34.107392 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.107357    2555 generic.go:356] "Generic (PLEG): container finished" podID="991e3c32-3fc6-4288-b22d-100e6ac79fc0" containerID="3b502bc80d57c849013722ca96828d8f7debe9bb7fa19b4670b6fd5c5b8d4c26" exitCode=0
Sep 09 16:35:34.107591 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.107421    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"
Sep 09 16:35:34.107591 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.107432    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6" event={"ID":"991e3c32-3fc6-4288-b22d-100e6ac79fc0","Type":"ContainerDied","Data":"3b502bc80d57c849013722ca96828d8f7debe9bb7fa19b4670b6fd5c5b8d4c26"}
Sep 09 16:35:34.107591 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.107483    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6" event={"ID":"991e3c32-3fc6-4288-b22d-100e6ac79fc0","Type":"ContainerDied","Data":"770e3158c2348e09116d435f8a5ae4d9fa09cc42d270038993279e60db4d5354"}
Sep 09 16:35:34.107591 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.107506    2555 scope.go:117] "RemoveContainer" containerID="3b502bc80d57c849013722ca96828d8f7debe9bb7fa19b4670b6fd5c5b8d4c26"
Sep 09 16:35:34.116650 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.116626    2555 scope.go:117] "RemoveContainer" containerID="3b502bc80d57c849013722ca96828d8f7debe9bb7fa19b4670b6fd5c5b8d4c26"
Sep 09 16:35:34.117083 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:35:34.117046    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b502bc80d57c849013722ca96828d8f7debe9bb7fa19b4670b6fd5c5b8d4c26\": container with ID starting with 3b502bc80d57c849013722ca96828d8f7debe9bb7fa19b4670b6fd5c5b8d4c26 not found: ID does not exist" containerID="3b502bc80d57c849013722ca96828d8f7debe9bb7fa19b4670b6fd5c5b8d4c26"
Sep 09 16:35:34.117204 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.117098    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b502bc80d57c849013722ca96828d8f7debe9bb7fa19b4670b6fd5c5b8d4c26"} err="failed to get container status \"3b502bc80d57c849013722ca96828d8f7debe9bb7fa19b4670b6fd5c5b8d4c26\": rpc error: code = NotFound desc = could not find container \"3b502bc80d57c849013722ca96828d8f7debe9bb7fa19b4670b6fd5c5b8d4c26\": container with ID starting with 3b502bc80d57c849013722ca96828d8f7debe9bb7fa19b4670b6fd5c5b8d4c26 not found: ID does not exist"
Sep 09 16:35:34.142432 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.142392    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/991e3c32-3fc6-4288-b22d-100e6ac79fc0-model-cache\") pod \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\" (UID: \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\") "
Sep 09 16:35:34.142645 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.142458    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991e3c32-3fc6-4288-b22d-100e6ac79fc0-tmp-dir\") pod \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\" (UID: \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\") "
Sep 09 16:35:34.142645 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.142511    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cp8tk\" (UniqueName: \"kubernetes.io/projected/991e3c32-3fc6-4288-b22d-100e6ac79fc0-kube-api-access-cp8tk\") pod \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\" (UID: \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\") "
Sep 09 16:35:34.142645 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.142529    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/991e3c32-3fc6-4288-b22d-100e6ac79fc0-tls-certs\") pod \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\" (UID: \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\") "
Sep 09 16:35:34.142645 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.142549    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/991e3c32-3fc6-4288-b22d-100e6ac79fc0-dshm\") pod \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\" (UID: \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\") "
Sep 09 16:35:34.142645 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.142574    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/991e3c32-3fc6-4288-b22d-100e6ac79fc0-home\") pod \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\" (UID: \"991e3c32-3fc6-4288-b22d-100e6ac79fc0\") "
Sep 09 16:35:34.142903 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.142742    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991e3c32-3fc6-4288-b22d-100e6ac79fc0-model-cache" (OuterVolumeSpecName: "model-cache") pod "991e3c32-3fc6-4288-b22d-100e6ac79fc0" (UID: "991e3c32-3fc6-4288-b22d-100e6ac79fc0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:35:34.142903 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.142787    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991e3c32-3fc6-4288-b22d-100e6ac79fc0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "991e3c32-3fc6-4288-b22d-100e6ac79fc0" (UID: "991e3c32-3fc6-4288-b22d-100e6ac79fc0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:35:34.143035 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.143010    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/991e3c32-3fc6-4288-b22d-100e6ac79fc0-model-cache\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:35:34.143090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.143056    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/991e3c32-3fc6-4288-b22d-100e6ac79fc0-tmp-dir\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:35:34.143280 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.143226    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991e3c32-3fc6-4288-b22d-100e6ac79fc0-home" (OuterVolumeSpecName: "home") pod "991e3c32-3fc6-4288-b22d-100e6ac79fc0" (UID: "991e3c32-3fc6-4288-b22d-100e6ac79fc0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:35:34.145076 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.145051    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991e3c32-3fc6-4288-b22d-100e6ac79fc0-dshm" (OuterVolumeSpecName: "dshm") pod "991e3c32-3fc6-4288-b22d-100e6ac79fc0" (UID: "991e3c32-3fc6-4288-b22d-100e6ac79fc0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:35:34.145248 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.145227    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/991e3c32-3fc6-4288-b22d-100e6ac79fc0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "991e3c32-3fc6-4288-b22d-100e6ac79fc0" (UID: "991e3c32-3fc6-4288-b22d-100e6ac79fc0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:35:34.145438 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.145407    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991e3c32-3fc6-4288-b22d-100e6ac79fc0-kube-api-access-cp8tk" (OuterVolumeSpecName: "kube-api-access-cp8tk") pod "991e3c32-3fc6-4288-b22d-100e6ac79fc0" (UID: "991e3c32-3fc6-4288-b22d-100e6ac79fc0"). InnerVolumeSpecName "kube-api-access-cp8tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:35:34.244626 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.244588    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp8tk\" (UniqueName: \"kubernetes.io/projected/991e3c32-3fc6-4288-b22d-100e6ac79fc0-kube-api-access-cp8tk\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:35:34.244626 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.244618    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/991e3c32-3fc6-4288-b22d-100e6ac79fc0-tls-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:35:34.244626 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.244629    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/991e3c32-3fc6-4288-b22d-100e6ac79fc0-dshm\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:35:34.244626 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.244638    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/991e3c32-3fc6-4288-b22d-100e6ac79fc0-home\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:35:34.430051 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.430015    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"]
Sep 09 16:35:34.432591 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.432565    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-05e496/autores-model-05e496-kserve-56db6979-tzbh6"]
Sep 09 16:35:34.609419 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:34.609383    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="991e3c32-3fc6-4288-b22d-100e6ac79fc0" path="/var/lib/kubelet/pods/991e3c32-3fc6-4288-b22d-100e6ac79fc0/volumes"
Sep 09 16:35:38.894033 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:38.893984    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:35:47.495579 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.495531    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"]
Sep 09 16:35:47.496129 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.496111    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c3a7510-7aa7-422c-bd35-654bcdda4167" containerName="maas-api"
Sep 09 16:35:47.496175 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.496133    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c3a7510-7aa7-422c-bd35-654bcdda4167" containerName="maas-api"
Sep 09 16:35:47.496175 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.496153    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0089347f-76b8-43a7-9dab-0def01e053e5" containerName="istio-proxy"
Sep 09 16:35:47.496175 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.496163    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0089347f-76b8-43a7-9dab-0def01e053e5" containerName="istio-proxy"
Sep 09 16:35:47.496267 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.496191    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22ca34f-0971-4d10-ae1a-bee3cfbf3452" containerName="payload-pre-processing"
Sep 09 16:35:47.496267 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.496200    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22ca34f-0971-4d10-ae1a-bee3cfbf3452" containerName="payload-pre-processing"
Sep 09 16:35:47.496267 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.496211    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="991e3c32-3fc6-4288-b22d-100e6ac79fc0" containerName="main"
Sep 09 16:35:47.496267 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.496219    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="991e3c32-3fc6-4288-b22d-100e6ac79fc0" containerName="main"
Sep 09 16:35:47.496267 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.496236    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0089347f-76b8-43a7-9dab-0def01e053e5" containerName="istio-proxy"
Sep 09 16:35:47.496267 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.496244    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0089347f-76b8-43a7-9dab-0def01e053e5" containerName="istio-proxy"
Sep 09 16:35:47.496493 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.496358    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c3a7510-7aa7-422c-bd35-654bcdda4167" containerName="maas-api"
Sep 09 16:35:47.496493 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.496373    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c22ca34f-0971-4d10-ae1a-bee3cfbf3452" containerName="payload-pre-processing"
Sep 09 16:35:47.496493 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.496409    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0089347f-76b8-43a7-9dab-0def01e053e5" containerName="istio-proxy"
Sep 09 16:35:47.496493 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.496417    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="991e3c32-3fc6-4288-b22d-100e6ac79fc0" containerName="main"
Sep 09 16:35:47.496493 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.496422    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0089347f-76b8-43a7-9dab-0def01e053e5" containerName="istio-proxy"
Sep 09 16:35:47.501569 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.501544    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:35:47.504451 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.504416    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-34141f\"/\"openshift-service-ca.crt\""
Sep 09 16:35:47.505724 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.505697    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-34141f\"/\"kube-root-ca.crt\""
Sep 09 16:35:47.505878 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.505703    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-34141f\"/\"default-dockercfg-54wqk\""
Sep 09 16:35:47.505878 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.505775    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-34141f\"/\"test-model-34141f-kserve-self-signed-certs\""
Sep 09 16:35:47.506511 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.506485    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"]
Sep 09 16:35:47.560717 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.560677    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51b453a1-fbde-49db-9760-c2232f773805-model-cache\") pod \"test-model-34141f-kserve-64b78d787f-fwz29\" (UID: \"51b453a1-fbde-49db-9760-c2232f773805\") " pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:35:47.560922 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.560776    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtw5d\" (UniqueName: \"kubernetes.io/projected/51b453a1-fbde-49db-9760-c2232f773805-kube-api-access-dtw5d\") pod \"test-model-34141f-kserve-64b78d787f-fwz29\" (UID: \"51b453a1-fbde-49db-9760-c2232f773805\") " pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:35:47.560922 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.560902    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51b453a1-fbde-49db-9760-c2232f773805-home\") pod \"test-model-34141f-kserve-64b78d787f-fwz29\" (UID: \"51b453a1-fbde-49db-9760-c2232f773805\") " pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:35:47.561046 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.560940    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51b453a1-fbde-49db-9760-c2232f773805-dshm\") pod \"test-model-34141f-kserve-64b78d787f-fwz29\" (UID: \"51b453a1-fbde-49db-9760-c2232f773805\") " pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:35:47.561046 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.560964    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51b453a1-fbde-49db-9760-c2232f773805-tls-certs\") pod \"test-model-34141f-kserve-64b78d787f-fwz29\" (UID: \"51b453a1-fbde-49db-9760-c2232f773805\") " pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:35:47.561046 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.561027    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51b453a1-fbde-49db-9760-c2232f773805-tmp-dir\") pod \"test-model-34141f-kserve-64b78d787f-fwz29\" (UID: \"51b453a1-fbde-49db-9760-c2232f773805\") " pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:35:47.661809 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.661771    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51b453a1-fbde-49db-9760-c2232f773805-home\") pod \"test-model-34141f-kserve-64b78d787f-fwz29\" (UID: \"51b453a1-fbde-49db-9760-c2232f773805\") " pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:35:47.661809 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.661813    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51b453a1-fbde-49db-9760-c2232f773805-dshm\") pod \"test-model-34141f-kserve-64b78d787f-fwz29\" (UID: \"51b453a1-fbde-49db-9760-c2232f773805\") " pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:35:47.662196 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.661837    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51b453a1-fbde-49db-9760-c2232f773805-tls-certs\") pod \"test-model-34141f-kserve-64b78d787f-fwz29\" (UID: \"51b453a1-fbde-49db-9760-c2232f773805\") " pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:35:47.662196 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.661877    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51b453a1-fbde-49db-9760-c2232f773805-tmp-dir\") pod \"test-model-34141f-kserve-64b78d787f-fwz29\" (UID: \"51b453a1-fbde-49db-9760-c2232f773805\") " pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:35:47.662196 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.661930    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51b453a1-fbde-49db-9760-c2232f773805-model-cache\") pod \"test-model-34141f-kserve-64b78d787f-fwz29\" (UID: \"51b453a1-fbde-49db-9760-c2232f773805\") " pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:35:47.662196 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.662003    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtw5d\" (UniqueName: \"kubernetes.io/projected/51b453a1-fbde-49db-9760-c2232f773805-kube-api-access-dtw5d\") pod \"test-model-34141f-kserve-64b78d787f-fwz29\" (UID: \"51b453a1-fbde-49db-9760-c2232f773805\") " pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:35:47.662655 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.662627    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51b453a1-fbde-49db-9760-c2232f773805-tmp-dir\") pod \"test-model-34141f-kserve-64b78d787f-fwz29\" (UID: \"51b453a1-fbde-49db-9760-c2232f773805\") " pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:35:47.662814 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.662715    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51b453a1-fbde-49db-9760-c2232f773805-model-cache\") pod \"test-model-34141f-kserve-64b78d787f-fwz29\" (UID: \"51b453a1-fbde-49db-9760-c2232f773805\") " pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:35:47.662934 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.662752    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51b453a1-fbde-49db-9760-c2232f773805-home\") pod \"test-model-34141f-kserve-64b78d787f-fwz29\" (UID: \"51b453a1-fbde-49db-9760-c2232f773805\") " pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:35:47.664943 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.664904    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51b453a1-fbde-49db-9760-c2232f773805-dshm\") pod \"test-model-34141f-kserve-64b78d787f-fwz29\" (UID: \"51b453a1-fbde-49db-9760-c2232f773805\") " pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:35:47.665237 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.665214    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51b453a1-fbde-49db-9760-c2232f773805-tls-certs\") pod \"test-model-34141f-kserve-64b78d787f-fwz29\" (UID: \"51b453a1-fbde-49db-9760-c2232f773805\") " pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:35:47.671251 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.671219    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtw5d\" (UniqueName: \"kubernetes.io/projected/51b453a1-fbde-49db-9760-c2232f773805-kube-api-access-dtw5d\") pod \"test-model-34141f-kserve-64b78d787f-fwz29\" (UID: \"51b453a1-fbde-49db-9760-c2232f773805\") " pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:35:47.814123 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:47.814026    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:35:48.090610 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:48.090577    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"]
Sep 09 16:35:48.092413 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:35:48.092356    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51b453a1_fbde_49db_9760_c2232f773805.slice/crio-f8988496cc59ebbc61418aec840a25930103893f953519a515b8570b4a5fd1b9 WatchSource:0}: Error finding container f8988496cc59ebbc61418aec840a25930103893f953519a515b8570b4a5fd1b9: Status 404 returned error can't find the container with id f8988496cc59ebbc61418aec840a25930103893f953519a515b8570b4a5fd1b9
Sep 09 16:35:48.194867 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:48.194827    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29" event={"ID":"51b453a1-fbde-49db-9760-c2232f773805","Type":"ContainerStarted","Data":"632cbd9090f81729237af43d8545e88e086881f4211dc4a1f0107006571b6eb7"}
Sep 09 16:35:48.195174 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:48.194879    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29" event={"ID":"51b453a1-fbde-49db-9760-c2232f773805","Type":"ContainerStarted","Data":"f8988496cc59ebbc61418aec840a25930103893f953519a515b8570b4a5fd1b9"}
Sep 09 16:35:48.211763 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:48.211690    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29" podStartSLOduration=1.211666679 podStartE2EDuration="1.211666679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:35:48.209130323 +0000 UTC m=+1314.039338380" watchObservedRunningTime="2026-09-09 16:35:48.211666679 +0000 UTC m=+1314.041874781"
Sep 09 16:35:52.801554 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:52.801516    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:35:57.814991 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:57.814954    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:35:57.815493 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:57.815005    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:35:57.827331 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:57.827284    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:35:58.266665 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:35:58.266637    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:36:04.450311 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:04.450258    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"]
Sep 09 16:36:04.450785 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:04.450542    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t" podUID="c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" containerName="istio-proxy" containerID="cri-o://201fd0a4106a35c4c5c0d7b293d78922263af86564ba906e7019f6bbcccfb3aa" gracePeriod=30
Sep 09 16:36:08.213987 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:08.213955    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:36:09.707730 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.707706    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:36:09.750009 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.749973    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"]
Sep 09 16:36:09.750625 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.750598    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" containerName="istio-proxy"
Sep 09 16:36:09.750807 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.750785    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" containerName="istio-proxy"
Sep 09 16:36:09.750909 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.750824    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" containerName="istio-proxy"
Sep 09 16:36:09.750909 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.750833    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" containerName="istio-proxy"
Sep 09 16:36:09.751014 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.750965    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" containerName="istio-proxy"
Sep 09 16:36:09.751014 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.750982    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" containerName="istio-proxy"
Sep 09 16:36:09.755134 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.755110    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"
Sep 09 16:36:09.757899 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.757876    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-26563f-serving-cert\""
Sep 09 16:36:09.762611 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.762592    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"]
Sep 09 16:36:09.765603 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.765570    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istio-data\") pod \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") "
Sep 09 16:36:09.765687 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.765611    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istio-envoy\") pod \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") "
Sep 09 16:36:09.765687 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.765632    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-workload-socket\") pod \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") "
Sep 09 16:36:09.765687 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.765660    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwrvz\" (UniqueName: \"kubernetes.io/projected/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-kube-api-access-nwrvz\") pod \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") "
Sep 09 16:36:09.766157 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.765689    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-credential-socket\") pod \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") "
Sep 09 16:36:09.766157 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.765715    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-workload-certs\") pod \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") "
Sep 09 16:36:09.766157 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.765758    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istio-token\") pod \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") "
Sep 09 16:36:09.766157 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.765790    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istio-podinfo\") pod \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") "
Sep 09 16:36:09.766157 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.765820    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istiod-ca-cert\") pod \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\" (UID: \"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe\") "
Sep 09 16:36:09.766157 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.765888    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istio-data" (OuterVolumeSpecName: "istio-data") pod "c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" (UID: "c89c30e4-ab2a-4026-ace4-d7973e8ef3fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:36:09.766157 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.766093    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" (UID: "c89c30e4-ab2a-4026-ace4-d7973e8ef3fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:36:09.767209 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.766191    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istio-data\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:36:09.767209 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.766210    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-workload-socket\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:36:09.767209 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.766408    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" (UID: "c89c30e4-ab2a-4026-ace4-d7973e8ef3fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:36:09.767209 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.766494    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" (UID: "c89c30e4-ab2a-4026-ace4-d7973e8ef3fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:36:09.767209 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.766719    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" (UID: "c89c30e4-ab2a-4026-ace4-d7973e8ef3fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:36:09.768569 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.768548    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" (UID: "c89c30e4-ab2a-4026-ace4-d7973e8ef3fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 16:36:09.768798 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.768776    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istio-token" (OuterVolumeSpecName: "istio-token") pod "c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" (UID: "c89c30e4-ab2a-4026-ace4-d7973e8ef3fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:36:09.768874 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.768773    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-kube-api-access-nwrvz" (OuterVolumeSpecName: "kube-api-access-nwrvz") pod "c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" (UID: "c89c30e4-ab2a-4026-ace4-d7973e8ef3fe"). InnerVolumeSpecName "kube-api-access-nwrvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:36:09.768935 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.768913    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" (UID: "c89c30e4-ab2a-4026-ace4-d7973e8ef3fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:36:09.867569 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.867537    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv2rn\" (UniqueName: \"kubernetes.io/projected/dfc91adb-3975-4bea-adbe-123dc342d630-kube-api-access-kv2rn\") pod \"maas-api-e2e-explicit-26563f-677f4dbfcf-788j9\" (UID: \"dfc91adb-3975-4bea-adbe-123dc342d630\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"
Sep 09 16:36:09.867758 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.867588    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc91adb-3975-4bea-adbe-123dc342d630-maas-api-tls\") pod \"maas-api-e2e-explicit-26563f-677f4dbfcf-788j9\" (UID: \"dfc91adb-3975-4bea-adbe-123dc342d630\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"
Sep 09 16:36:09.867758 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.867657    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc91adb-3975-4bea-adbe-123dc342d630-openshift-service-ca\") pod \"maas-api-e2e-explicit-26563f-677f4dbfcf-788j9\" (UID: \"dfc91adb-3975-4bea-adbe-123dc342d630\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"
Sep 09 16:36:09.867758 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.867700    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc91adb-3975-4bea-adbe-123dc342d630-metrics-certs\") pod \"maas-api-e2e-explicit-26563f-677f4dbfcf-788j9\" (UID: \"dfc91adb-3975-4bea-adbe-123dc342d630\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"
Sep 09 16:36:09.867948 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.867776    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwrvz\" (UniqueName: \"kubernetes.io/projected/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-kube-api-access-nwrvz\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:36:09.867948 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.867822    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-credential-socket\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:36:09.867948 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.867832    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-workload-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:36:09.867948 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.867844    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istio-token\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:36:09.867948 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.867855    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istio-podinfo\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:36:09.867948 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.867867    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istiod-ca-cert\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:36:09.867948 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.867878    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe-istio-envoy\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:36:09.968838 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.968806    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv2rn\" (UniqueName: \"kubernetes.io/projected/dfc91adb-3975-4bea-adbe-123dc342d630-kube-api-access-kv2rn\") pod \"maas-api-e2e-explicit-26563f-677f4dbfcf-788j9\" (UID: \"dfc91adb-3975-4bea-adbe-123dc342d630\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"
Sep 09 16:36:09.969014 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.968857    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc91adb-3975-4bea-adbe-123dc342d630-maas-api-tls\") pod \"maas-api-e2e-explicit-26563f-677f4dbfcf-788j9\" (UID: \"dfc91adb-3975-4bea-adbe-123dc342d630\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"
Sep 09 16:36:09.969014 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.968914    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc91adb-3975-4bea-adbe-123dc342d630-openshift-service-ca\") pod \"maas-api-e2e-explicit-26563f-677f4dbfcf-788j9\" (UID: \"dfc91adb-3975-4bea-adbe-123dc342d630\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"
Sep 09 16:36:09.969138 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:36:09.969040    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-serving-cert: secret "maas-api-e2e-explicit-26563f-serving-cert" not found
Sep 09 16:36:09.969138 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:36:09.969124    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc91adb-3975-4bea-adbe-123dc342d630-maas-api-tls podName:dfc91adb-3975-4bea-adbe-123dc342d630 nodeName:}" failed. No retries permitted until 2026-09-09 16:36:10.469102639 +0000 UTC m=+1336.299310681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dfc91adb-3975-4bea-adbe-123dc342d630-maas-api-tls") pod "maas-api-e2e-explicit-26563f-677f4dbfcf-788j9" (UID: "dfc91adb-3975-4bea-adbe-123dc342d630") : secret "maas-api-e2e-explicit-26563f-serving-cert" not found
Sep 09 16:36:09.969258 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.969146    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc91adb-3975-4bea-adbe-123dc342d630-metrics-certs\") pod \"maas-api-e2e-explicit-26563f-677f4dbfcf-788j9\" (UID: \"dfc91adb-3975-4bea-adbe-123dc342d630\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"
Sep 09 16:36:09.969669 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.969649    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc91adb-3975-4bea-adbe-123dc342d630-openshift-service-ca\") pod \"maas-api-e2e-explicit-26563f-677f4dbfcf-788j9\" (UID: \"dfc91adb-3975-4bea-adbe-123dc342d630\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"
Sep 09 16:36:09.971813 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.971792    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc91adb-3975-4bea-adbe-123dc342d630-metrics-certs\") pod \"maas-api-e2e-explicit-26563f-677f4dbfcf-788j9\" (UID: \"dfc91adb-3975-4bea-adbe-123dc342d630\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"
Sep 09 16:36:09.977418 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:09.977391    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv2rn\" (UniqueName: \"kubernetes.io/projected/dfc91adb-3975-4bea-adbe-123dc342d630-kube-api-access-kv2rn\") pod \"maas-api-e2e-explicit-26563f-677f4dbfcf-788j9\" (UID: \"dfc91adb-3975-4bea-adbe-123dc342d630\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"
Sep 09 16:36:10.326470 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:10.326437    2555 generic.go:356] "Generic (PLEG): container finished" podID="c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" containerID="201fd0a4106a35c4c5c0d7b293d78922263af86564ba906e7019f6bbcccfb3aa" exitCode=0
Sep 09 16:36:10.326655 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:10.326492    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"
Sep 09 16:36:10.326655 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:10.326529    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t" event={"ID":"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe","Type":"ContainerDied","Data":"201fd0a4106a35c4c5c0d7b293d78922263af86564ba906e7019f6bbcccfb3aa"}
Sep 09 16:36:10.326655 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:10.326599    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t" event={"ID":"c89c30e4-ab2a-4026-ace4-d7973e8ef3fe","Type":"ContainerDied","Data":"d2db09ea1f9264f26e622fd61dff93d1facc05a52a5145ed4031f90332105a40"}
Sep 09 16:36:10.326655 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:10.326628    2555 scope.go:117] "RemoveContainer" containerID="201fd0a4106a35c4c5c0d7b293d78922263af86564ba906e7019f6bbcccfb3aa"
Sep 09 16:36:10.336627 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:10.336608    2555 scope.go:117] "RemoveContainer" containerID="e6e71e07fed1abf07332aaf034b7f1e9aa0c785e3b34ca3c13a1f87655beb8ec"
Sep 09 16:36:10.346388 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:10.346359    2555 scope.go:117] "RemoveContainer" containerID="201fd0a4106a35c4c5c0d7b293d78922263af86564ba906e7019f6bbcccfb3aa"
Sep 09 16:36:10.346984 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:36:10.346752    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"201fd0a4106a35c4c5c0d7b293d78922263af86564ba906e7019f6bbcccfb3aa\": container with ID starting with 201fd0a4106a35c4c5c0d7b293d78922263af86564ba906e7019f6bbcccfb3aa not found: ID does not exist" containerID="201fd0a4106a35c4c5c0d7b293d78922263af86564ba906e7019f6bbcccfb3aa"
Sep 09 16:36:10.346984 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:10.346788    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"201fd0a4106a35c4c5c0d7b293d78922263af86564ba906e7019f6bbcccfb3aa"} err="failed to get container status \"201fd0a4106a35c4c5c0d7b293d78922263af86564ba906e7019f6bbcccfb3aa\": rpc error: code = NotFound desc = could not find container \"201fd0a4106a35c4c5c0d7b293d78922263af86564ba906e7019f6bbcccfb3aa\": container with ID starting with 201fd0a4106a35c4c5c0d7b293d78922263af86564ba906e7019f6bbcccfb3aa not found: ID does not exist"
Sep 09 16:36:10.346984 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:10.346814    2555 scope.go:117] "RemoveContainer" containerID="e6e71e07fed1abf07332aaf034b7f1e9aa0c785e3b34ca3c13a1f87655beb8ec"
Sep 09 16:36:10.347217 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:36:10.347190    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6e71e07fed1abf07332aaf034b7f1e9aa0c785e3b34ca3c13a1f87655beb8ec\": container with ID starting with e6e71e07fed1abf07332aaf034b7f1e9aa0c785e3b34ca3c13a1f87655beb8ec not found: ID does not exist" containerID="e6e71e07fed1abf07332aaf034b7f1e9aa0c785e3b34ca3c13a1f87655beb8ec"
Sep 09 16:36:10.347352 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:10.347223    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e71e07fed1abf07332aaf034b7f1e9aa0c785e3b34ca3c13a1f87655beb8ec"} err="failed to get container status \"e6e71e07fed1abf07332aaf034b7f1e9aa0c785e3b34ca3c13a1f87655beb8ec\": rpc error: code = NotFound desc = could not find container \"e6e71e07fed1abf07332aaf034b7f1e9aa0c785e3b34ca3c13a1f87655beb8ec\": container with ID starting with e6e71e07fed1abf07332aaf034b7f1e9aa0c785e3b34ca3c13a1f87655beb8ec not found: ID does not exist"
Sep 09 16:36:10.357159 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:10.357135    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"]
Sep 09 16:36:10.363484 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:10.363461    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-05e496-openshift-default-754fbb964d-2wb5t"]
Sep 09 16:36:10.474744 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:10.474710    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc91adb-3975-4bea-adbe-123dc342d630-maas-api-tls\") pod \"maas-api-e2e-explicit-26563f-677f4dbfcf-788j9\" (UID: \"dfc91adb-3975-4bea-adbe-123dc342d630\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"
Sep 09 16:36:10.477286 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:10.477247    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc91adb-3975-4bea-adbe-123dc342d630-maas-api-tls\") pod \"maas-api-e2e-explicit-26563f-677f4dbfcf-788j9\" (UID: \"dfc91adb-3975-4bea-adbe-123dc342d630\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"
Sep 09 16:36:10.608622 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:10.608540    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c89c30e4-ab2a-4026-ace4-d7973e8ef3fe" path="/var/lib/kubelet/pods/c89c30e4-ab2a-4026-ace4-d7973e8ef3fe/volumes"
Sep 09 16:36:10.669446 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:10.669418    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"
Sep 09 16:36:10.943199 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:10.943169    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"]
Sep 09 16:36:10.944996 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:36:10.944965    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc91adb_3975_4bea_adbe_123dc342d630.slice/crio-d597293ce46eec052acde8e8dac801d16c315e68497a561899eb940cf5410585 WatchSource:0}: Error finding container d597293ce46eec052acde8e8dac801d16c315e68497a561899eb940cf5410585: Status 404 returned error can't find the container with id d597293ce46eec052acde8e8dac801d16c315e68497a561899eb940cf5410585
Sep 09 16:36:10.946592 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:10.946566    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 16:36:11.335321 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:11.335278    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9" event={"ID":"dfc91adb-3975-4bea-adbe-123dc342d630","Type":"ContainerStarted","Data":"d597293ce46eec052acde8e8dac801d16c315e68497a561899eb940cf5410585"}
Sep 09 16:36:12.342796 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:12.342763    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9" event={"ID":"dfc91adb-3975-4bea-adbe-123dc342d630","Type":"ContainerStarted","Data":"d190c2d006c9ea14363ae1af01f4a0fa223e72a6b7ec5e0939f052689ae8ecb3"}
Sep 09 16:36:12.343177 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:12.342953    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"
Sep 09 16:36:12.360213 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:12.360163    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9" podStartSLOduration=2.965416174 podStartE2EDuration="3.360146054s" totalImagesPullingTime="394.72988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:36:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:36:12.357472245 +0000 UTC m=+1338.187680305" watchObservedRunningTime="2026-09-09 16:36:12.360146054 +0000 UTC m=+1338.190354154"
Sep 09 16:36:18.353261 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:18.353227    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"
Sep 09 16:36:26.587731 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:26.587698    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:36:34.951736 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:34.951701    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"]
Sep 09 16:36:34.956708 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:34.956687    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"
Sep 09 16:36:34.959282 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:34.959257    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-77176368-serving-cert\""
Sep 09 16:36:34.971261 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:34.971235    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"]
Sep 09 16:36:35.012630 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.012601    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-maas-api-tls\") pod \"maas-api-e2e-mt-77176368-66f85c676f-nght2\" (UID: \"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"
Sep 09 16:36:35.012758 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.012638    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-metrics-certs\") pod \"maas-api-e2e-mt-77176368-66f85c676f-nght2\" (UID: \"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"
Sep 09 16:36:35.012758 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.012679    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-openshift-service-ca\") pod \"maas-api-e2e-mt-77176368-66f85c676f-nght2\" (UID: \"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"
Sep 09 16:36:35.012758 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.012711    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7gjs\" (UniqueName: \"kubernetes.io/projected/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-kube-api-access-n7gjs\") pod \"maas-api-e2e-mt-77176368-66f85c676f-nght2\" (UID: \"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"
Sep 09 16:36:35.113673 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.113645    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7gjs\" (UniqueName: \"kubernetes.io/projected/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-kube-api-access-n7gjs\") pod \"maas-api-e2e-mt-77176368-66f85c676f-nght2\" (UID: \"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"
Sep 09 16:36:35.113846 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.113709    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-maas-api-tls\") pod \"maas-api-e2e-mt-77176368-66f85c676f-nght2\" (UID: \"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"
Sep 09 16:36:35.113846 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.113731    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-metrics-certs\") pod \"maas-api-e2e-mt-77176368-66f85c676f-nght2\" (UID: \"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"
Sep 09 16:36:35.113846 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.113762    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-openshift-service-ca\") pod \"maas-api-e2e-mt-77176368-66f85c676f-nght2\" (UID: \"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"
Sep 09 16:36:35.114034 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:36:35.113886    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-77176368-serving-cert: secret "maas-api-e2e-mt-77176368-serving-cert" not found
Sep 09 16:36:35.114034 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:36:35.113976    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-maas-api-tls podName:3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4 nodeName:}" failed. No retries permitted until 2026-09-09 16:36:35.613953187 +0000 UTC m=+1361.444161229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-maas-api-tls") pod "maas-api-e2e-mt-77176368-66f85c676f-nght2" (UID: "3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4") : secret "maas-api-e2e-mt-77176368-serving-cert" not found
Sep 09 16:36:35.114485 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.114467    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-openshift-service-ca\") pod \"maas-api-e2e-mt-77176368-66f85c676f-nght2\" (UID: \"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"
Sep 09 16:36:35.116285 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.116264    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-metrics-certs\") pod \"maas-api-e2e-mt-77176368-66f85c676f-nght2\" (UID: \"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"
Sep 09 16:36:35.121617 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.121595    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7gjs\" (UniqueName: \"kubernetes.io/projected/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-kube-api-access-n7gjs\") pod \"maas-api-e2e-mt-77176368-66f85c676f-nght2\" (UID: \"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"
Sep 09 16:36:35.551858 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.551826    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-77176368-f6ff4df49-4krb5"]
Sep 09 16:36:35.555344 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.555328    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-77176368-f6ff4df49-4krb5"
Sep 09 16:36:35.558177 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.558053    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-77176368-dockercfg-zbfdc\""
Sep 09 16:36:35.558177 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.558052    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-77176368\""
Sep 09 16:36:35.564530 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.564505    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-77176368-f6ff4df49-4krb5"]
Sep 09 16:36:35.618748 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.618712    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzfk5\" (UniqueName: \"kubernetes.io/projected/c55859b4-dce2-4c0a-ad42-dfde959d2773-kube-api-access-wzfk5\") pod \"payload-processing-e2e-mt-77176368-f6ff4df49-4krb5\" (UID: \"c55859b4-dce2-4c0a-ad42-dfde959d2773\") " pod="openshift-ingress/payload-processing-e2e-mt-77176368-f6ff4df49-4krb5"
Sep 09 16:36:35.618949 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.618861    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c55859b4-dce2-4c0a-ad42-dfde959d2773-plugins-config-volume\") pod \"payload-processing-e2e-mt-77176368-f6ff4df49-4krb5\" (UID: \"c55859b4-dce2-4c0a-ad42-dfde959d2773\") " pod="openshift-ingress/payload-processing-e2e-mt-77176368-f6ff4df49-4krb5"
Sep 09 16:36:35.618949 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.618899    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-maas-api-tls\") pod \"maas-api-e2e-mt-77176368-66f85c676f-nght2\" (UID: \"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"
Sep 09 16:36:35.622283 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.622255    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-maas-api-tls\") pod \"maas-api-e2e-mt-77176368-66f85c676f-nght2\" (UID: \"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"
Sep 09 16:36:35.720444 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.720415    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c55859b4-dce2-4c0a-ad42-dfde959d2773-plugins-config-volume\") pod \"payload-processing-e2e-mt-77176368-f6ff4df49-4krb5\" (UID: \"c55859b4-dce2-4c0a-ad42-dfde959d2773\") " pod="openshift-ingress/payload-processing-e2e-mt-77176368-f6ff4df49-4krb5"
Sep 09 16:36:35.720613 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.720501    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzfk5\" (UniqueName: \"kubernetes.io/projected/c55859b4-dce2-4c0a-ad42-dfde959d2773-kube-api-access-wzfk5\") pod \"payload-processing-e2e-mt-77176368-f6ff4df49-4krb5\" (UID: \"c55859b4-dce2-4c0a-ad42-dfde959d2773\") " pod="openshift-ingress/payload-processing-e2e-mt-77176368-f6ff4df49-4krb5"
Sep 09 16:36:35.721148 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.721126    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c55859b4-dce2-4c0a-ad42-dfde959d2773-plugins-config-volume\") pod \"payload-processing-e2e-mt-77176368-f6ff4df49-4krb5\" (UID: \"c55859b4-dce2-4c0a-ad42-dfde959d2773\") " pod="openshift-ingress/payload-processing-e2e-mt-77176368-f6ff4df49-4krb5"
Sep 09 16:36:35.729913 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.729888    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzfk5\" (UniqueName: \"kubernetes.io/projected/c55859b4-dce2-4c0a-ad42-dfde959d2773-kube-api-access-wzfk5\") pod \"payload-processing-e2e-mt-77176368-f6ff4df49-4krb5\" (UID: \"c55859b4-dce2-4c0a-ad42-dfde959d2773\") " pod="openshift-ingress/payload-processing-e2e-mt-77176368-f6ff4df49-4krb5"
Sep 09 16:36:35.868439 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.868337    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-77176368-f6ff4df49-4krb5"
Sep 09 16:36:35.870444 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:35.870410    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"
Sep 09 16:36:36.141818 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.141788    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-77176368-f6ff4df49-4krb5"]
Sep 09 16:36:36.143089 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:36:36.143059    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc55859b4_dce2_4c0a_ad42_dfde959d2773.slice/crio-a10b8e99992e42cdeae0cc0755fd5578d193da56606f5bdc6ada29387abbe105 WatchSource:0}: Error finding container a10b8e99992e42cdeae0cc0755fd5578d193da56606f5bdc6ada29387abbe105: Status 404 returned error can't find the container with id a10b8e99992e42cdeae0cc0755fd5578d193da56606f5bdc6ada29387abbe105
Sep 09 16:36:36.148955 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.148926    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"]
Sep 09 16:36:36.149342 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:36:36.149319    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e2e9e9e_e6c8_4466_ba72_0f59d6ab85e4.slice/crio-bd5c6ec6941b86e5c998d2dd056538f13413836edd5cff09f76af93df60bafb4 WatchSource:0}: Error finding container bd5c6ec6941b86e5c998d2dd056538f13413836edd5cff09f76af93df60bafb4: Status 404 returned error can't find the container with id bd5c6ec6941b86e5c998d2dd056538f13413836edd5cff09f76af93df60bafb4
Sep 09 16:36:36.186611 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.186587    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"]
Sep 09 16:36:36.191145 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.191128    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:36:36.193964 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.193948    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-26563f\"/\"default-dockercfg-mht2t\""
Sep 09 16:36:36.194046 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.193947    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-26563f\"/\"openshift-service-ca.crt\""
Sep 09 16:36:36.195045 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.195024    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-26563f\"/\"kube-root-ca.crt\""
Sep 09 16:36:36.195156 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.195143    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-26563f\"/\"explicit-model-26563f-kserve-self-signed-certs\""
Sep 09 16:36:36.198110 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.198088    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"]
Sep 09 16:36:36.223777 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.223744    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-model-cache\") pod \"explicit-model-26563f-kserve-86cfb88689-9cqgj\" (UID: \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\") " pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:36:36.223864 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.223795    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-tls-certs\") pod \"explicit-model-26563f-kserve-86cfb88689-9cqgj\" (UID: \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\") " pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:36:36.223864 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.223830    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-tmp-dir\") pod \"explicit-model-26563f-kserve-86cfb88689-9cqgj\" (UID: \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\") " pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:36:36.223948 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.223899    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhlnl\" (UniqueName: \"kubernetes.io/projected/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-kube-api-access-fhlnl\") pod \"explicit-model-26563f-kserve-86cfb88689-9cqgj\" (UID: \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\") " pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:36:36.223990 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.223955    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-dshm\") pod \"explicit-model-26563f-kserve-86cfb88689-9cqgj\" (UID: \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\") " pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:36:36.224060 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.223998    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-home\") pod \"explicit-model-26563f-kserve-86cfb88689-9cqgj\" (UID: \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\") " pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:36:36.324998 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.324963    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-tmp-dir\") pod \"explicit-model-26563f-kserve-86cfb88689-9cqgj\" (UID: \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\") " pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:36:36.325164 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.325023    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhlnl\" (UniqueName: \"kubernetes.io/projected/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-kube-api-access-fhlnl\") pod \"explicit-model-26563f-kserve-86cfb88689-9cqgj\" (UID: \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\") " pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:36:36.325164 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.325055    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-dshm\") pod \"explicit-model-26563f-kserve-86cfb88689-9cqgj\" (UID: \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\") " pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:36:36.325164 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.325079    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-home\") pod \"explicit-model-26563f-kserve-86cfb88689-9cqgj\" (UID: \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\") " pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:36:36.325164 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.325160    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-model-cache\") pod \"explicit-model-26563f-kserve-86cfb88689-9cqgj\" (UID: \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\") " pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:36:36.325415 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.325190    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-tls-certs\") pod \"explicit-model-26563f-kserve-86cfb88689-9cqgj\" (UID: \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\") " pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:36:36.325475 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.325431    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-tmp-dir\") pod \"explicit-model-26563f-kserve-86cfb88689-9cqgj\" (UID: \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\") " pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:36:36.325576 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.325557    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-home\") pod \"explicit-model-26563f-kserve-86cfb88689-9cqgj\" (UID: \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\") " pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:36:36.325643 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.325580    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-model-cache\") pod \"explicit-model-26563f-kserve-86cfb88689-9cqgj\" (UID: \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\") " pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:36:36.327479 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.327455    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-dshm\") pod \"explicit-model-26563f-kserve-86cfb88689-9cqgj\" (UID: \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\") " pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:36:36.327910 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.327893    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-tls-certs\") pod \"explicit-model-26563f-kserve-86cfb88689-9cqgj\" (UID: \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\") " pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:36:36.332379 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.332353    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhlnl\" (UniqueName: \"kubernetes.io/projected/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-kube-api-access-fhlnl\") pod \"explicit-model-26563f-kserve-86cfb88689-9cqgj\" (UID: \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\") " pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:36:36.495365 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.495333    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-77176368-f6ff4df49-4krb5" event={"ID":"c55859b4-dce2-4c0a-ad42-dfde959d2773","Type":"ContainerStarted","Data":"75ea650a0a1620a12fbb77492887c80ae63a95522ec57f29b0423a6bb908b505"}
Sep 09 16:36:36.495556 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.495397    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-77176368-f6ff4df49-4krb5"
Sep 09 16:36:36.495556 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.495421    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-77176368-f6ff4df49-4krb5" event={"ID":"c55859b4-dce2-4c0a-ad42-dfde959d2773","Type":"ContainerStarted","Data":"a10b8e99992e42cdeae0cc0755fd5578d193da56606f5bdc6ada29387abbe105"}
Sep 09 16:36:36.496753 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.496724    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2" event={"ID":"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4","Type":"ContainerStarted","Data":"bd5c6ec6941b86e5c998d2dd056538f13413836edd5cff09f76af93df60bafb4"}
Sep 09 16:36:36.512321 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.512261    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-77176368-f6ff4df49-4krb5" podStartSLOduration=1.512247562 podStartE2EDuration="1.512247562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:36:36.51095199 +0000 UTC m=+1362.341160052" watchObservedRunningTime="2026-09-09 16:36:36.512247562 +0000 UTC m=+1362.342455611"
Sep 09 16:36:36.515390 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.515362    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:36:36.795322 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:36.793345    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"]
Sep 09 16:36:37.503969 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:37.503933    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj" event={"ID":"e6fe2c8e-a9d5-4435-ae36-a57265b01b57","Type":"ContainerStarted","Data":"acbca82db292ee6e8c62c5d90d4b0d94b334c9aecbcbd31fcccbea139fb95267"}
Sep 09 16:36:37.504455 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:37.503983    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj" event={"ID":"e6fe2c8e-a9d5-4435-ae36-a57265b01b57","Type":"ContainerStarted","Data":"907a7e0044618480e822ce757a7a0ec46eebf0f507a13d717480b3437036be6d"}
Sep 09 16:36:37.505631 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:37.505603    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2" event={"ID":"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4","Type":"ContainerStarted","Data":"aa45717e270f6a2d81c2275ce1350ef1e2151836f9e41c968654491f9358dc79"}
Sep 09 16:36:37.505825 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:37.505813    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"
Sep 09 16:36:37.523521 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:37.523477    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj" podStartSLOduration=1.52346428 podStartE2EDuration="1.52346428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:36:37.521011733 +0000 UTC m=+1363.351219792" watchObservedRunningTime="2026-09-09 16:36:37.52346428 +0000 UTC m=+1363.353672337"
Sep 09 16:36:37.541411 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:37.541364    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2" podStartSLOduration=2.858380165 podStartE2EDuration="3.541346268s" totalImagesPullingTime="682.966103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:36:37.538335211 +0000 UTC m=+1363.368543278" watchObservedRunningTime="2026-09-09 16:36:37.541346268 +0000 UTC m=+1363.371554329"
Sep 09 16:36:42.706322 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:42.706271    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:36:43.516897 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:43.516868    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"
Sep 09 16:36:46.515855 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:46.515811    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:36:46.516220 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:46.515874    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:36:46.528405 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:46.528356    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:36:46.571731 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:46.571704    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:36:47.508044 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:47.508010    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-77176368-f6ff4df49-4krb5"
Sep 09 16:36:52.698373 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:52.698343    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"]
Sep 09 16:36:54.887619 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:54.887590    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"]
Sep 09 16:36:59.792037 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:36:59.792004    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:37:03.562641 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:03.562609    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"]
Sep 09 16:37:03.563107 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:03.562945    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9" podUID="dfc91adb-3975-4bea-adbe-123dc342d630" containerName="maas-api" containerID="cri-o://d190c2d006c9ea14363ae1af01f4a0fa223e72a6b7ec5e0939f052689ae8ecb3" gracePeriod=30
Sep 09 16:37:03.809189 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:03.809162    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"
Sep 09 16:37:03.990147 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:03.990117    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kv2rn\" (UniqueName: \"kubernetes.io/projected/dfc91adb-3975-4bea-adbe-123dc342d630-kube-api-access-kv2rn\") pod \"dfc91adb-3975-4bea-adbe-123dc342d630\" (UID: \"dfc91adb-3975-4bea-adbe-123dc342d630\") "
Sep 09 16:37:03.990342 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:03.990201    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc91adb-3975-4bea-adbe-123dc342d630-metrics-certs\") pod \"dfc91adb-3975-4bea-adbe-123dc342d630\" (UID: \"dfc91adb-3975-4bea-adbe-123dc342d630\") "
Sep 09 16:37:03.990342 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:03.990234    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc91adb-3975-4bea-adbe-123dc342d630-maas-api-tls\") pod \"dfc91adb-3975-4bea-adbe-123dc342d630\" (UID: \"dfc91adb-3975-4bea-adbe-123dc342d630\") "
Sep 09 16:37:03.990342 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:03.990251    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc91adb-3975-4bea-adbe-123dc342d630-openshift-service-ca\") pod \"dfc91adb-3975-4bea-adbe-123dc342d630\" (UID: \"dfc91adb-3975-4bea-adbe-123dc342d630\") "
Sep 09 16:37:03.990689 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:03.990659    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc91adb-3975-4bea-adbe-123dc342d630-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dfc91adb-3975-4bea-adbe-123dc342d630" (UID: "dfc91adb-3975-4bea-adbe-123dc342d630"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:37:03.992256 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:03.992228    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc91adb-3975-4bea-adbe-123dc342d630-kube-api-access-kv2rn" (OuterVolumeSpecName: "kube-api-access-kv2rn") pod "dfc91adb-3975-4bea-adbe-123dc342d630" (UID: "dfc91adb-3975-4bea-adbe-123dc342d630"). InnerVolumeSpecName "kube-api-access-kv2rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:37:03.992401 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:03.992255    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfc91adb-3975-4bea-adbe-123dc342d630-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dfc91adb-3975-4bea-adbe-123dc342d630" (UID: "dfc91adb-3975-4bea-adbe-123dc342d630"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:37:03.992401 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:03.992282    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfc91adb-3975-4bea-adbe-123dc342d630-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dfc91adb-3975-4bea-adbe-123dc342d630" (UID: "dfc91adb-3975-4bea-adbe-123dc342d630"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:37:04.092110 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:04.092070    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv2rn\" (UniqueName: \"kubernetes.io/projected/dfc91adb-3975-4bea-adbe-123dc342d630-kube-api-access-kv2rn\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:04.092110 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:04.092103    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfc91adb-3975-4bea-adbe-123dc342d630-metrics-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:04.092110 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:04.092114    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc91adb-3975-4bea-adbe-123dc342d630-maas-api-tls\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:04.092405 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:04.092124    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc91adb-3975-4bea-adbe-123dc342d630-openshift-service-ca\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:04.668599 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:04.668560    2555 generic.go:356] "Generic (PLEG): container finished" podID="dfc91adb-3975-4bea-adbe-123dc342d630" containerID="d190c2d006c9ea14363ae1af01f4a0fa223e72a6b7ec5e0939f052689ae8ecb3" exitCode=0
Sep 09 16:37:04.668999 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:04.668622    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"
Sep 09 16:37:04.668999 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:04.668641    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9" event={"ID":"dfc91adb-3975-4bea-adbe-123dc342d630","Type":"ContainerDied","Data":"d190c2d006c9ea14363ae1af01f4a0fa223e72a6b7ec5e0939f052689ae8ecb3"}
Sep 09 16:37:04.668999 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:04.668699    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9" event={"ID":"dfc91adb-3975-4bea-adbe-123dc342d630","Type":"ContainerDied","Data":"d597293ce46eec052acde8e8dac801d16c315e68497a561899eb940cf5410585"}
Sep 09 16:37:04.668999 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:04.668723    2555 scope.go:117] "RemoveContainer" containerID="d190c2d006c9ea14363ae1af01f4a0fa223e72a6b7ec5e0939f052689ae8ecb3"
Sep 09 16:37:04.677772 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:04.677753    2555 scope.go:117] "RemoveContainer" containerID="d190c2d006c9ea14363ae1af01f4a0fa223e72a6b7ec5e0939f052689ae8ecb3"
Sep 09 16:37:04.678058 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:37:04.678039    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d190c2d006c9ea14363ae1af01f4a0fa223e72a6b7ec5e0939f052689ae8ecb3\": container with ID starting with d190c2d006c9ea14363ae1af01f4a0fa223e72a6b7ec5e0939f052689ae8ecb3 not found: ID does not exist" containerID="d190c2d006c9ea14363ae1af01f4a0fa223e72a6b7ec5e0939f052689ae8ecb3"
Sep 09 16:37:04.678136 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:04.678068    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d190c2d006c9ea14363ae1af01f4a0fa223e72a6b7ec5e0939f052689ae8ecb3"} err="failed to get container status \"d190c2d006c9ea14363ae1af01f4a0fa223e72a6b7ec5e0939f052689ae8ecb3\": rpc error: code = NotFound desc = could not find container \"d190c2d006c9ea14363ae1af01f4a0fa223e72a6b7ec5e0939f052689ae8ecb3\": container with ID starting with d190c2d006c9ea14363ae1af01f4a0fa223e72a6b7ec5e0939f052689ae8ecb3 not found: ID does not exist"
Sep 09 16:37:04.686573 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:04.686546    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"]
Sep 09 16:37:04.694574 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:04.694552    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-26563f-677f4dbfcf-788j9"]
Sep 09 16:37:06.608223 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:06.608188    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfc91adb-3975-4bea-adbe-123dc342d630" path="/var/lib/kubelet/pods/dfc91adb-3975-4bea-adbe-123dc342d630/volumes"
Sep 09 16:37:07.465882 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:07.465834    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"]
Sep 09 16:37:07.466103 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:07.466072    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr" podUID="da1cef8f-f790-4448-b049-4912347443ff" containerName="maas-api" containerID="cri-o://cb25d84828e257eb80256e28f8ac0613b8d178728f1123803079d5c19b206d40" gracePeriod=30
Sep 09 16:37:07.524244 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:07.524212    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd"]
Sep 09 16:37:07.524723 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:07.524693    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd" podUID="1bf1cf22-47e8-4729-9049-ff4c40bb6e8c" containerName="payload-pre-processing" containerID="cri-o://12bac8ce95c8ef820166dd208f884da0bf63a9f3c80fe67927c0b61f142b85e2" gracePeriod=30
Sep 09 16:37:07.688915 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:07.688884    2555 generic.go:356] "Generic (PLEG): container finished" podID="1bf1cf22-47e8-4729-9049-ff4c40bb6e8c" containerID="12bac8ce95c8ef820166dd208f884da0bf63a9f3c80fe67927c0b61f142b85e2" exitCode=0
Sep 09 16:37:07.689271 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:07.688971    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd" event={"ID":"1bf1cf22-47e8-4729-9049-ff4c40bb6e8c","Type":"ContainerDied","Data":"12bac8ce95c8ef820166dd208f884da0bf63a9f3c80fe67927c0b61f142b85e2"}
Sep 09 16:37:07.718854 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:07.718777    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj" podUID="e6fe2c8e-a9d5-4435-ae36-a57265b01b57" containerName="main" containerID="cri-o://acbca82db292ee6e8c62c5d90d4b0d94b334c9aecbcbd31fcccbea139fb95267" gracePeriod=45
Sep 09 16:37:07.865739 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:07.865715    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd"
Sep 09 16:37:07.972057 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:07.971993    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"
Sep 09 16:37:08.008356 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.008326    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:37:08.029741 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.029718    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ctlq\" (UniqueName: \"kubernetes.io/projected/1bf1cf22-47e8-4729-9049-ff4c40bb6e8c-kube-api-access-7ctlq\") pod \"1bf1cf22-47e8-4729-9049-ff4c40bb6e8c\" (UID: \"1bf1cf22-47e8-4729-9049-ff4c40bb6e8c\") "
Sep 09 16:37:08.029864 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.029759    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf1cf22-47e8-4729-9049-ff4c40bb6e8c-plugins-config-volume\") pod \"1bf1cf22-47e8-4729-9049-ff4c40bb6e8c\" (UID: \"1bf1cf22-47e8-4729-9049-ff4c40bb6e8c\") "
Sep 09 16:37:08.030445 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.030418    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf1cf22-47e8-4729-9049-ff4c40bb6e8c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1bf1cf22-47e8-4729-9049-ff4c40bb6e8c" (UID: "1bf1cf22-47e8-4729-9049-ff4c40bb6e8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:37:08.031830 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.031808    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf1cf22-47e8-4729-9049-ff4c40bb6e8c-kube-api-access-7ctlq" (OuterVolumeSpecName: "kube-api-access-7ctlq") pod "1bf1cf22-47e8-4729-9049-ff4c40bb6e8c" (UID: "1bf1cf22-47e8-4729-9049-ff4c40bb6e8c"). InnerVolumeSpecName "kube-api-access-7ctlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:37:08.130416 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.130388    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-tls-certs\") pod \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\" (UID: \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\") "
Sep 09 16:37:08.130416 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.130422    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da1cef8f-f790-4448-b049-4912347443ff-metrics-certs\") pod \"da1cef8f-f790-4448-b049-4912347443ff\" (UID: \"da1cef8f-f790-4448-b049-4912347443ff\") "
Sep 09 16:37:08.130655 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.130438    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da1cef8f-f790-4448-b049-4912347443ff-maas-api-tls\") pod \"da1cef8f-f790-4448-b049-4912347443ff\" (UID: \"da1cef8f-f790-4448-b049-4912347443ff\") "
Sep 09 16:37:08.130655 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.130482    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-model-cache\") pod \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\" (UID: \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\") "
Sep 09 16:37:08.130655 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.130512    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhlnl\" (UniqueName: \"kubernetes.io/projected/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-kube-api-access-fhlnl\") pod \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\" (UID: \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\") "
Sep 09 16:37:08.130655 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.130527    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-tmp-dir\") pod \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\" (UID: \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\") "
Sep 09 16:37:08.130655 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.130568    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq5b6\" (UniqueName: \"kubernetes.io/projected/da1cef8f-f790-4448-b049-4912347443ff-kube-api-access-zq5b6\") pod \"da1cef8f-f790-4448-b049-4912347443ff\" (UID: \"da1cef8f-f790-4448-b049-4912347443ff\") "
Sep 09 16:37:08.130655 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.130621    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da1cef8f-f790-4448-b049-4912347443ff-openshift-service-ca\") pod \"da1cef8f-f790-4448-b049-4912347443ff\" (UID: \"da1cef8f-f790-4448-b049-4912347443ff\") "
Sep 09 16:37:08.130655 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.130645    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-dshm\") pod \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\" (UID: \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\") "
Sep 09 16:37:08.131010 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.130671    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-home\") pod \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\" (UID: \"e6fe2c8e-a9d5-4435-ae36-a57265b01b57\") "
Sep 09 16:37:08.131010 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.130763    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6fe2c8e-a9d5-4435-ae36-a57265b01b57" (UID: "e6fe2c8e-a9d5-4435-ae36-a57265b01b57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:08.131010 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.130801    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6fe2c8e-a9d5-4435-ae36-a57265b01b57" (UID: "e6fe2c8e-a9d5-4435-ae36-a57265b01b57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:08.131010 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.130946    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-home" (OuterVolumeSpecName: "home") pod "e6fe2c8e-a9d5-4435-ae36-a57265b01b57" (UID: "e6fe2c8e-a9d5-4435-ae36-a57265b01b57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:08.131010 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.130967    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-model-cache\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:08.131010 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.131002    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-tmp-dir\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:08.131348 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.131019    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ctlq\" (UniqueName: \"kubernetes.io/projected/1bf1cf22-47e8-4729-9049-ff4c40bb6e8c-kube-api-access-7ctlq\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:08.131348 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.131036    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf1cf22-47e8-4729-9049-ff4c40bb6e8c-plugins-config-volume\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:08.131348 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.131111    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da1cef8f-f790-4448-b049-4912347443ff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "da1cef8f-f790-4448-b049-4912347443ff" (UID: "da1cef8f-f790-4448-b049-4912347443ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:37:08.133169 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.133031    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da1cef8f-f790-4448-b049-4912347443ff-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "da1cef8f-f790-4448-b049-4912347443ff" (UID: "da1cef8f-f790-4448-b049-4912347443ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:37:08.133169 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.133128    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-kube-api-access-fhlnl" (OuterVolumeSpecName: "kube-api-access-fhlnl") pod "e6fe2c8e-a9d5-4435-ae36-a57265b01b57" (UID: "e6fe2c8e-a9d5-4435-ae36-a57265b01b57"). InnerVolumeSpecName "kube-api-access-fhlnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:37:08.133368 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.133231    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da1cef8f-f790-4448-b049-4912347443ff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "da1cef8f-f790-4448-b049-4912347443ff" (UID: "da1cef8f-f790-4448-b049-4912347443ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:37:08.133368 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.133351    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1cef8f-f790-4448-b049-4912347443ff-kube-api-access-zq5b6" (OuterVolumeSpecName: "kube-api-access-zq5b6") pod "da1cef8f-f790-4448-b049-4912347443ff" (UID: "da1cef8f-f790-4448-b049-4912347443ff"). InnerVolumeSpecName "kube-api-access-zq5b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:37:08.133456 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.133385    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-dshm" (OuterVolumeSpecName: "dshm") pod "e6fe2c8e-a9d5-4435-ae36-a57265b01b57" (UID: "e6fe2c8e-a9d5-4435-ae36-a57265b01b57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:08.133964 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.133945    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6fe2c8e-a9d5-4435-ae36-a57265b01b57" (UID: "e6fe2c8e-a9d5-4435-ae36-a57265b01b57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:37:08.231746 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.231653    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhlnl\" (UniqueName: \"kubernetes.io/projected/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-kube-api-access-fhlnl\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:08.231746 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.231682    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq5b6\" (UniqueName: \"kubernetes.io/projected/da1cef8f-f790-4448-b049-4912347443ff-kube-api-access-zq5b6\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:08.231746 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.231691    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da1cef8f-f790-4448-b049-4912347443ff-openshift-service-ca\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:08.231746 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.231700    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-dshm\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:08.231746 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.231709    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-home\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:08.231746 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.231717    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6fe2c8e-a9d5-4435-ae36-a57265b01b57-tls-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:08.231746 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.231728    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da1cef8f-f790-4448-b049-4912347443ff-metrics-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:08.231746 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.231735    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da1cef8f-f790-4448-b049-4912347443ff-maas-api-tls\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:08.697913 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.697879    2555 generic.go:356] "Generic (PLEG): container finished" podID="da1cef8f-f790-4448-b049-4912347443ff" containerID="cb25d84828e257eb80256e28f8ac0613b8d178728f1123803079d5c19b206d40" exitCode=0
Sep 09 16:37:08.698364 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.697931    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"
Sep 09 16:37:08.698364 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.697962    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr" event={"ID":"da1cef8f-f790-4448-b049-4912347443ff","Type":"ContainerDied","Data":"cb25d84828e257eb80256e28f8ac0613b8d178728f1123803079d5c19b206d40"}
Sep 09 16:37:08.698364 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.698020    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr" event={"ID":"da1cef8f-f790-4448-b049-4912347443ff","Type":"ContainerDied","Data":"7e5001402858bc72f0dd7ed963d0b0741725ffbdb2b867a9c291b1f95388dbf0"}
Sep 09 16:37:08.698364 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.698063    2555 scope.go:117] "RemoveContainer" containerID="cb25d84828e257eb80256e28f8ac0613b8d178728f1123803079d5c19b206d40"
Sep 09 16:37:08.699469 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.699428    2555 generic.go:356] "Generic (PLEG): container finished" podID="e6fe2c8e-a9d5-4435-ae36-a57265b01b57" containerID="acbca82db292ee6e8c62c5d90d4b0d94b334c9aecbcbd31fcccbea139fb95267" exitCode=0
Sep 09 16:37:08.699565 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.699457    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj" event={"ID":"e6fe2c8e-a9d5-4435-ae36-a57265b01b57","Type":"ContainerDied","Data":"acbca82db292ee6e8c62c5d90d4b0d94b334c9aecbcbd31fcccbea139fb95267"}
Sep 09 16:37:08.699565 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.699486    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"
Sep 09 16:37:08.699565 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.699497    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj" event={"ID":"e6fe2c8e-a9d5-4435-ae36-a57265b01b57","Type":"ContainerDied","Data":"907a7e0044618480e822ce757a7a0ec46eebf0f507a13d717480b3437036be6d"}
Sep 09 16:37:08.701279 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.701252    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd"
Sep 09 16:37:08.701383 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.701281    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd" event={"ID":"1bf1cf22-47e8-4729-9049-ff4c40bb6e8c","Type":"ContainerDied","Data":"07e6117fe75a707afd11f6477c98739c75a07febeb15325bda71cd8bb7d01c5d"}
Sep 09 16:37:08.709117 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.709102    2555 scope.go:117] "RemoveContainer" containerID="cb25d84828e257eb80256e28f8ac0613b8d178728f1123803079d5c19b206d40"
Sep 09 16:37:08.709389 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:37:08.709370    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb25d84828e257eb80256e28f8ac0613b8d178728f1123803079d5c19b206d40\": container with ID starting with cb25d84828e257eb80256e28f8ac0613b8d178728f1123803079d5c19b206d40 not found: ID does not exist" containerID="cb25d84828e257eb80256e28f8ac0613b8d178728f1123803079d5c19b206d40"
Sep 09 16:37:08.709449 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.709399    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb25d84828e257eb80256e28f8ac0613b8d178728f1123803079d5c19b206d40"} err="failed to get container status \"cb25d84828e257eb80256e28f8ac0613b8d178728f1123803079d5c19b206d40\": rpc error: code = NotFound desc = could not find container \"cb25d84828e257eb80256e28f8ac0613b8d178728f1123803079d5c19b206d40\": container with ID starting with cb25d84828e257eb80256e28f8ac0613b8d178728f1123803079d5c19b206d40 not found: ID does not exist"
Sep 09 16:37:08.709449 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.709417    2555 scope.go:117] "RemoveContainer" containerID="acbca82db292ee6e8c62c5d90d4b0d94b334c9aecbcbd31fcccbea139fb95267"
Sep 09 16:37:08.717436 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.717415    2555 scope.go:117] "RemoveContainer" containerID="acbca82db292ee6e8c62c5d90d4b0d94b334c9aecbcbd31fcccbea139fb95267"
Sep 09 16:37:08.717776 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:37:08.717753    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acbca82db292ee6e8c62c5d90d4b0d94b334c9aecbcbd31fcccbea139fb95267\": container with ID starting with acbca82db292ee6e8c62c5d90d4b0d94b334c9aecbcbd31fcccbea139fb95267 not found: ID does not exist" containerID="acbca82db292ee6e8c62c5d90d4b0d94b334c9aecbcbd31fcccbea139fb95267"
Sep 09 16:37:08.717877 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.717791    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acbca82db292ee6e8c62c5d90d4b0d94b334c9aecbcbd31fcccbea139fb95267"} err="failed to get container status \"acbca82db292ee6e8c62c5d90d4b0d94b334c9aecbcbd31fcccbea139fb95267\": rpc error: code = NotFound desc = could not find container \"acbca82db292ee6e8c62c5d90d4b0d94b334c9aecbcbd31fcccbea139fb95267\": container with ID starting with acbca82db292ee6e8c62c5d90d4b0d94b334c9aecbcbd31fcccbea139fb95267 not found: ID does not exist"
Sep 09 16:37:08.717877 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.717840    2555 scope.go:117] "RemoveContainer" containerID="12bac8ce95c8ef820166dd208f884da0bf63a9f3c80fe67927c0b61f142b85e2"
Sep 09 16:37:08.720812 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.720790    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"]
Sep 09 16:37:08.724031 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.724011    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-26563f/explicit-model-26563f-kserve-86cfb88689-9cqgj"]
Sep 09 16:37:08.733696 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.733673    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd"]
Sep 09 16:37:08.739176 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.739154    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-630acf-c7d8b4dc8-289qd"]
Sep 09 16:37:08.749573 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.749547    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"]
Sep 09 16:37:08.754443 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:08.754164    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-630acf-d67fb46c-8p7gr"]
Sep 09 16:37:09.915431 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:09.915392    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29" podUID="51b453a1-fbde-49db-9760-c2232f773805" containerName="main" containerID="cri-o://632cbd9090f81729237af43d8545e88e086881f4211dc4a1f0107006571b6eb7" gracePeriod=45
Sep 09 16:37:10.185770 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.185748    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:37:10.248609 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.248576    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51b453a1-fbde-49db-9760-c2232f773805-tmp-dir\") pod \"51b453a1-fbde-49db-9760-c2232f773805\" (UID: \"51b453a1-fbde-49db-9760-c2232f773805\") "
Sep 09 16:37:10.248767 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.248616    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51b453a1-fbde-49db-9760-c2232f773805-home\") pod \"51b453a1-fbde-49db-9760-c2232f773805\" (UID: \"51b453a1-fbde-49db-9760-c2232f773805\") "
Sep 09 16:37:10.248767 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.248639    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtw5d\" (UniqueName: \"kubernetes.io/projected/51b453a1-fbde-49db-9760-c2232f773805-kube-api-access-dtw5d\") pod \"51b453a1-fbde-49db-9760-c2232f773805\" (UID: \"51b453a1-fbde-49db-9760-c2232f773805\") "
Sep 09 16:37:10.248767 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.248663    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51b453a1-fbde-49db-9760-c2232f773805-dshm\") pod \"51b453a1-fbde-49db-9760-c2232f773805\" (UID: \"51b453a1-fbde-49db-9760-c2232f773805\") "
Sep 09 16:37:10.248767 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.248683    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51b453a1-fbde-49db-9760-c2232f773805-tls-certs\") pod \"51b453a1-fbde-49db-9760-c2232f773805\" (UID: \"51b453a1-fbde-49db-9760-c2232f773805\") "
Sep 09 16:37:10.248767 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.248751    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51b453a1-fbde-49db-9760-c2232f773805-model-cache\") pod \"51b453a1-fbde-49db-9760-c2232f773805\" (UID: \"51b453a1-fbde-49db-9760-c2232f773805\") "
Sep 09 16:37:10.249039 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.248894    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b453a1-fbde-49db-9760-c2232f773805-home" (OuterVolumeSpecName: "home") pod "51b453a1-fbde-49db-9760-c2232f773805" (UID: "51b453a1-fbde-49db-9760-c2232f773805"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:10.249039 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.248930    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b453a1-fbde-49db-9760-c2232f773805-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "51b453a1-fbde-49db-9760-c2232f773805" (UID: "51b453a1-fbde-49db-9760-c2232f773805"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:10.249144 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.249078    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b453a1-fbde-49db-9760-c2232f773805-model-cache" (OuterVolumeSpecName: "model-cache") pod "51b453a1-fbde-49db-9760-c2232f773805" (UID: "51b453a1-fbde-49db-9760-c2232f773805"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:10.249144 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.249102    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51b453a1-fbde-49db-9760-c2232f773805-home\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:10.249144 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.249121    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51b453a1-fbde-49db-9760-c2232f773805-tmp-dir\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:10.251013 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.250990    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51b453a1-fbde-49db-9760-c2232f773805-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51b453a1-fbde-49db-9760-c2232f773805" (UID: "51b453a1-fbde-49db-9760-c2232f773805"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:37:10.251113 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.251033    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b453a1-fbde-49db-9760-c2232f773805-dshm" (OuterVolumeSpecName: "dshm") pod "51b453a1-fbde-49db-9760-c2232f773805" (UID: "51b453a1-fbde-49db-9760-c2232f773805"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:10.251113 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.251096    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b453a1-fbde-49db-9760-c2232f773805-kube-api-access-dtw5d" (OuterVolumeSpecName: "kube-api-access-dtw5d") pod "51b453a1-fbde-49db-9760-c2232f773805" (UID: "51b453a1-fbde-49db-9760-c2232f773805"). InnerVolumeSpecName "kube-api-access-dtw5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:37:10.349894 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.349851    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtw5d\" (UniqueName: \"kubernetes.io/projected/51b453a1-fbde-49db-9760-c2232f773805-kube-api-access-dtw5d\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:10.349894 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.349878    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51b453a1-fbde-49db-9760-c2232f773805-dshm\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:10.349894 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.349889    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51b453a1-fbde-49db-9760-c2232f773805-tls-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:10.349894 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.349898    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51b453a1-fbde-49db-9760-c2232f773805-model-cache\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:10.608818 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.608772    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bf1cf22-47e8-4729-9049-ff4c40bb6e8c" path="/var/lib/kubelet/pods/1bf1cf22-47e8-4729-9049-ff4c40bb6e8c/volumes"
Sep 09 16:37:10.609262 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.609241    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da1cef8f-f790-4448-b049-4912347443ff" path="/var/lib/kubelet/pods/da1cef8f-f790-4448-b049-4912347443ff/volumes"
Sep 09 16:37:10.609712 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.609693    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6fe2c8e-a9d5-4435-ae36-a57265b01b57" path="/var/lib/kubelet/pods/e6fe2c8e-a9d5-4435-ae36-a57265b01b57/volumes"
Sep 09 16:37:10.717659 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.717629    2555 generic.go:356] "Generic (PLEG): container finished" podID="51b453a1-fbde-49db-9760-c2232f773805" containerID="632cbd9090f81729237af43d8545e88e086881f4211dc4a1f0107006571b6eb7" exitCode=0
Sep 09 16:37:10.717838 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.717673    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29" event={"ID":"51b453a1-fbde-49db-9760-c2232f773805","Type":"ContainerDied","Data":"632cbd9090f81729237af43d8545e88e086881f4211dc4a1f0107006571b6eb7"}
Sep 09 16:37:10.717838 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.717674    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"
Sep 09 16:37:10.717838 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.717702    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29" event={"ID":"51b453a1-fbde-49db-9760-c2232f773805","Type":"ContainerDied","Data":"f8988496cc59ebbc61418aec840a25930103893f953519a515b8570b4a5fd1b9"}
Sep 09 16:37:10.717838 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.717719    2555 scope.go:117] "RemoveContainer" containerID="632cbd9090f81729237af43d8545e88e086881f4211dc4a1f0107006571b6eb7"
Sep 09 16:37:10.728478 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.728459    2555 scope.go:117] "RemoveContainer" containerID="632cbd9090f81729237af43d8545e88e086881f4211dc4a1f0107006571b6eb7"
Sep 09 16:37:10.728761 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:37:10.728736    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"632cbd9090f81729237af43d8545e88e086881f4211dc4a1f0107006571b6eb7\": container with ID starting with 632cbd9090f81729237af43d8545e88e086881f4211dc4a1f0107006571b6eb7 not found: ID does not exist" containerID="632cbd9090f81729237af43d8545e88e086881f4211dc4a1f0107006571b6eb7"
Sep 09 16:37:10.728821 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.728763    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"632cbd9090f81729237af43d8545e88e086881f4211dc4a1f0107006571b6eb7"} err="failed to get container status \"632cbd9090f81729237af43d8545e88e086881f4211dc4a1f0107006571b6eb7\": rpc error: code = NotFound desc = could not find container \"632cbd9090f81729237af43d8545e88e086881f4211dc4a1f0107006571b6eb7\": container with ID starting with 632cbd9090f81729237af43d8545e88e086881f4211dc4a1f0107006571b6eb7 not found: ID does not exist"
Sep 09 16:37:10.736406 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.736364    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"]
Sep 09 16:37:10.740006 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:10.739987    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-34141f/test-model-34141f-kserve-64b78d787f-fwz29"]
Sep 09 16:37:11.101420 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.101386    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:37:11.340771 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.340737    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"]
Sep 09 16:37:11.341322 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.341285    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da1cef8f-f790-4448-b049-4912347443ff" containerName="maas-api"
Sep 09 16:37:11.341322 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.341321    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="da1cef8f-f790-4448-b049-4912347443ff" containerName="maas-api"
Sep 09 16:37:11.341504 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.341334    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bf1cf22-47e8-4729-9049-ff4c40bb6e8c" containerName="payload-pre-processing"
Sep 09 16:37:11.341504 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.341344    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bf1cf22-47e8-4729-9049-ff4c40bb6e8c" containerName="payload-pre-processing"
Sep 09 16:37:11.341504 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.341375    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfc91adb-3975-4bea-adbe-123dc342d630" containerName="maas-api"
Sep 09 16:37:11.341504 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.341384    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfc91adb-3975-4bea-adbe-123dc342d630" containerName="maas-api"
Sep 09 16:37:11.341504 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.341395    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6fe2c8e-a9d5-4435-ae36-a57265b01b57" containerName="main"
Sep 09 16:37:11.341504 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.341404    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6fe2c8e-a9d5-4435-ae36-a57265b01b57" containerName="main"
Sep 09 16:37:11.341504 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.341418    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51b453a1-fbde-49db-9760-c2232f773805" containerName="main"
Sep 09 16:37:11.341504 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.341426    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="51b453a1-fbde-49db-9760-c2232f773805" containerName="main"
Sep 09 16:37:11.341504 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.341502    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bf1cf22-47e8-4729-9049-ff4c40bb6e8c" containerName="payload-pre-processing"
Sep 09 16:37:11.341912 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.341517    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfc91adb-3975-4bea-adbe-123dc342d630" containerName="maas-api"
Sep 09 16:37:11.341912 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.341526    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="da1cef8f-f790-4448-b049-4912347443ff" containerName="maas-api"
Sep 09 16:37:11.341912 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.341535    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="51b453a1-fbde-49db-9760-c2232f773805" containerName="main"
Sep 09 16:37:11.341912 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.341546    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6fe2c8e-a9d5-4435-ae36-a57265b01b57" containerName="main"
Sep 09 16:37:11.346524 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.346501    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.349090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.349067    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-89d246-openshift-default-dockercfg-7ltrh\""
Sep 09 16:37:11.356421 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.356127    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"]
Sep 09 16:37:11.358024 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.357995    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istio-token\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.358184 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.358037    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-workload-certs\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.358184 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.358105    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-workload-socket\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.358184 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.358135    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm6tn\" (UniqueName: \"kubernetes.io/projected/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-kube-api-access-lm6tn\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.358184 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.358163    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istio-data\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.358534 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.358227    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istio-podinfo\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.358534 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.358367    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istiod-ca-cert\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.358534 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.358402    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-credential-socket\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.358534 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.358430    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istio-envoy\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.459002 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.458967    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istio-podinfo\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.459163 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.459014    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istiod-ca-cert\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.459163 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.459033    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-credential-socket\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.459163 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.459123    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istio-envoy\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.459163 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.459147    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istio-token\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.459163 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.459162    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-workload-certs\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.459467 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.459189    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-workload-socket\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.459467 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.459207    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm6tn\" (UniqueName: \"kubernetes.io/projected/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-kube-api-access-lm6tn\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.459467 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.459224    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istio-data\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.459646 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.459547    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-credential-socket\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.459646 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.459600    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-workload-socket\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.459740 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.459652    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istio-data\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.459740 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.459701    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-workload-certs\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.459846 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.459738    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istiod-ca-cert\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.461561 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.461536    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istio-envoy\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.461666 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.461600    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istio-podinfo\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.466511 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.466486    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istio-token\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.466773 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.466755    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm6tn\" (UniqueName: \"kubernetes.io/projected/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-kube-api-access-lm6tn\") pod \"e2e-noten-89d246-openshift-default-5c6957c978-hk6c9\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") " pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.661406 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.661329    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:11.935608 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.935579    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"]
Sep 09 16:37:11.936863 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:37:11.936835    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd23fd4a8_bbdd_4b1d_b182_42ea8c433626.slice/crio-22a11160816edb438d782619d58b3fab705640f844d488292fe1ebf499a05262 WatchSource:0}: Error finding container 22a11160816edb438d782619d58b3fab705640f844d488292fe1ebf499a05262: Status 404 returned error can't find the container with id 22a11160816edb438d782619d58b3fab705640f844d488292fe1ebf499a05262
Sep 09 16:37:11.939047 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.939015    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:37:11.939137 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.939093    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:37:11.939194 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:11.939138    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:37:12.608801 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:12.608772    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51b453a1-fbde-49db-9760-c2232f773805" path="/var/lib/kubelet/pods/51b453a1-fbde-49db-9760-c2232f773805/volumes"
Sep 09 16:37:12.735948 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:12.735912    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9" event={"ID":"d23fd4a8-bbdd-4b1d-b182-42ea8c433626","Type":"ContainerStarted","Data":"b30e577809bc2e22ec9dcdd91e3661d175ac9c5652314cf5ae1a07bf90a28c43"}
Sep 09 16:37:12.736133 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:12.735963    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9" event={"ID":"d23fd4a8-bbdd-4b1d-b182-42ea8c433626","Type":"ContainerStarted","Data":"22a11160816edb438d782619d58b3fab705640f844d488292fe1ebf499a05262"}
Sep 09 16:37:12.753458 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:12.753403    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9" podStartSLOduration=1.75338658 podStartE2EDuration="1.75338658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:37:12.752230244 +0000 UTC m=+1398.582438353" watchObservedRunningTime="2026-09-09 16:37:12.75338658 +0000 UTC m=+1398.583594639"
Sep 09 16:37:13.662178 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:13.662139    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:13.667315 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:13.667277    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:13.741514 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:13.741489    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:13.742377 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:13.742357    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:37:14.547483 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:14.547448    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"]
Sep 09 16:37:14.547705 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:14.547672    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt" podUID="47bc83af-995c-4e42-bc90-29961e944677" containerName="istio-proxy" containerID="cri-o://ac8380b7bbcb54e200cf1135a836275faca688c52dfccf326106870a3005d758" gracePeriod=30
Sep 09 16:37:16.367467 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.367432    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"]
Sep 09 16:37:16.367841 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.367716    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2" podUID="3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4" containerName="maas-api" containerID="cri-o://aa45717e270f6a2d81c2275ce1350ef1e2151836f9e41c968654491f9358dc79" gracePeriod=30
Sep 09 16:37:16.419642 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.419610    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-77176368-f6ff4df49-4krb5"]
Sep 09 16:37:16.419833 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.419808    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-77176368-f6ff4df49-4krb5" podUID="c55859b4-dce2-4c0a-ad42-dfde959d2773" containerName="payload-processing" containerID="cri-o://75ea650a0a1620a12fbb77492887c80ae63a95522ec57f29b0423a6bb908b505" gracePeriod=30
Sep 09 16:37:16.692111 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.692088    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"
Sep 09 16:37:16.694989 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.694968    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-77176368-f6ff4df49-4krb5"
Sep 09 16:37:16.703338 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.703315    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-maas-api-tls\") pod \"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4\" (UID: \"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4\") "
Sep 09 16:37:16.703423 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.703348    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzfk5\" (UniqueName: \"kubernetes.io/projected/c55859b4-dce2-4c0a-ad42-dfde959d2773-kube-api-access-wzfk5\") pod \"c55859b4-dce2-4c0a-ad42-dfde959d2773\" (UID: \"c55859b4-dce2-4c0a-ad42-dfde959d2773\") "
Sep 09 16:37:16.703423 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.703373    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-openshift-service-ca\") pod \"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4\" (UID: \"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4\") "
Sep 09 16:37:16.703538 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.703439    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-metrics-certs\") pod \"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4\" (UID: \"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4\") "
Sep 09 16:37:16.703538 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.703453    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7gjs\" (UniqueName: \"kubernetes.io/projected/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-kube-api-access-n7gjs\") pod \"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4\" (UID: \"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4\") "
Sep 09 16:37:16.703538 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.703468    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c55859b4-dce2-4c0a-ad42-dfde959d2773-plugins-config-volume\") pod \"c55859b4-dce2-4c0a-ad42-dfde959d2773\" (UID: \"c55859b4-dce2-4c0a-ad42-dfde959d2773\") "
Sep 09 16:37:16.703827 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.703793    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4" (UID: "3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:37:16.703927 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.703865    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c55859b4-dce2-4c0a-ad42-dfde959d2773-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c55859b4-dce2-4c0a-ad42-dfde959d2773" (UID: "c55859b4-dce2-4c0a-ad42-dfde959d2773"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:37:16.705528 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.705503    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c55859b4-dce2-4c0a-ad42-dfde959d2773-kube-api-access-wzfk5" (OuterVolumeSpecName: "kube-api-access-wzfk5") pod "c55859b4-dce2-4c0a-ad42-dfde959d2773" (UID: "c55859b4-dce2-4c0a-ad42-dfde959d2773"). InnerVolumeSpecName "kube-api-access-wzfk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:37:16.705617 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.705527    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4" (UID: "3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:37:16.705617 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.705597    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-kube-api-access-n7gjs" (OuterVolumeSpecName: "kube-api-access-n7gjs") pod "3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4" (UID: "3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4"). InnerVolumeSpecName "kube-api-access-n7gjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:37:16.705744 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.705722    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4" (UID: "3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:37:16.762799 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.762764    2555 generic.go:356] "Generic (PLEG): container finished" podID="3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4" containerID="aa45717e270f6a2d81c2275ce1350ef1e2151836f9e41c968654491f9358dc79" exitCode=0
Sep 09 16:37:16.762977 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.762813    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"
Sep 09 16:37:16.762977 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.762853    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2" event={"ID":"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4","Type":"ContainerDied","Data":"aa45717e270f6a2d81c2275ce1350ef1e2151836f9e41c968654491f9358dc79"}
Sep 09 16:37:16.762977 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.762910    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2" event={"ID":"3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4","Type":"ContainerDied","Data":"bd5c6ec6941b86e5c998d2dd056538f13413836edd5cff09f76af93df60bafb4"}
Sep 09 16:37:16.762977 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.762932    2555 scope.go:117] "RemoveContainer" containerID="aa45717e270f6a2d81c2275ce1350ef1e2151836f9e41c968654491f9358dc79"
Sep 09 16:37:16.764231 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.764209    2555 generic.go:356] "Generic (PLEG): container finished" podID="c55859b4-dce2-4c0a-ad42-dfde959d2773" containerID="75ea650a0a1620a12fbb77492887c80ae63a95522ec57f29b0423a6bb908b505" exitCode=0
Sep 09 16:37:16.764347 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.764251    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-77176368-f6ff4df49-4krb5"
Sep 09 16:37:16.764347 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.764260    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-77176368-f6ff4df49-4krb5" event={"ID":"c55859b4-dce2-4c0a-ad42-dfde959d2773","Type":"ContainerDied","Data":"75ea650a0a1620a12fbb77492887c80ae63a95522ec57f29b0423a6bb908b505"}
Sep 09 16:37:16.764347 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.764327    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-77176368-f6ff4df49-4krb5" event={"ID":"c55859b4-dce2-4c0a-ad42-dfde959d2773","Type":"ContainerDied","Data":"a10b8e99992e42cdeae0cc0755fd5578d193da56606f5bdc6ada29387abbe105"}
Sep 09 16:37:16.773500 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.773480    2555 scope.go:117] "RemoveContainer" containerID="aa45717e270f6a2d81c2275ce1350ef1e2151836f9e41c968654491f9358dc79"
Sep 09 16:37:16.773846 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:37:16.773818    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa45717e270f6a2d81c2275ce1350ef1e2151836f9e41c968654491f9358dc79\": container with ID starting with aa45717e270f6a2d81c2275ce1350ef1e2151836f9e41c968654491f9358dc79 not found: ID does not exist" containerID="aa45717e270f6a2d81c2275ce1350ef1e2151836f9e41c968654491f9358dc79"
Sep 09 16:37:16.773950 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.773855    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa45717e270f6a2d81c2275ce1350ef1e2151836f9e41c968654491f9358dc79"} err="failed to get container status \"aa45717e270f6a2d81c2275ce1350ef1e2151836f9e41c968654491f9358dc79\": rpc error: code = NotFound desc = could not find container \"aa45717e270f6a2d81c2275ce1350ef1e2151836f9e41c968654491f9358dc79\": container with ID starting with aa45717e270f6a2d81c2275ce1350ef1e2151836f9e41c968654491f9358dc79 not found: ID does not exist"
Sep 09 16:37:16.773950 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.773881    2555 scope.go:117] "RemoveContainer" containerID="75ea650a0a1620a12fbb77492887c80ae63a95522ec57f29b0423a6bb908b505"
Sep 09 16:37:16.782314 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.782282    2555 scope.go:117] "RemoveContainer" containerID="75ea650a0a1620a12fbb77492887c80ae63a95522ec57f29b0423a6bb908b505"
Sep 09 16:37:16.782565 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:37:16.782547    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ea650a0a1620a12fbb77492887c80ae63a95522ec57f29b0423a6bb908b505\": container with ID starting with 75ea650a0a1620a12fbb77492887c80ae63a95522ec57f29b0423a6bb908b505 not found: ID does not exist" containerID="75ea650a0a1620a12fbb77492887c80ae63a95522ec57f29b0423a6bb908b505"
Sep 09 16:37:16.782622 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.782570    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ea650a0a1620a12fbb77492887c80ae63a95522ec57f29b0423a6bb908b505"} err="failed to get container status \"75ea650a0a1620a12fbb77492887c80ae63a95522ec57f29b0423a6bb908b505\": rpc error: code = NotFound desc = could not find container \"75ea650a0a1620a12fbb77492887c80ae63a95522ec57f29b0423a6bb908b505\": container with ID starting with 75ea650a0a1620a12fbb77492887c80ae63a95522ec57f29b0423a6bb908b505 not found: ID does not exist"
Sep 09 16:37:16.787986 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.787965    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"]
Sep 09 16:37:16.792210 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.792190    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-77176368-66f85c676f-nght2"]
Sep 09 16:37:16.804222 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.804201    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-metrics-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:16.804341 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.804227    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7gjs\" (UniqueName: \"kubernetes.io/projected/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-kube-api-access-n7gjs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:16.804341 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.804241    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c55859b4-dce2-4c0a-ad42-dfde959d2773-plugins-config-volume\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:16.804341 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.804254    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-maas-api-tls\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:16.804341 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.804266    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzfk5\" (UniqueName: \"kubernetes.io/projected/c55859b4-dce2-4c0a-ad42-dfde959d2773-kube-api-access-wzfk5\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:16.804341 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.804278    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4-openshift-service-ca\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:16.806373 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.806355    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-77176368-f6ff4df49-4krb5"]
Sep 09 16:37:16.818653 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:16.818629    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-77176368-f6ff4df49-4krb5"]
Sep 09 16:37:18.608215 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:18.608180    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4" path="/var/lib/kubelet/pods/3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4/volumes"
Sep 09 16:37:18.608636 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:18.608620    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c55859b4-dce2-4c0a-ad42-dfde959d2773" path="/var/lib/kubelet/pods/c55859b4-dce2-4c0a-ad42-dfde959d2773/volumes"
Sep 09 16:37:19.385795 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.385759    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:37:19.786182 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.786148    2555 generic.go:356] "Generic (PLEG): container finished" podID="47bc83af-995c-4e42-bc90-29961e944677" containerID="ac8380b7bbcb54e200cf1135a836275faca688c52dfccf326106870a3005d758" exitCode=0
Sep 09 16:37:19.786611 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.786221    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt" event={"ID":"47bc83af-995c-4e42-bc90-29961e944677","Type":"ContainerDied","Data":"ac8380b7bbcb54e200cf1135a836275faca688c52dfccf326106870a3005d758"}
Sep 09 16:37:19.786611 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.786282    2555 scope.go:117] "RemoveContainer" containerID="c358c8ff041a4b99cdb0489577849c054d6a7f5ec90401c27c0c25da76c52ee8"
Sep 09 16:37:19.821740 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.821721    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:37:19.934509 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.934430    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47bc83af-995c-4e42-bc90-29961e944677-istiod-ca-cert\") pod \"47bc83af-995c-4e42-bc90-29961e944677\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") "
Sep 09 16:37:19.934509 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.934468    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-workload-socket\") pod \"47bc83af-995c-4e42-bc90-29961e944677\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") "
Sep 09 16:37:19.934509 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.934485    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-istio-data\") pod \"47bc83af-995c-4e42-bc90-29961e944677\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") "
Sep 09 16:37:19.934772 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.934532    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-istio-envoy\") pod \"47bc83af-995c-4e42-bc90-29961e944677\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") "
Sep 09 16:37:19.934772 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.934550    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-credential-socket\") pod \"47bc83af-995c-4e42-bc90-29961e944677\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") "
Sep 09 16:37:19.934772 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.934572    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4m5bk\" (UniqueName: \"kubernetes.io/projected/47bc83af-995c-4e42-bc90-29961e944677-kube-api-access-4m5bk\") pod \"47bc83af-995c-4e42-bc90-29961e944677\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") "
Sep 09 16:37:19.934772 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.934614    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47bc83af-995c-4e42-bc90-29961e944677-istio-token\") pod \"47bc83af-995c-4e42-bc90-29961e944677\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") "
Sep 09 16:37:19.934772 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.934649    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47bc83af-995c-4e42-bc90-29961e944677-istio-podinfo\") pod \"47bc83af-995c-4e42-bc90-29961e944677\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") "
Sep 09 16:37:19.934772 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.934683    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-workload-certs\") pod \"47bc83af-995c-4e42-bc90-29961e944677\" (UID: \"47bc83af-995c-4e42-bc90-29961e944677\") "
Sep 09 16:37:19.934772 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.934730    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "47bc83af-995c-4e42-bc90-29961e944677" (UID: "47bc83af-995c-4e42-bc90-29961e944677"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:19.935075 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.934872    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "47bc83af-995c-4e42-bc90-29961e944677" (UID: "47bc83af-995c-4e42-bc90-29961e944677"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:19.935075 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.934931    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47bc83af-995c-4e42-bc90-29961e944677-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "47bc83af-995c-4e42-bc90-29961e944677" (UID: "47bc83af-995c-4e42-bc90-29961e944677"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:37:19.935075 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.934930    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-istio-data" (OuterVolumeSpecName: "istio-data") pod "47bc83af-995c-4e42-bc90-29961e944677" (UID: "47bc83af-995c-4e42-bc90-29961e944677"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:19.935075 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.935070    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47bc83af-995c-4e42-bc90-29961e944677-istiod-ca-cert\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:19.935263 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.935084    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "47bc83af-995c-4e42-bc90-29961e944677" (UID: "47bc83af-995c-4e42-bc90-29961e944677"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:19.935263 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.935090    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-workload-socket\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:19.935263 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.935124    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-istio-data\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:19.935263 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.935134    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-credential-socket\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:19.936902 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.936873    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "47bc83af-995c-4e42-bc90-29961e944677" (UID: "47bc83af-995c-4e42-bc90-29961e944677"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:19.936902 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.936882    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47bc83af-995c-4e42-bc90-29961e944677-istio-token" (OuterVolumeSpecName: "istio-token") pod "47bc83af-995c-4e42-bc90-29961e944677" (UID: "47bc83af-995c-4e42-bc90-29961e944677"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:37:19.937069 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.936884    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47bc83af-995c-4e42-bc90-29961e944677-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "47bc83af-995c-4e42-bc90-29961e944677" (UID: "47bc83af-995c-4e42-bc90-29961e944677"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 16:37:19.937069 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:19.936889    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47bc83af-995c-4e42-bc90-29961e944677-kube-api-access-4m5bk" (OuterVolumeSpecName: "kube-api-access-4m5bk") pod "47bc83af-995c-4e42-bc90-29961e944677" (UID: "47bc83af-995c-4e42-bc90-29961e944677"). InnerVolumeSpecName "kube-api-access-4m5bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:37:20.036012 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:20.035978    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-istio-envoy\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:20.036012 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:20.036005    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m5bk\" (UniqueName: \"kubernetes.io/projected/47bc83af-995c-4e42-bc90-29961e944677-kube-api-access-4m5bk\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:20.036012 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:20.036014    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47bc83af-995c-4e42-bc90-29961e944677-istio-token\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:20.036226 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:20.036025    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47bc83af-995c-4e42-bc90-29961e944677-istio-podinfo\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:20.036226 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:20.036033    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47bc83af-995c-4e42-bc90-29961e944677-workload-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:20.793372 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:20.793344    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"
Sep 09 16:37:20.793372 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:20.793358    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt" event={"ID":"47bc83af-995c-4e42-bc90-29961e944677","Type":"ContainerDied","Data":"316e4e08a20d21baf8cf36c6ebecf05971c3b111ee4b57db2a59c7f8a436f5d5"}
Sep 09 16:37:20.793876 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:20.793413    2555 scope.go:117] "RemoveContainer" containerID="ac8380b7bbcb54e200cf1135a836275faca688c52dfccf326106870a3005d758"
Sep 09 16:37:20.812201 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:20.812180    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"]
Sep 09 16:37:20.814278 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:20.814258    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-630acf-openshift-default-764bc445c6-n9spt"]
Sep 09 16:37:22.607757 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:22.607722    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47bc83af-995c-4e42-bc90-29961e944677" path="/var/lib/kubelet/pods/47bc83af-995c-4e42-bc90-29961e944677/volumes"
Sep 09 16:37:25.283911 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:25.283877    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"]
Sep 09 16:37:25.284440 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:25.284107    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m" podUID="90c83958-db6b-48cb-9468-7ea61b946a3a" containerName="maas-api" containerID="cri-o://1471f6e6b8d479b36c4040359074c172f453dd4b180a14b0e132dff631b74a3b" gracePeriod=30
Sep 09 16:37:25.736734 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:25.736706    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"
Sep 09 16:37:25.791580 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:25.791547    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90c83958-db6b-48cb-9468-7ea61b946a3a-maas-api-tls\") pod \"90c83958-db6b-48cb-9468-7ea61b946a3a\" (UID: \"90c83958-db6b-48cb-9468-7ea61b946a3a\") "
Sep 09 16:37:25.791731 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:25.791588    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2dnw\" (UniqueName: \"kubernetes.io/projected/90c83958-db6b-48cb-9468-7ea61b946a3a-kube-api-access-p2dnw\") pod \"90c83958-db6b-48cb-9468-7ea61b946a3a\" (UID: \"90c83958-db6b-48cb-9468-7ea61b946a3a\") "
Sep 09 16:37:25.791731 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:25.791661    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c83958-db6b-48cb-9468-7ea61b946a3a-openshift-service-ca\") pod \"90c83958-db6b-48cb-9468-7ea61b946a3a\" (UID: \"90c83958-db6b-48cb-9468-7ea61b946a3a\") "
Sep 09 16:37:25.791731 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:25.791677    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c83958-db6b-48cb-9468-7ea61b946a3a-metrics-certs\") pod \"90c83958-db6b-48cb-9468-7ea61b946a3a\" (UID: \"90c83958-db6b-48cb-9468-7ea61b946a3a\") "
Sep 09 16:37:25.791995 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:25.791970    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c83958-db6b-48cb-9468-7ea61b946a3a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "90c83958-db6b-48cb-9468-7ea61b946a3a" (UID: "90c83958-db6b-48cb-9468-7ea61b946a3a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:37:25.793641 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:25.793620    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90c83958-db6b-48cb-9468-7ea61b946a3a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "90c83958-db6b-48cb-9468-7ea61b946a3a" (UID: "90c83958-db6b-48cb-9468-7ea61b946a3a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:37:25.793698 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:25.793681    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c83958-db6b-48cb-9468-7ea61b946a3a-kube-api-access-p2dnw" (OuterVolumeSpecName: "kube-api-access-p2dnw") pod "90c83958-db6b-48cb-9468-7ea61b946a3a" (UID: "90c83958-db6b-48cb-9468-7ea61b946a3a"). InnerVolumeSpecName "kube-api-access-p2dnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:37:25.793748 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:25.793728    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90c83958-db6b-48cb-9468-7ea61b946a3a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "90c83958-db6b-48cb-9468-7ea61b946a3a" (UID: "90c83958-db6b-48cb-9468-7ea61b946a3a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:37:25.824497 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:25.824438    2555 generic.go:356] "Generic (PLEG): container finished" podID="90c83958-db6b-48cb-9468-7ea61b946a3a" containerID="1471f6e6b8d479b36c4040359074c172f453dd4b180a14b0e132dff631b74a3b" exitCode=0
Sep 09 16:37:25.824497 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:25.824483    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"
Sep 09 16:37:25.824622 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:25.824524    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m" event={"ID":"90c83958-db6b-48cb-9468-7ea61b946a3a","Type":"ContainerDied","Data":"1471f6e6b8d479b36c4040359074c172f453dd4b180a14b0e132dff631b74a3b"}
Sep 09 16:37:25.824622 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:25.824572    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m" event={"ID":"90c83958-db6b-48cb-9468-7ea61b946a3a","Type":"ContainerDied","Data":"e8a67acaf6143c566c4775331f9bb72598c6875de19b1bc4725fb192bc21fe8b"}
Sep 09 16:37:25.824622 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:25.824593    2555 scope.go:117] "RemoveContainer" containerID="1471f6e6b8d479b36c4040359074c172f453dd4b180a14b0e132dff631b74a3b"
Sep 09 16:37:25.834353 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:25.834336    2555 scope.go:117] "RemoveContainer" containerID="1471f6e6b8d479b36c4040359074c172f453dd4b180a14b0e132dff631b74a3b"
Sep 09 16:37:25.834633 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:37:25.834614    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1471f6e6b8d479b36c4040359074c172f453dd4b180a14b0e132dff631b74a3b\": container with ID starting with 1471f6e6b8d479b36c4040359074c172f453dd4b180a14b0e132dff631b74a3b not found: ID does not exist" containerID="1471f6e6b8d479b36c4040359074c172f453dd4b180a14b0e132dff631b74a3b"
Sep 09 16:37:25.834700 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:25.834641    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1471f6e6b8d479b36c4040359074c172f453dd4b180a14b0e132dff631b74a3b"} err="failed to get container status \"1471f6e6b8d479b36c4040359074c172f453dd4b180a14b0e132dff631b74a3b\": rpc error: code = NotFound desc = could not find container \"1471f6e6b8d479b36c4040359074c172f453dd4b180a14b0e132dff631b74a3b\": container with ID starting with 1471f6e6b8d479b36c4040359074c172f453dd4b180a14b0e132dff631b74a3b not found: ID does not exist"
Sep 09 16:37:25.846538 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:25.846515    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"]
Sep 09 16:37:25.857741 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:25.857724    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-34141f-6cf4c7b67d-8rr7m"]
Sep 09 16:37:25.892288 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:25.892263    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90c83958-db6b-48cb-9468-7ea61b946a3a-maas-api-tls\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:25.892288 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:25.892288    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2dnw\" (UniqueName: \"kubernetes.io/projected/90c83958-db6b-48cb-9468-7ea61b946a3a-kube-api-access-p2dnw\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:25.892498 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:25.892316    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c83958-db6b-48cb-9468-7ea61b946a3a-openshift-service-ca\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:25.892498 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:25.892336    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c83958-db6b-48cb-9468-7ea61b946a3a-metrics-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:37:26.608116 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:26.608080    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90c83958-db6b-48cb-9468-7ea61b946a3a" path="/var/lib/kubelet/pods/90c83958-db6b-48cb-9468-7ea61b946a3a/volumes"
Sep 09 16:37:27.297598 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:27.297567    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:37:32.570715 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.570672    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"]
Sep 09 16:37:32.571198 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.571177    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90c83958-db6b-48cb-9468-7ea61b946a3a" containerName="maas-api"
Sep 09 16:37:32.571278 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.571201    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="90c83958-db6b-48cb-9468-7ea61b946a3a" containerName="maas-api"
Sep 09 16:37:32.571278 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.571213    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c55859b4-dce2-4c0a-ad42-dfde959d2773" containerName="payload-processing"
Sep 09 16:37:32.571278 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.571222    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c55859b4-dce2-4c0a-ad42-dfde959d2773" containerName="payload-processing"
Sep 09 16:37:32.571278 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.571240    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47bc83af-995c-4e42-bc90-29961e944677" containerName="istio-proxy"
Sep 09 16:37:32.571278 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.571248    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="47bc83af-995c-4e42-bc90-29961e944677" containerName="istio-proxy"
Sep 09 16:37:32.571278 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.571260    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47bc83af-995c-4e42-bc90-29961e944677" containerName="istio-proxy"
Sep 09 16:37:32.571278 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.571268    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="47bc83af-995c-4e42-bc90-29961e944677" containerName="istio-proxy"
Sep 09 16:37:32.571625 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.571287    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4" containerName="maas-api"
Sep 09 16:37:32.571625 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.571311    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4" containerName="maas-api"
Sep 09 16:37:32.571625 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.571432    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="90c83958-db6b-48cb-9468-7ea61b946a3a" containerName="maas-api"
Sep 09 16:37:32.571625 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.571448    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="47bc83af-995c-4e42-bc90-29961e944677" containerName="istio-proxy"
Sep 09 16:37:32.571625 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.571458    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e2e9e9e-e6c8-4466-ba72-0f59d6ab85e4" containerName="maas-api"
Sep 09 16:37:32.571625 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.571470    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="47bc83af-995c-4e42-bc90-29961e944677" containerName="istio-proxy"
Sep 09 16:37:32.571625 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.571479    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c55859b4-dce2-4c0a-ad42-dfde959d2773" containerName="payload-processing"
Sep 09 16:37:32.576770 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.576258    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.578850 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.578830    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-331bf8-openshift-default-dockercfg-xvqwv\""
Sep 09 16:37:32.586801 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.586772    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"]
Sep 09 16:37:32.653518 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.653488    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-workload-socket\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.653667 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.653541    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-workload-certs\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.653667 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.653578    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdprs\" (UniqueName: \"kubernetes.io/projected/a3e0ba21-b30d-40e3-8950-46e7e68958ec-kube-api-access-jdprs\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.653667 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.653619    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istio-podinfo\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.653819 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.653696    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istio-data\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.653819 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.653721    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istiod-ca-cert\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.653819 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.653748    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-credential-socket\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.653819 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.653763    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istio-token\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.653945 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.653842    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istio-envoy\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.754520 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.754488    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istio-data\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.754520 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.754523    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istiod-ca-cert\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.754761 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.754548    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-credential-socket\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.754761 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.754564    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istio-token\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.754761 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.754602    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istio-envoy\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.754761 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.754629    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-workload-socket\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.754761 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.754659    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-workload-certs\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.754761 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.754684    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdprs\" (UniqueName: \"kubernetes.io/projected/a3e0ba21-b30d-40e3-8950-46e7e68958ec-kube-api-access-jdprs\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.754761 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.754730    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istio-podinfo\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.755072 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.754915    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istio-data\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.755072 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.754969    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-credential-socket\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.755178 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.755077    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-workload-certs\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.755236 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.755171    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-workload-socket\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.755362 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.755337    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istiod-ca-cert\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.756966 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.756943    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istio-envoy\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.757225 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.757208    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istio-podinfo\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.762864 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.762840    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdprs\" (UniqueName: \"kubernetes.io/projected/a3e0ba21-b30d-40e3-8950-46e7e68958ec-kube-api-access-jdprs\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.763081 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.762894    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istio-token\") pod \"e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") " pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:32.889973 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:32.889896    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:33.162533 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:33.162505    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"]
Sep 09 16:37:33.163740 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:37:33.163714    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e0ba21_b30d_40e3_8950_46e7e68958ec.slice/crio-0c8723a7c6aa3285f719060e50578eef34d480658ee2c05873626d06c3ea3074 WatchSource:0}: Error finding container 0c8723a7c6aa3285f719060e50578eef34d480658ee2c05873626d06c3ea3074: Status 404 returned error can't find the container with id 0c8723a7c6aa3285f719060e50578eef34d480658ee2c05873626d06c3ea3074
Sep 09 16:37:33.165689 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:33.165654    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:37:33.165764 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:33.165740    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:37:33.165803 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:33.165770    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:37:33.875879 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:33.875843    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76" event={"ID":"a3e0ba21-b30d-40e3-8950-46e7e68958ec","Type":"ContainerStarted","Data":"75a1f82de876d72d188f9dff5a21645b296ee0deb4669fc3ee245e68810104a7"}
Sep 09 16:37:33.876318 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:33.875895    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76" event={"ID":"a3e0ba21-b30d-40e3-8950-46e7e68958ec","Type":"ContainerStarted","Data":"0c8723a7c6aa3285f719060e50578eef34d480658ee2c05873626d06c3ea3074"}
Sep 09 16:37:33.890542 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:33.890513    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:33.894495 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:33.894449    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76" podStartSLOduration=1.894435615 podStartE2EDuration="1.894435615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:37:33.89169248 +0000 UTC m=+1419.721900537" watchObservedRunningTime="2026-09-09 16:37:33.894435615 +0000 UTC m=+1419.724643673"
Sep 09 16:37:34.895195 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:34.895167    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:35.887107 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:35.887080    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:35.888124 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:35.888096    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:36.777149 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.777111    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"]
Sep 09 16:37:36.781307 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.781276    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.783833 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.783814    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-2bdf45-openshift-default-dockercfg-85rjd\""
Sep 09 16:37:36.788716 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.788690    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"]
Sep 09 16:37:36.892425 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.892373    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-workload-socket\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.892736 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.892701    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istio-token\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.892930 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.892915    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istiod-ca-cert\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.893080 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.893062    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istio-podinfo\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.893469 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.893437    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istio-data\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.893613 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.893597    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istio-envoy\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.893705 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.893665    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-workload-certs\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.893775 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.893749    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96rds\" (UniqueName: \"kubernetes.io/projected/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-kube-api-access-96rds\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.893928 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.893910    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-credential-socket\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.995529 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.995485    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istio-token\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.995708 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.995568    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istiod-ca-cert\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.995708 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.995598    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istio-podinfo\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.995949 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.995916    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istio-data\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.996084 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.995964    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istio-envoy\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.996084 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.995992    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-workload-certs\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.996084 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.996036    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96rds\" (UniqueName: \"kubernetes.io/projected/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-kube-api-access-96rds\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.996084 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.996067    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-credential-socket\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.996307 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.996134    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-workload-socket\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.996658 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.996415    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istiod-ca-cert\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.996752 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.996687    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istio-data\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.997062 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.997042    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-workload-socket\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.997169 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.997097    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-credential-socket\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.997169 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.997157    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-workload-certs\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.998654 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.998632    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istio-podinfo\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:36.998972 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:36.998950    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istio-envoy\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:37.007861 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:37.007832    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istio-token\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:37.007947 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:37.007869    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96rds\" (UniqueName: \"kubernetes.io/projected/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-kube-api-access-96rds\") pod \"e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") " pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:37.096725 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:37.096652    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:37.371513 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:37.371484    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"]
Sep 09 16:37:37.372872 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:37:37.372850    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeadb9d1f_2b00_4442_a24e_e8e6f6dcc15a.slice/crio-7c6035acafcbbc984eb258552825ad454d9656cd15978f76c646fdc42846b92c WatchSource:0}: Error finding container 7c6035acafcbbc984eb258552825ad454d9656cd15978f76c646fdc42846b92c: Status 404 returned error can't find the container with id 7c6035acafcbbc984eb258552825ad454d9656cd15978f76c646fdc42846b92c
Sep 09 16:37:37.375638 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:37.375604    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:37:37.375726 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:37.375684    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:37:37.375782 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:37.375723    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:37:37.547550 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:37.547516    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk"]
Sep 09 16:37:37.550970 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:37.550946    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk"
Sep 09 16:37:37.553845 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:37.553820    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-331bf8\""
Sep 09 16:37:37.553961 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:37.553866    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-331bf8-dockercfg-pqbbt\""
Sep 09 16:37:37.561897 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:37.561873    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk"]
Sep 09 16:37:37.703927 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:37.703849    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455643fc-fc8d-486d-b035-0c2d9b482612-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk\" (UID: \"455643fc-fc8d-486d-b035-0c2d9b482612\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk"
Sep 09 16:37:37.704141 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:37.704035    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm79c\" (UniqueName: \"kubernetes.io/projected/455643fc-fc8d-486d-b035-0c2d9b482612-kube-api-access-zm79c\") pod \"payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk\" (UID: \"455643fc-fc8d-486d-b035-0c2d9b482612\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk"
Sep 09 16:37:37.804775 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:37.804737    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm79c\" (UniqueName: \"kubernetes.io/projected/455643fc-fc8d-486d-b035-0c2d9b482612-kube-api-access-zm79c\") pod \"payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk\" (UID: \"455643fc-fc8d-486d-b035-0c2d9b482612\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk"
Sep 09 16:37:37.805358 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:37.804794    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455643fc-fc8d-486d-b035-0c2d9b482612-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk\" (UID: \"455643fc-fc8d-486d-b035-0c2d9b482612\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk"
Sep 09 16:37:37.805630 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:37.805607    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455643fc-fc8d-486d-b035-0c2d9b482612-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk\" (UID: \"455643fc-fc8d-486d-b035-0c2d9b482612\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk"
Sep 09 16:37:37.812661 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:37.812640    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm79c\" (UniqueName: \"kubernetes.io/projected/455643fc-fc8d-486d-b035-0c2d9b482612-kube-api-access-zm79c\") pod \"payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk\" (UID: \"455643fc-fc8d-486d-b035-0c2d9b482612\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk"
Sep 09 16:37:37.864529 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:37.864497    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk"
Sep 09 16:37:37.907616 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:37.907578    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p" event={"ID":"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a","Type":"ContainerStarted","Data":"ea4fb7d9f27929664db7a48562105da8c65654d0c1510f6f41ceba0ed5efdfd3"}
Sep 09 16:37:37.907754 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:37.907639    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p" event={"ID":"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a","Type":"ContainerStarted","Data":"7c6035acafcbbc984eb258552825ad454d9656cd15978f76c646fdc42846b92c"}
Sep 09 16:37:37.929996 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:37.929940    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p" podStartSLOduration=1.929919674 podStartE2EDuration="1.929919674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:37:37.929320747 +0000 UTC m=+1423.759528799" watchObservedRunningTime="2026-09-09 16:37:37.929919674 +0000 UTC m=+1423.760127732"
Sep 09 16:37:38.096817 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:38.096785    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:38.102652 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:38.102622    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:38.134068 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:38.134040    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk"]
Sep 09 16:37:38.135250 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:37:38.135225    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455643fc_fc8d_486d_b035_0c2d9b482612.slice/crio-291ec6eea29630634ac11e99b4847efba6d58d1b99733245a32bc9a334296a8f WatchSource:0}: Error finding container 291ec6eea29630634ac11e99b4847efba6d58d1b99733245a32bc9a334296a8f: Status 404 returned error can't find the container with id 291ec6eea29630634ac11e99b4847efba6d58d1b99733245a32bc9a334296a8f
Sep 09 16:37:38.914638 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:38.914602    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk" event={"ID":"455643fc-fc8d-486d-b035-0c2d9b482612","Type":"ContainerStarted","Data":"de00597574f27c829dc84aee60b25e55017b6c7d83c1b6be08e2754ca8322c2f"}
Sep 09 16:37:38.915100 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:38.914662    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk" event={"ID":"455643fc-fc8d-486d-b035-0c2d9b482612","Type":"ContainerStarted","Data":"291ec6eea29630634ac11e99b4847efba6d58d1b99733245a32bc9a334296a8f"}
Sep 09 16:37:38.915100 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:38.914797    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk"
Sep 09 16:37:38.915100 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:38.914837    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:38.915849 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:38.915834    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:37:38.930572 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:38.930531    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk" podStartSLOduration=1.930518908 podStartE2EDuration="1.930518908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:37:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:37:38.929051294 +0000 UTC m=+1424.759259345" watchObservedRunningTime="2026-09-09 16:37:38.930518908 +0000 UTC m=+1424.760727014"
Sep 09 16:37:39.802985 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:39.802943    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:37:41.237393 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:41.237258    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28"]
Sep 09 16:37:41.241738 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:41.241713    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28"
Sep 09 16:37:41.245940 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:41.245902    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-2bdf45-serving-cert\""
Sep 09 16:37:41.251251 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:41.251228    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28"]
Sep 09 16:37:41.337506 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:41.337453    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f63431c3-a6e0-4c22-b7b5-2facc77d6e63-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28\" (UID: \"f63431c3-a6e0-4c22-b7b5-2facc77d6e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28"
Sep 09 16:37:41.337710 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:41.337517    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f63431c3-a6e0-4c22-b7b5-2facc77d6e63-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28\" (UID: \"f63431c3-a6e0-4c22-b7b5-2facc77d6e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28"
Sep 09 16:37:41.337710 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:41.337583    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f63431c3-a6e0-4c22-b7b5-2facc77d6e63-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28\" (UID: \"f63431c3-a6e0-4c22-b7b5-2facc77d6e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28"
Sep 09 16:37:41.337710 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:41.337620    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9jms\" (UniqueName: \"kubernetes.io/projected/f63431c3-a6e0-4c22-b7b5-2facc77d6e63-kube-api-access-p9jms\") pod \"maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28\" (UID: \"f63431c3-a6e0-4c22-b7b5-2facc77d6e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28"
Sep 09 16:37:41.439156 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:41.439118    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f63431c3-a6e0-4c22-b7b5-2facc77d6e63-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28\" (UID: \"f63431c3-a6e0-4c22-b7b5-2facc77d6e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28"
Sep 09 16:37:41.439383 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:41.439314    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f63431c3-a6e0-4c22-b7b5-2facc77d6e63-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28\" (UID: \"f63431c3-a6e0-4c22-b7b5-2facc77d6e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28"
Sep 09 16:37:41.439383 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:41.439374    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9jms\" (UniqueName: \"kubernetes.io/projected/f63431c3-a6e0-4c22-b7b5-2facc77d6e63-kube-api-access-p9jms\") pod \"maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28\" (UID: \"f63431c3-a6e0-4c22-b7b5-2facc77d6e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28"
Sep 09 16:37:41.439565 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:41.439543    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f63431c3-a6e0-4c22-b7b5-2facc77d6e63-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28\" (UID: \"f63431c3-a6e0-4c22-b7b5-2facc77d6e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28"
Sep 09 16:37:41.440054 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:41.440035    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f63431c3-a6e0-4c22-b7b5-2facc77d6e63-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28\" (UID: \"f63431c3-a6e0-4c22-b7b5-2facc77d6e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28"
Sep 09 16:37:41.446311 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:41.442188    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f63431c3-a6e0-4c22-b7b5-2facc77d6e63-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28\" (UID: \"f63431c3-a6e0-4c22-b7b5-2facc77d6e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28"
Sep 09 16:37:41.449395 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:41.449367    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9jms\" (UniqueName: \"kubernetes.io/projected/f63431c3-a6e0-4c22-b7b5-2facc77d6e63-kube-api-access-p9jms\") pod \"maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28\" (UID: \"f63431c3-a6e0-4c22-b7b5-2facc77d6e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28"
Sep 09 16:37:41.454822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:41.454791    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f63431c3-a6e0-4c22-b7b5-2facc77d6e63-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28\" (UID: \"f63431c3-a6e0-4c22-b7b5-2facc77d6e63\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28"
Sep 09 16:37:41.560805 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:41.560767    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28"
Sep 09 16:37:41.840053 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:41.840022    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28"]
Sep 09 16:37:41.857455 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:37:41.850925    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf63431c3_a6e0_4c22_b7b5_2facc77d6e63.slice/crio-036c1df7d13119da6f69b491dd5dcd8ac02d8a51c2af8d1a3416c916336ca967 WatchSource:0}: Error finding container 036c1df7d13119da6f69b491dd5dcd8ac02d8a51c2af8d1a3416c916336ca967: Status 404 returned error can't find the container with id 036c1df7d13119da6f69b491dd5dcd8ac02d8a51c2af8d1a3416c916336ca967
Sep 09 16:37:41.935585 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:41.935552    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28" event={"ID":"f63431c3-a6e0-4c22-b7b5-2facc77d6e63","Type":"ContainerStarted","Data":"036c1df7d13119da6f69b491dd5dcd8ac02d8a51c2af8d1a3416c916336ca967"}
Sep 09 16:37:43.955280 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:43.955232    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28" event={"ID":"f63431c3-a6e0-4c22-b7b5-2facc77d6e63","Type":"ContainerStarted","Data":"21781cf50affdc305e5809277be911f11a6361959923d585352146fcb09ae4c1"}
Sep 09 16:37:43.956108 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:43.956081    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28"
Sep 09 16:37:43.972629 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:43.972581    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28" podStartSLOduration=1.8547934179999999 podStartE2EDuration="2.972565692s" totalImagesPullingTime="1.117772274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:37:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:37:43.971417255 +0000 UTC m=+1429.801625314" watchObservedRunningTime="2026-09-09 16:37:43.972565692 +0000 UTC m=+1429.802773749"
Sep 09 16:37:48.679031 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.678913    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"]
Sep 09 16:37:48.682758 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.682739    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:37:48.686333 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.686307    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-89d246\"/\"noten-model-89d246-kserve-self-signed-certs\""
Sep 09 16:37:48.686463 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.686309    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-89d246\"/\"openshift-service-ca.crt\""
Sep 09 16:37:48.686463 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.686315    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-89d246\"/\"kube-root-ca.crt\""
Sep 09 16:37:48.686463 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.686410    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-89d246\"/\"default-dockercfg-7f4pp\""
Sep 09 16:37:48.692007 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.691843    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"]
Sep 09 16:37:48.823726 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.823677    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-dshm\") pod \"noten-model-89d246-kserve-9f9c9756c-2rpv8\" (UID: \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\") " pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:37:48.823937 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.823755    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-tmp-dir\") pod \"noten-model-89d246-kserve-9f9c9756c-2rpv8\" (UID: \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\") " pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:37:48.823937 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.823798    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-tls-certs\") pod \"noten-model-89d246-kserve-9f9c9756c-2rpv8\" (UID: \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\") " pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:37:48.823937 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.823830    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp8ld\" (UniqueName: \"kubernetes.io/projected/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-kube-api-access-mp8ld\") pod \"noten-model-89d246-kserve-9f9c9756c-2rpv8\" (UID: \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\") " pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:37:48.823937 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.823895    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-model-cache\") pod \"noten-model-89d246-kserve-9f9c9756c-2rpv8\" (UID: \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\") " pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:37:48.823937 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.823935    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-home\") pod \"noten-model-89d246-kserve-9f9c9756c-2rpv8\" (UID: \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\") " pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:37:48.925068 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.925026    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp8ld\" (UniqueName: \"kubernetes.io/projected/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-kube-api-access-mp8ld\") pod \"noten-model-89d246-kserve-9f9c9756c-2rpv8\" (UID: \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\") " pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:37:48.925068 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.925070    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-model-cache\") pod \"noten-model-89d246-kserve-9f9c9756c-2rpv8\" (UID: \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\") " pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:37:48.925377 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.925093    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-home\") pod \"noten-model-89d246-kserve-9f9c9756c-2rpv8\" (UID: \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\") " pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:37:48.925377 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.925118    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-dshm\") pod \"noten-model-89d246-kserve-9f9c9756c-2rpv8\" (UID: \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\") " pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:37:48.925377 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.925172    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-tmp-dir\") pod \"noten-model-89d246-kserve-9f9c9756c-2rpv8\" (UID: \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\") " pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:37:48.925377 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.925230    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-tls-certs\") pod \"noten-model-89d246-kserve-9f9c9756c-2rpv8\" (UID: \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\") " pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:37:48.925617 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.925582    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-model-cache\") pod \"noten-model-89d246-kserve-9f9c9756c-2rpv8\" (UID: \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\") " pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:37:48.925687 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.925631    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-home\") pod \"noten-model-89d246-kserve-9f9c9756c-2rpv8\" (UID: \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\") " pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:37:48.925748 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.925686    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-tmp-dir\") pod \"noten-model-89d246-kserve-9f9c9756c-2rpv8\" (UID: \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\") " pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:37:48.927497 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.927474    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-dshm\") pod \"noten-model-89d246-kserve-9f9c9756c-2rpv8\" (UID: \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\") " pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:37:48.927766 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.927745    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-tls-certs\") pod \"noten-model-89d246-kserve-9f9c9756c-2rpv8\" (UID: \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\") " pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:37:48.932336 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.932256    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp8ld\" (UniqueName: \"kubernetes.io/projected/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-kube-api-access-mp8ld\") pod \"noten-model-89d246-kserve-9f9c9756c-2rpv8\" (UID: \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\") " pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:37:48.995016 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:48.994978    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:37:49.265802 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:49.265769    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"]
Sep 09 16:37:49.268058 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:37:49.268032    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b6f2f29_47de_463b_a23c_ff71b5b74b4d.slice/crio-b6f1b2b4ce3cd09b5b8613577a10641c433d388710f719fae1abb51b342ce19f WatchSource:0}: Error finding container b6f1b2b4ce3cd09b5b8613577a10641c433d388710f719fae1abb51b342ce19f: Status 404 returned error can't find the container with id b6f1b2b4ce3cd09b5b8613577a10641c433d388710f719fae1abb51b342ce19f
Sep 09 16:37:49.922842 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:49.922753    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk"
Sep 09 16:37:49.998887 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:49.998846    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8" event={"ID":"6b6f2f29-47de-463b-a23c-ff71b5b74b4d","Type":"ContainerStarted","Data":"fbe14009805d6453ce6ff1a1f25e7f8a58409f4c3676cb587aab34f4dcabc161"}
Sep 09 16:37:49.999070 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:49.998905    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8" event={"ID":"6b6f2f29-47de-463b-a23c-ff71b5b74b4d","Type":"ContainerStarted","Data":"b6f1b2b4ce3cd09b5b8613577a10641c433d388710f719fae1abb51b342ce19f"}
Sep 09 16:37:50.000444 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:50.000421    2555 generic.go:356] "Generic (PLEG): container finished" podID="a3e0ba21-b30d-40e3-8950-46e7e68958ec" containerID="75a1f82de876d72d188f9dff5a21645b296ee0deb4669fc3ee245e68810104a7" exitCode=0
Sep 09 16:37:50.000536 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:50.000493    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76" event={"ID":"a3e0ba21-b30d-40e3-8950-46e7e68958ec","Type":"ContainerDied","Data":"75a1f82de876d72d188f9dff5a21645b296ee0deb4669fc3ee245e68810104a7"}
Sep 09 16:37:50.000852 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:50.000836    2555 scope.go:117] "RemoveContainer" containerID="75a1f82de876d72d188f9dff5a21645b296ee0deb4669fc3ee245e68810104a7"
Sep 09 16:37:50.001803 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:50.001778    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:37:50.001865 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:50.001843    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:37:50.001904 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:50.001875    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:37:50.014548 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:50.014511    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8" podStartSLOduration=2.014498974 podStartE2EDuration="2.014498974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:37:50.013082083 +0000 UTC m=+1435.843290143" watchObservedRunningTime="2026-09-09 16:37:50.014498974 +0000 UTC m=+1435.844707032"
Sep 09 16:37:50.887500 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:50.887465    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:50.980859 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:50.980830    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28"
Sep 09 16:37:51.008092 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:51.008045    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76" event={"ID":"a3e0ba21-b30d-40e3-8950-46e7e68958ec","Type":"ContainerStarted","Data":"89ac5ce27a743d5ba7f988818bd04328b2da6862a5d3308898325d8be9d46fae"}
Sep 09 16:37:51.890368 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:51.890336    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:51.891527 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:51.891488    2555 patch_prober.go:28] interesting pod/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 09 16:37:51.891589 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:51.891556    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76" podUID="a3e0ba21-b30d-40e3-8950-46e7e68958ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 09 16:37:52.891333 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:52.891262    2555 patch_prober.go:28] interesting pod/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 09 16:37:52.891716 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:52.891352    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76" podUID="a3e0ba21-b30d-40e3-8950-46e7e68958ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 09 16:37:53.403057 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:53.403022    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:37:53.890596 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:53.890567    2555 patch_prober.go:28] interesting pod/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 09 16:37:53.890763 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:53.890619    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76" podUID="a3e0ba21-b30d-40e3-8950-46e7e68958ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 09 16:37:54.891060 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:54.891020    2555 patch_prober.go:28] interesting pod/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 09 16:37:54.891487 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:54.891078    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76" podUID="a3e0ba21-b30d-40e3-8950-46e7e68958ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 09 16:37:55.896341 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:55.896277    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:55.896780 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:55.896759    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:55.897666 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:55.897645    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:37:58.995170 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:58.995133    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:37:58.995566 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:58.995194    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:37:59.007536 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:59.007513    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:37:59.075762 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:37:59.075737    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:38:06.559057 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:06.559013    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff"]
Sep 09 16:38:06.564767 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:06.564741    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff"
Sep 09 16:38:06.567368 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:06.567347    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-50fcbc\""
Sep 09 16:38:06.567494 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:06.567350    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-50fcbc-dockercfg-qwwpm\""
Sep 09 16:38:06.572675 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:06.572644    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff"]
Sep 09 16:38:06.598120 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:06.598081    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x"]
Sep 09 16:38:06.602284 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:06.602257    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x"
Sep 09 16:38:06.613625 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:06.613603    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x"]
Sep 09 16:38:06.729269 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:06.729224    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djgsk\" (UniqueName: \"kubernetes.io/projected/f50bdeb2-c287-4b13-af6f-2a8b29d91469-kube-api-access-djgsk\") pod \"payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff\" (UID: \"f50bdeb2-c287-4b13-af6f-2a8b29d91469\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff"
Sep 09 16:38:06.729269 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:06.729276    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f50bdeb2-c287-4b13-af6f-2a8b29d91469-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff\" (UID: \"f50bdeb2-c287-4b13-af6f-2a8b29d91469\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff"
Sep 09 16:38:06.729536 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:06.729378    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0241646d-a637-4ddf-9c1e-e9e42314ac97-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x\" (UID: \"0241646d-a637-4ddf-9c1e-e9e42314ac97\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x"
Sep 09 16:38:06.729599 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:06.729533    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c579f\" (UniqueName: \"kubernetes.io/projected/0241646d-a637-4ddf-9c1e-e9e42314ac97-kube-api-access-c579f\") pod \"payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x\" (UID: \"0241646d-a637-4ddf-9c1e-e9e42314ac97\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x"
Sep 09 16:38:06.830633 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:06.830544    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c579f\" (UniqueName: \"kubernetes.io/projected/0241646d-a637-4ddf-9c1e-e9e42314ac97-kube-api-access-c579f\") pod \"payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x\" (UID: \"0241646d-a637-4ddf-9c1e-e9e42314ac97\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x"
Sep 09 16:38:06.830633 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:06.830621    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djgsk\" (UniqueName: \"kubernetes.io/projected/f50bdeb2-c287-4b13-af6f-2a8b29d91469-kube-api-access-djgsk\") pod \"payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff\" (UID: \"f50bdeb2-c287-4b13-af6f-2a8b29d91469\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff"
Sep 09 16:38:06.830867 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:06.830652    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f50bdeb2-c287-4b13-af6f-2a8b29d91469-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff\" (UID: \"f50bdeb2-c287-4b13-af6f-2a8b29d91469\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff"
Sep 09 16:38:06.830867 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:06.830702    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0241646d-a637-4ddf-9c1e-e9e42314ac97-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x\" (UID: \"0241646d-a637-4ddf-9c1e-e9e42314ac97\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x"
Sep 09 16:38:06.838342 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:06.831927    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0241646d-a637-4ddf-9c1e-e9e42314ac97-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x\" (UID: \"0241646d-a637-4ddf-9c1e-e9e42314ac97\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x"
Sep 09 16:38:06.838342 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:06.832126    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f50bdeb2-c287-4b13-af6f-2a8b29d91469-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff\" (UID: \"f50bdeb2-c287-4b13-af6f-2a8b29d91469\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff"
Sep 09 16:38:06.842534 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:06.842405    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djgsk\" (UniqueName: \"kubernetes.io/projected/f50bdeb2-c287-4b13-af6f-2a8b29d91469-kube-api-access-djgsk\") pod \"payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff\" (UID: \"f50bdeb2-c287-4b13-af6f-2a8b29d91469\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff"
Sep 09 16:38:06.843182 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:06.843131    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c579f\" (UniqueName: \"kubernetes.io/projected/0241646d-a637-4ddf-9c1e-e9e42314ac97-kube-api-access-c579f\") pod \"payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x\" (UID: \"0241646d-a637-4ddf-9c1e-e9e42314ac97\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x"
Sep 09 16:38:06.879108 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:06.879071    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff"
Sep 09 16:38:06.918444 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:06.918101    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x"
Sep 09 16:38:07.148957 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:07.148928    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff"]
Sep 09 16:38:07.149871 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:38:07.149846    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf50bdeb2_c287_4b13_af6f_2a8b29d91469.slice/crio-2dbc5b1cf0499d51451912a39636e5e37e3079bb47a94d6de4844994a4a040ab WatchSource:0}: Error finding container 2dbc5b1cf0499d51451912a39636e5e37e3079bb47a94d6de4844994a4a040ab: Status 404 returned error can't find the container with id 2dbc5b1cf0499d51451912a39636e5e37e3079bb47a94d6de4844994a4a040ab
Sep 09 16:38:07.220315 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:38:07.220250    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0241646d_a637_4ddf_9c1e_e9e42314ac97.slice/crio-f9b709adca886507160c457312785ffbfe191bfb8e1c133947d91c8b486871f7 WatchSource:0}: Error finding container f9b709adca886507160c457312785ffbfe191bfb8e1c133947d91c8b486871f7: Status 404 returned error can't find the container with id f9b709adca886507160c457312785ffbfe191bfb8e1c133947d91c8b486871f7
Sep 09 16:38:07.220695 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:07.220671    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x"]
Sep 09 16:38:08.118802 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:08.118770    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x" event={"ID":"0241646d-a637-4ddf-9c1e-e9e42314ac97","Type":"ContainerStarted","Data":"c1d88d4fceb84bd04dc0aa86ba6e161dd04bd6564974f1c9743108bd8e49694b"}
Sep 09 16:38:08.119222 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:08.118827    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x" event={"ID":"0241646d-a637-4ddf-9c1e-e9e42314ac97","Type":"ContainerStarted","Data":"f9b709adca886507160c457312785ffbfe191bfb8e1c133947d91c8b486871f7"}
Sep 09 16:38:08.119222 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:08.118975    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x"
Sep 09 16:38:08.120531 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:08.120509    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff" event={"ID":"f50bdeb2-c287-4b13-af6f-2a8b29d91469","Type":"ContainerStarted","Data":"8880099fa4367bbaf5113fffbcdb39cd35f45daf812c57f817ff47ba90faa79c"}
Sep 09 16:38:08.120631 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:08.120544    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff" event={"ID":"f50bdeb2-c287-4b13-af6f-2a8b29d91469","Type":"ContainerStarted","Data":"2dbc5b1cf0499d51451912a39636e5e37e3079bb47a94d6de4844994a4a040ab"}
Sep 09 16:38:08.120631 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:08.120571    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff"
Sep 09 16:38:08.140387 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:08.140342    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x" podStartSLOduration=2.140285786 podStartE2EDuration="2.140285786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:38:08.138043653 +0000 UTC m=+1453.968251712" watchObservedRunningTime="2026-09-09 16:38:08.140285786 +0000 UTC m=+1453.970493844"
Sep 09 16:38:08.157017 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:08.156971    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff" podStartSLOduration=2.156955206 podStartE2EDuration="2.156955206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:38:08.154505903 +0000 UTC m=+1453.984713968" watchObservedRunningTime="2026-09-09 16:38:08.156955206 +0000 UTC m=+1453.987163304"
Sep 09 16:38:08.915706 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:08.915676    2555 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 09 16:38:08.915895 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:08.915730    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p" podUID="eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 09 16:38:09.110221 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:09.110130    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:38:09.126938 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:09.126911    2555 generic.go:356] "Generic (PLEG): container finished" podID="eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" containerID="ea4fb7d9f27929664db7a48562105da8c65654d0c1510f6f41ceba0ed5efdfd3" exitCode=0
Sep 09 16:38:09.127381 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:09.126983    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p" event={"ID":"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a","Type":"ContainerDied","Data":"ea4fb7d9f27929664db7a48562105da8c65654d0c1510f6f41ceba0ed5efdfd3"}
Sep 09 16:38:09.127535 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:09.127515    2555 scope.go:117] "RemoveContainer" containerID="ea4fb7d9f27929664db7a48562105da8c65654d0c1510f6f41ceba0ed5efdfd3"
Sep 09 16:38:09.128586 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:09.128552    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:38:09.128689 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:09.128634    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:38:09.128689 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:09.128680    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 16:38:10.134543 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:10.134511    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p" event={"ID":"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a","Type":"ContainerStarted","Data":"bee8fe89f7d4a9e87955522a9eb1906774b710f7a73e02cc1920bf71043ed003"}
Sep 09 16:38:11.098314 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:11.097782    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:38:11.098314 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:11.097998    2555 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 09 16:38:11.098314 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:11.098055    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p" podUID="eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 09 16:38:11.807137 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:11.807104    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d"]
Sep 09 16:38:11.810787 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:11.810769    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d"
Sep 09 16:38:11.813251 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:11.813225    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-1d59dd-dockercfg-874xt\""
Sep 09 16:38:11.813374 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:11.813225    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-1d59dd\""
Sep 09 16:38:11.820068 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:11.820045    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d"]
Sep 09 16:38:11.989268 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:11.989231    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/258b7a1e-a5b1-4041-9e22-4d618e367bce-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d\" (UID: \"258b7a1e-a5b1-4041-9e22-4d618e367bce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d"
Sep 09 16:38:11.989481 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:11.989278    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwhrz\" (UniqueName: \"kubernetes.io/projected/258b7a1e-a5b1-4041-9e22-4d618e367bce-kube-api-access-qwhrz\") pod \"payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d\" (UID: \"258b7a1e-a5b1-4041-9e22-4d618e367bce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d"
Sep 09 16:38:12.090440 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:12.090357    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/258b7a1e-a5b1-4041-9e22-4d618e367bce-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d\" (UID: \"258b7a1e-a5b1-4041-9e22-4d618e367bce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d"
Sep 09 16:38:12.090440 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:12.090409    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwhrz\" (UniqueName: \"kubernetes.io/projected/258b7a1e-a5b1-4041-9e22-4d618e367bce-kube-api-access-qwhrz\") pod \"payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d\" (UID: \"258b7a1e-a5b1-4041-9e22-4d618e367bce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d"
Sep 09 16:38:12.091084 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:12.091064    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/258b7a1e-a5b1-4041-9e22-4d618e367bce-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d\" (UID: \"258b7a1e-a5b1-4041-9e22-4d618e367bce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d"
Sep 09 16:38:12.097273 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:12.097243    2555 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 09 16:38:12.097426 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:12.097316    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p" podUID="eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 09 16:38:12.098048 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:12.098027    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwhrz\" (UniqueName: \"kubernetes.io/projected/258b7a1e-a5b1-4041-9e22-4d618e367bce-kube-api-access-qwhrz\") pod \"payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d\" (UID: \"258b7a1e-a5b1-4041-9e22-4d618e367bce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d"
Sep 09 16:38:12.122636 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:12.122600    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d"
Sep 09 16:38:12.417876 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:12.417848    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d"]
Sep 09 16:38:13.098092 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:13.098051    2555 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 09 16:38:13.098560 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:13.098113    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p" podUID="eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 09 16:38:13.173074 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:13.173030    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d" event={"ID":"258b7a1e-a5b1-4041-9e22-4d618e367bce","Type":"ContainerStarted","Data":"20797f56bdd8d0060673559eac370b1774a0b25db5e80dbe351b8cec3010f5e8"}
Sep 09 16:38:13.173269 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:13.173102    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d" event={"ID":"258b7a1e-a5b1-4041-9e22-4d618e367bce","Type":"ContainerStarted","Data":"111e81722b62a3ce5d92ed37cdf0dbfd4717b9ac67bc56ecebc3961a01a08ff1"}
Sep 09 16:38:13.173391 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:13.173368    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d"
Sep 09 16:38:13.189008 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:13.188965    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d" podStartSLOduration=2.188951164 podStartE2EDuration="2.188951164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:38:13.188619125 +0000 UTC m=+1459.018827185" watchObservedRunningTime="2026-09-09 16:38:13.188951164 +0000 UTC m=+1459.019159601"
Sep 09 16:38:14.097327 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:14.097272    2555 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 09 16:38:14.097474 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:14.097345    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p" podUID="eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 09 16:38:15.132069 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:15.132026    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:38:15.132565 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:15.132382    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:38:15.133314 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:15.133280    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:38:19.129884 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:19.129850    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff"
Sep 09 16:38:19.130273 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:19.130010    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x"
Sep 09 16:38:23.088216 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:23.088180    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:38:24.183689 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:24.183660    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d"
Sep 09 16:38:25.299687 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:25.299652    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"]
Sep 09 16:38:25.300084 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:25.299856    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9" podUID="d23fd4a8-bbdd-4b1d-b182-42ea8c433626" containerName="istio-proxy" containerID="cri-o://b30e577809bc2e22ec9dcdd91e3661d175ac9c5652314cf5ae1a07bf90a28c43" gracePeriod=30
Sep 09 16:38:28.744551 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:28.744508    2555 patch_prober.go:28] interesting pod/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 16:38:28.745007 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:28.744586    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9" podUID="d23fd4a8-bbdd-4b1d-b182-42ea8c433626" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 16:38:30.505492 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.505462    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"]
Sep 09 16:38:30.581192 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.581165    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:38:30.665911 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.665879    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istiod-ca-cert\") pod \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") "
Sep 09 16:38:30.666057 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.665996    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istio-podinfo\") pod \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") "
Sep 09 16:38:30.666057 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.666051    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istio-data\") pod \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") "
Sep 09 16:38:30.666152 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.666090    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm6tn\" (UniqueName: \"kubernetes.io/projected/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-kube-api-access-lm6tn\") pod \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") "
Sep 09 16:38:30.666152 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.666127    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-workload-certs\") pod \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") "
Sep 09 16:38:30.666239 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.666160    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istio-token\") pod \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") "
Sep 09 16:38:30.666239 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.666193    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-credential-socket\") pod \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") "
Sep 09 16:38:30.666239 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.666232    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-workload-socket\") pod \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") "
Sep 09 16:38:30.666428 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.666225    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d23fd4a8-bbdd-4b1d-b182-42ea8c433626" (UID: "d23fd4a8-bbdd-4b1d-b182-42ea8c433626"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:38:30.666428 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.666256    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istio-envoy\") pod \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\" (UID: \"d23fd4a8-bbdd-4b1d-b182-42ea8c433626\") "
Sep 09 16:38:30.666540 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.666447    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istio-data" (OuterVolumeSpecName: "istio-data") pod "d23fd4a8-bbdd-4b1d-b182-42ea8c433626" (UID: "d23fd4a8-bbdd-4b1d-b182-42ea8c433626"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:38:30.666540 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.666481    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d23fd4a8-bbdd-4b1d-b182-42ea8c433626" (UID: "d23fd4a8-bbdd-4b1d-b182-42ea8c433626"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:38:30.666641 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.666558    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d23fd4a8-bbdd-4b1d-b182-42ea8c433626" (UID: "d23fd4a8-bbdd-4b1d-b182-42ea8c433626"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:38:30.666728 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.666706    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d23fd4a8-bbdd-4b1d-b182-42ea8c433626" (UID: "d23fd4a8-bbdd-4b1d-b182-42ea8c433626"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:38:30.666864 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.666733    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-credential-socket\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:38:30.666864 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.666784    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istiod-ca-cert\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:38:30.666864 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.666801    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istio-data\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:38:30.666864 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.666826    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-workload-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:38:30.668350 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.668317    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d23fd4a8-bbdd-4b1d-b182-42ea8c433626" (UID: "d23fd4a8-bbdd-4b1d-b182-42ea8c433626"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 16:38:30.668448 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.668436    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d23fd4a8-bbdd-4b1d-b182-42ea8c433626" (UID: "d23fd4a8-bbdd-4b1d-b182-42ea8c433626"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:38:30.668759 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.668725    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-kube-api-access-lm6tn" (OuterVolumeSpecName: "kube-api-access-lm6tn") pod "d23fd4a8-bbdd-4b1d-b182-42ea8c433626" (UID: "d23fd4a8-bbdd-4b1d-b182-42ea8c433626"). InnerVolumeSpecName "kube-api-access-lm6tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:38:30.668896 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.668749    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istio-token" (OuterVolumeSpecName: "istio-token") pod "d23fd4a8-bbdd-4b1d-b182-42ea8c433626" (UID: "d23fd4a8-bbdd-4b1d-b182-42ea8c433626"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:38:30.767372 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.767336    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istio-podinfo\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:38:30.767372 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.767365    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm6tn\" (UniqueName: \"kubernetes.io/projected/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-kube-api-access-lm6tn\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:38:30.767372 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.767374    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istio-token\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:38:30.767593 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.767384    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-workload-socket\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:38:30.767593 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:30.767393    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d23fd4a8-bbdd-4b1d-b182-42ea8c433626-istio-envoy\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:38:31.296423 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:31.296387    2555 generic.go:356] "Generic (PLEG): container finished" podID="d23fd4a8-bbdd-4b1d-b182-42ea8c433626" containerID="b30e577809bc2e22ec9dcdd91e3661d175ac9c5652314cf5ae1a07bf90a28c43" exitCode=0
Sep 09 16:38:31.296606 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:31.296446    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"
Sep 09 16:38:31.296606 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:31.296473    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9" event={"ID":"d23fd4a8-bbdd-4b1d-b182-42ea8c433626","Type":"ContainerDied","Data":"b30e577809bc2e22ec9dcdd91e3661d175ac9c5652314cf5ae1a07bf90a28c43"}
Sep 09 16:38:31.296606 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:31.296531    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9" event={"ID":"d23fd4a8-bbdd-4b1d-b182-42ea8c433626","Type":"ContainerDied","Data":"22a11160816edb438d782619d58b3fab705640f844d488292fe1ebf499a05262"}
Sep 09 16:38:31.296606 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:31.296553    2555 scope.go:117] "RemoveContainer" containerID="b30e577809bc2e22ec9dcdd91e3661d175ac9c5652314cf5ae1a07bf90a28c43"
Sep 09 16:38:31.306474 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:31.306459    2555 scope.go:117] "RemoveContainer" containerID="b30e577809bc2e22ec9dcdd91e3661d175ac9c5652314cf5ae1a07bf90a28c43"
Sep 09 16:38:31.306750 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:38:31.306731    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b30e577809bc2e22ec9dcdd91e3661d175ac9c5652314cf5ae1a07bf90a28c43\": container with ID starting with b30e577809bc2e22ec9dcdd91e3661d175ac9c5652314cf5ae1a07bf90a28c43 not found: ID does not exist" containerID="b30e577809bc2e22ec9dcdd91e3661d175ac9c5652314cf5ae1a07bf90a28c43"
Sep 09 16:38:31.306819 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:31.306763    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30e577809bc2e22ec9dcdd91e3661d175ac9c5652314cf5ae1a07bf90a28c43"} err="failed to get container status \"b30e577809bc2e22ec9dcdd91e3661d175ac9c5652314cf5ae1a07bf90a28c43\": rpc error: code = NotFound desc = could not find container \"b30e577809bc2e22ec9dcdd91e3661d175ac9c5652314cf5ae1a07bf90a28c43\": container with ID starting with b30e577809bc2e22ec9dcdd91e3661d175ac9c5652314cf5ae1a07bf90a28c43 not found: ID does not exist"
Sep 09 16:38:31.318943 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:31.318921    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"]
Sep 09 16:38:31.322756 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:31.322734    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-89d246-openshift-default-5c6957c978-hk6c9"]
Sep 09 16:38:32.608409 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:32.608377    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d23fd4a8-bbdd-4b1d-b182-42ea8c433626" path="/var/lib/kubelet/pods/d23fd4a8-bbdd-4b1d-b182-42ea8c433626/volumes"
Sep 09 16:38:34.470252 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:34.470200    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6954-140011489899696"]
Sep 09 16:38:34.470795 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:34.470774    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d23fd4a8-bbdd-4b1d-b182-42ea8c433626" containerName="istio-proxy"
Sep 09 16:38:34.470884 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:34.470798    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d23fd4a8-bbdd-4b1d-b182-42ea8c433626" containerName="istio-proxy"
Sep 09 16:38:34.470939 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:34.470908    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d23fd4a8-bbdd-4b1d-b182-42ea8c433626" containerName="istio-proxy"
Sep 09 16:38:34.475320 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:34.475279    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6954-140011489899696"
Sep 09 16:38:34.477752 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:34.477727    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6954-140011489899696"]
Sep 09 16:38:34.477895 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:34.477878    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-npvh9\""
Sep 09 16:38:34.602989 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:34.602955    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82gdp\" (UniqueName: \"kubernetes.io/projected/1dd84295-d947-4543-8817-2c6f162047ed-kube-api-access-82gdp\") pod \"test-curl-6954-140011489899696\" (UID: \"1dd84295-d947-4543-8817-2c6f162047ed\") " pod="opendatahub/test-curl-6954-140011489899696"
Sep 09 16:38:34.704376 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:34.704341    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82gdp\" (UniqueName: \"kubernetes.io/projected/1dd84295-d947-4543-8817-2c6f162047ed-kube-api-access-82gdp\") pod \"test-curl-6954-140011489899696\" (UID: \"1dd84295-d947-4543-8817-2c6f162047ed\") " pod="opendatahub/test-curl-6954-140011489899696"
Sep 09 16:38:34.712212 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:34.712187    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82gdp\" (UniqueName: \"kubernetes.io/projected/1dd84295-d947-4543-8817-2c6f162047ed-kube-api-access-82gdp\") pod \"test-curl-6954-140011489899696\" (UID: \"1dd84295-d947-4543-8817-2c6f162047ed\") " pod="opendatahub/test-curl-6954-140011489899696"
Sep 09 16:38:34.785210 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:34.785186    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6954-140011489899696"
Sep 09 16:38:35.063946 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:35.063919    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6954-140011489899696"]
Sep 09 16:38:35.065599 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:38:35.065567    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1dd84295_d947_4543_8817_2c6f162047ed.slice/crio-550d0e8d4e2e1f8e34ae23cf2e41ad0039d9db9b747270dd952b8e6a5d8ffcf1 WatchSource:0}: Error finding container 550d0e8d4e2e1f8e34ae23cf2e41ad0039d9db9b747270dd952b8e6a5d8ffcf1: Status 404 returned error can't find the container with id 550d0e8d4e2e1f8e34ae23cf2e41ad0039d9db9b747270dd952b8e6a5d8ffcf1
Sep 09 16:38:35.321368 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:35.321273    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6954-140011489899696" event={"ID":"1dd84295-d947-4543-8817-2c6f162047ed","Type":"ContainerStarted","Data":"550d0e8d4e2e1f8e34ae23cf2e41ad0039d9db9b747270dd952b8e6a5d8ffcf1"}
Sep 09 16:38:36.327991 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:36.327951    2555 generic.go:356] "Generic (PLEG): container finished" podID="1dd84295-d947-4543-8817-2c6f162047ed" containerID="0c17a7e4bc72afca1c4be017c858e9d9a9999edf819a2a5e8090c59abf3d4419" exitCode=0
Sep 09 16:38:36.328396 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:36.328009    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6954-140011489899696" event={"ID":"1dd84295-d947-4543-8817-2c6f162047ed","Type":"ContainerDied","Data":"0c17a7e4bc72afca1c4be017c858e9d9a9999edf819a2a5e8090c59abf3d4419"}
Sep 09 16:38:36.795211 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:36.795175    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"]
Sep 09 16:38:36.798838 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:36.798818    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:38:36.801321 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:36.801285    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-331bf8\"/\"openshift-service-ca.crt\""
Sep 09 16:38:36.802631 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:36.802612    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-331bf8\"/\"default-dockercfg-9jrw7\""
Sep 09 16:38:36.802858 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:36.802836    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-331bf8\"/\"kube-root-ca.crt\""
Sep 09 16:38:36.803165 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:36.803138    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-331bf8\"/\"auth-test-model-331bf8-kserve-self-signed-certs\""
Sep 09 16:38:36.806415 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:36.806392    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"]
Sep 09 16:38:36.924246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:36.924206    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-dshm\") pod \"auth-test-model-331bf8-kserve-6b49f79659-74jdc\" (UID: \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:38:36.924465 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:36.924254    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85t2j\" (UniqueName: \"kubernetes.io/projected/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-kube-api-access-85t2j\") pod \"auth-test-model-331bf8-kserve-6b49f79659-74jdc\" (UID: \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:38:36.924465 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:36.924288    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-model-cache\") pod \"auth-test-model-331bf8-kserve-6b49f79659-74jdc\" (UID: \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:38:36.924465 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:36.924379    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-tls-certs\") pod \"auth-test-model-331bf8-kserve-6b49f79659-74jdc\" (UID: \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:38:36.924465 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:36.924448    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-home\") pod \"auth-test-model-331bf8-kserve-6b49f79659-74jdc\" (UID: \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:38:36.924612 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:36.924476    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-tmp-dir\") pod \"auth-test-model-331bf8-kserve-6b49f79659-74jdc\" (UID: \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:38:37.025639 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:37.025596    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85t2j\" (UniqueName: \"kubernetes.io/projected/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-kube-api-access-85t2j\") pod \"auth-test-model-331bf8-kserve-6b49f79659-74jdc\" (UID: \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:38:37.025639 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:37.025645    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-model-cache\") pod \"auth-test-model-331bf8-kserve-6b49f79659-74jdc\" (UID: \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:38:37.025919 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:37.025682    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-tls-certs\") pod \"auth-test-model-331bf8-kserve-6b49f79659-74jdc\" (UID: \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:38:37.025919 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:37.025707    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-home\") pod \"auth-test-model-331bf8-kserve-6b49f79659-74jdc\" (UID: \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:38:37.025919 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:37.025723    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-tmp-dir\") pod \"auth-test-model-331bf8-kserve-6b49f79659-74jdc\" (UID: \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:38:37.025919 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:37.025784    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-dshm\") pod \"auth-test-model-331bf8-kserve-6b49f79659-74jdc\" (UID: \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:38:37.026154 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:37.026131    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-model-cache\") pod \"auth-test-model-331bf8-kserve-6b49f79659-74jdc\" (UID: \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:38:37.026244 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:37.026223    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-tmp-dir\") pod \"auth-test-model-331bf8-kserve-6b49f79659-74jdc\" (UID: \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:38:37.026244 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:37.026237    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-home\") pod \"auth-test-model-331bf8-kserve-6b49f79659-74jdc\" (UID: \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:38:37.027977 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:37.027958    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-dshm\") pod \"auth-test-model-331bf8-kserve-6b49f79659-74jdc\" (UID: \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:38:37.028237 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:37.028219    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-tls-certs\") pod \"auth-test-model-331bf8-kserve-6b49f79659-74jdc\" (UID: \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:38:37.033186 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:37.033162    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85t2j\" (UniqueName: \"kubernetes.io/projected/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-kube-api-access-85t2j\") pod \"auth-test-model-331bf8-kserve-6b49f79659-74jdc\" (UID: \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:38:37.110474 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:37.110393    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:38:37.381090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:37.381061    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"]
Sep 09 16:38:37.382283 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:38:37.382257    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd06cfd5b_3d5b_4d1b_8f8d_f69be17d30c7.slice/crio-cb586121af2bc10165b1fee31e51a1cc184cebdc30813cd66bd0a7629a46d65e WatchSource:0}: Error finding container cb586121af2bc10165b1fee31e51a1cc184cebdc30813cd66bd0a7629a46d65e: Status 404 returned error can't find the container with id cb586121af2bc10165b1fee31e51a1cc184cebdc30813cd66bd0a7629a46d65e
Sep 09 16:38:37.462193 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:37.462167    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6954-140011489899696"
Sep 09 16:38:37.494211 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:37.493659    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-6954-140011489899696_1dd84295-d947-4543-8817-2c6f162047ed/test-curl-6954-140011489899696/0.log"
Sep 09 16:38:37.515570 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:37.515540    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6954-140011489899696"]
Sep 09 16:38:37.518084 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:37.518060    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6954-140011489899696"]
Sep 09 16:38:37.530644 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:37.530624    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82gdp\" (UniqueName: \"kubernetes.io/projected/1dd84295-d947-4543-8817-2c6f162047ed-kube-api-access-82gdp\") pod \"1dd84295-d947-4543-8817-2c6f162047ed\" (UID: \"1dd84295-d947-4543-8817-2c6f162047ed\") "
Sep 09 16:38:37.532663 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:37.532635    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd84295-d947-4543-8817-2c6f162047ed-kube-api-access-82gdp" (OuterVolumeSpecName: "kube-api-access-82gdp") pod "1dd84295-d947-4543-8817-2c6f162047ed" (UID: "1dd84295-d947-4543-8817-2c6f162047ed"). InnerVolumeSpecName "kube-api-access-82gdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:38:37.631627 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:37.631551    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82gdp\" (UniqueName: \"kubernetes.io/projected/1dd84295-d947-4543-8817-2c6f162047ed-kube-api-access-82gdp\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:38:38.341256 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:38.341131    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc" event={"ID":"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7","Type":"ContainerStarted","Data":"9fd8f287dc8353b1208b45ec81cb338341b128923bab9d571b427ea646d24158"}
Sep 09 16:38:38.341256 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:38.341183    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc" event={"ID":"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7","Type":"ContainerStarted","Data":"cb586121af2bc10165b1fee31e51a1cc184cebdc30813cd66bd0a7629a46d65e"}
Sep 09 16:38:38.342735 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:38.342715    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="550d0e8d4e2e1f8e34ae23cf2e41ad0039d9db9b747270dd952b8e6a5d8ffcf1"
Sep 09 16:38:38.342854 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:38.342737    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6954-140011489899696"
Sep 09 16:38:38.357750 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:38.357708    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc" podStartSLOduration=2.357693331 podStartE2EDuration="2.357693331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:38:38.356094634 +0000 UTC m=+1484.186302693" watchObservedRunningTime="2026-09-09 16:38:38.357693331 +0000 UTC m=+1484.187901390"
Sep 09 16:38:38.358307 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:38:38.358272    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6954-140011489899696\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="1dd84295-d947-4543-8817-2c6f162047ed" pod="opendatahub/test-curl-6954-140011489899696"
Sep 09 16:38:38.608541 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:38.608463    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dd84295-d947-4543-8817-2c6f162047ed" path="/var/lib/kubelet/pods/1dd84295-d947-4543-8817-2c6f162047ed/volumes"
Sep 09 16:38:41.099891 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:41.099856    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:38:44.407644 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:44.407609    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6954-140011489900272"]
Sep 09 16:38:44.408008 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:44.407997    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dd84295-d947-4543-8817-2c6f162047ed" containerName="test-curl-6954-140011489899696"
Sep 09 16:38:44.408008 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:44.408009    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dd84295-d947-4543-8817-2c6f162047ed" containerName="test-curl-6954-140011489899696"
Sep 09 16:38:44.408085 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:44.408079    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dd84295-d947-4543-8817-2c6f162047ed" containerName="test-curl-6954-140011489899696"
Sep 09 16:38:44.412682 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:44.412655    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6954-140011489900272"
Sep 09 16:38:44.414899 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:44.414878    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6954-140011489900272"]
Sep 09 16:38:44.415179 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:44.415154    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-npvh9\""
Sep 09 16:38:44.496935 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:44.496903    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjd4z\" (UniqueName: \"kubernetes.io/projected/4614ac8a-fb05-42dd-b974-caded7f1c235-kube-api-access-tjd4z\") pod \"test-curl-6954-140011489900272\" (UID: \"4614ac8a-fb05-42dd-b974-caded7f1c235\") " pod="opendatahub/test-curl-6954-140011489900272"
Sep 09 16:38:44.598507 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:44.598464    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjd4z\" (UniqueName: \"kubernetes.io/projected/4614ac8a-fb05-42dd-b974-caded7f1c235-kube-api-access-tjd4z\") pod \"test-curl-6954-140011489900272\" (UID: \"4614ac8a-fb05-42dd-b974-caded7f1c235\") " pod="opendatahub/test-curl-6954-140011489900272"
Sep 09 16:38:44.606562 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:44.606517    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjd4z\" (UniqueName: \"kubernetes.io/projected/4614ac8a-fb05-42dd-b974-caded7f1c235-kube-api-access-tjd4z\") pod \"test-curl-6954-140011489900272\" (UID: \"4614ac8a-fb05-42dd-b974-caded7f1c235\") " pod="opendatahub/test-curl-6954-140011489900272"
Sep 09 16:38:44.723889 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:44.723811    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6954-140011489900272"
Sep 09 16:38:44.996562 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:44.996536    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6954-140011489900272"]
Sep 09 16:38:44.998044 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:38:44.998018    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4614ac8a_fb05_42dd_b974_caded7f1c235.slice/crio-e557fd8008731fdf8b4e0a9eac445e27fba762723af620ad80003a4b131f7d43 WatchSource:0}: Error finding container e557fd8008731fdf8b4e0a9eac445e27fba762723af620ad80003a4b131f7d43: Status 404 returned error can't find the container with id e557fd8008731fdf8b4e0a9eac445e27fba762723af620ad80003a4b131f7d43
Sep 09 16:38:45.384655 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:45.384563    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6954-140011489900272" event={"ID":"4614ac8a-fb05-42dd-b974-caded7f1c235","Type":"ContainerStarted","Data":"e557fd8008731fdf8b4e0a9eac445e27fba762723af620ad80003a4b131f7d43"}
Sep 09 16:38:45.531228 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:45.531187    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8" podUID="6b6f2f29-47de-463b-a23c-ff71b5b74b4d" containerName="main" containerID="cri-o://fbe14009805d6453ce6ff1a1f25e7f8a58409f4c3676cb587aab34f4dcabc161" gracePeriod=45
Sep 09 16:38:45.795584 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:45.795554    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:38:45.910334 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:45.910254    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-model-cache\") pod \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\" (UID: \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\") "
Sep 09 16:38:45.910485 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:45.910358    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp8ld\" (UniqueName: \"kubernetes.io/projected/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-kube-api-access-mp8ld\") pod \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\" (UID: \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\") "
Sep 09 16:38:45.910485 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:45.910392    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-tls-certs\") pod \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\" (UID: \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\") "
Sep 09 16:38:45.910485 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:45.910431    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-home\") pod \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\" (UID: \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\") "
Sep 09 16:38:45.910485 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:45.910466    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b6f2f29-47de-463b-a23c-ff71b5b74b4d" (UID: "6b6f2f29-47de-463b-a23c-ff71b5b74b4d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:38:45.910721 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:45.910496    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-dshm\") pod \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\" (UID: \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\") "
Sep 09 16:38:45.910721 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:45.910543    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-tmp-dir\") pod \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\" (UID: \"6b6f2f29-47de-463b-a23c-ff71b5b74b4d\") "
Sep 09 16:38:45.910721 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:45.910610    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-home" (OuterVolumeSpecName: "home") pod "6b6f2f29-47de-463b-a23c-ff71b5b74b4d" (UID: "6b6f2f29-47de-463b-a23c-ff71b5b74b4d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:38:45.910884 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:45.910815    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6b6f2f29-47de-463b-a23c-ff71b5b74b4d" (UID: "6b6f2f29-47de-463b-a23c-ff71b5b74b4d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:38:45.910884 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:45.910830    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-model-cache\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:38:45.910884 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:45.910847    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-home\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:38:45.912665 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:45.912639    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b6f2f29-47de-463b-a23c-ff71b5b74b4d" (UID: "6b6f2f29-47de-463b-a23c-ff71b5b74b4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:38:45.912665 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:45.912652    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-dshm" (OuterVolumeSpecName: "dshm") pod "6b6f2f29-47de-463b-a23c-ff71b5b74b4d" (UID: "6b6f2f29-47de-463b-a23c-ff71b5b74b4d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:38:45.912665 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:45.912655    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-kube-api-access-mp8ld" (OuterVolumeSpecName: "kube-api-access-mp8ld") pod "6b6f2f29-47de-463b-a23c-ff71b5b74b4d" (UID: "6b6f2f29-47de-463b-a23c-ff71b5b74b4d"). InnerVolumeSpecName "kube-api-access-mp8ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:38:46.011514 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:46.011485    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-tmp-dir\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:38:46.011514 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:46.011515    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp8ld\" (UniqueName: \"kubernetes.io/projected/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-kube-api-access-mp8ld\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:38:46.011658 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:46.011525    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-tls-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:38:46.011658 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:46.011534    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b6f2f29-47de-463b-a23c-ff71b5b74b4d-dshm\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:38:46.391750 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:46.391712    2555 generic.go:356] "Generic (PLEG): container finished" podID="4614ac8a-fb05-42dd-b974-caded7f1c235" containerID="8d180a9e961218e4aca67fea8e869920332578e2f68e2df985b3e4eadc484175" exitCode=0
Sep 09 16:38:46.391972 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:46.391797    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6954-140011489900272" event={"ID":"4614ac8a-fb05-42dd-b974-caded7f1c235","Type":"ContainerDied","Data":"8d180a9e961218e4aca67fea8e869920332578e2f68e2df985b3e4eadc484175"}
Sep 09 16:38:46.393204 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:46.393180    2555 generic.go:356] "Generic (PLEG): container finished" podID="6b6f2f29-47de-463b-a23c-ff71b5b74b4d" containerID="fbe14009805d6453ce6ff1a1f25e7f8a58409f4c3676cb587aab34f4dcabc161" exitCode=0
Sep 09 16:38:46.393338 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:46.393250    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"
Sep 09 16:38:46.393338 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:46.393253    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8" event={"ID":"6b6f2f29-47de-463b-a23c-ff71b5b74b4d","Type":"ContainerDied","Data":"fbe14009805d6453ce6ff1a1f25e7f8a58409f4c3676cb587aab34f4dcabc161"}
Sep 09 16:38:46.393338 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:46.393328    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8" event={"ID":"6b6f2f29-47de-463b-a23c-ff71b5b74b4d","Type":"ContainerDied","Data":"b6f1b2b4ce3cd09b5b8613577a10641c433d388710f719fae1abb51b342ce19f"}
Sep 09 16:38:46.393499 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:46.393345    2555 scope.go:117] "RemoveContainer" containerID="fbe14009805d6453ce6ff1a1f25e7f8a58409f4c3676cb587aab34f4dcabc161"
Sep 09 16:38:46.402646 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:46.402629    2555 scope.go:117] "RemoveContainer" containerID="fbe14009805d6453ce6ff1a1f25e7f8a58409f4c3676cb587aab34f4dcabc161"
Sep 09 16:38:46.402921 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:38:46.402901    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbe14009805d6453ce6ff1a1f25e7f8a58409f4c3676cb587aab34f4dcabc161\": container with ID starting with fbe14009805d6453ce6ff1a1f25e7f8a58409f4c3676cb587aab34f4dcabc161 not found: ID does not exist" containerID="fbe14009805d6453ce6ff1a1f25e7f8a58409f4c3676cb587aab34f4dcabc161"
Sep 09 16:38:46.402973 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:46.402930    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbe14009805d6453ce6ff1a1f25e7f8a58409f4c3676cb587aab34f4dcabc161"} err="failed to get container status \"fbe14009805d6453ce6ff1a1f25e7f8a58409f4c3676cb587aab34f4dcabc161\": rpc error: code = NotFound desc = could not find container \"fbe14009805d6453ce6ff1a1f25e7f8a58409f4c3676cb587aab34f4dcabc161\": container with ID starting with fbe14009805d6453ce6ff1a1f25e7f8a58409f4c3676cb587aab34f4dcabc161 not found: ID does not exist"
Sep 09 16:38:46.418804 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:46.418780    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"]
Sep 09 16:38:46.422819 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:46.422798    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-89d246/noten-model-89d246-kserve-9f9c9756c-2rpv8"]
Sep 09 16:38:46.608377 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:46.608342    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b6f2f29-47de-463b-a23c-ff71b5b74b4d" path="/var/lib/kubelet/pods/6b6f2f29-47de-463b-a23c-ff71b5b74b4d/volumes"
Sep 09 16:38:47.111355 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:47.111251    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:38:47.111355 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:47.111314    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:38:47.124077 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:47.124053    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:38:47.415022 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:47.414997    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:38:47.555586 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:47.555563    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6954-140011489900272"
Sep 09 16:38:47.574483 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:47.574416    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-6954-140011489900272_4614ac8a-fb05-42dd-b974-caded7f1c235/test-curl-6954-140011489900272/0.log"
Sep 09 16:38:47.601361 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:47.601328    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6954-140011489900272"]
Sep 09 16:38:47.606303 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:47.606268    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6954-140011489900272"]
Sep 09 16:38:47.630391 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:47.630369    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjd4z\" (UniqueName: \"kubernetes.io/projected/4614ac8a-fb05-42dd-b974-caded7f1c235-kube-api-access-tjd4z\") pod \"4614ac8a-fb05-42dd-b974-caded7f1c235\" (UID: \"4614ac8a-fb05-42dd-b974-caded7f1c235\") "
Sep 09 16:38:47.632406 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:47.632376    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4614ac8a-fb05-42dd-b974-caded7f1c235-kube-api-access-tjd4z" (OuterVolumeSpecName: "kube-api-access-tjd4z") pod "4614ac8a-fb05-42dd-b974-caded7f1c235" (UID: "4614ac8a-fb05-42dd-b974-caded7f1c235"). InnerVolumeSpecName "kube-api-access-tjd4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:38:47.731761 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:47.731685    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjd4z\" (UniqueName: \"kubernetes.io/projected/4614ac8a-fb05-42dd-b974-caded7f1c235-kube-api-access-tjd4z\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:38:48.410712 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:48.410683    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6954-140011489900272"
Sep 09 16:38:48.410916 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:48.410727    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e557fd8008731fdf8b4e0a9eac445e27fba762723af620ad80003a4b131f7d43"
Sep 09 16:38:48.420744 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:38:48.420710    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6954-140011489900272\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="4614ac8a-fb05-42dd-b974-caded7f1c235" pod="opendatahub/test-curl-6954-140011489900272"
Sep 09 16:38:48.611005 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:48.610972    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4614ac8a-fb05-42dd-b974-caded7f1c235" path="/var/lib/kubelet/pods/4614ac8a-fb05-42dd-b974-caded7f1c235/volumes"
Sep 09 16:38:54.675120 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:54.675091    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgcjl_561d9dd7-04c3-42dc-bcf6-2c92fd61706b/console-operator/1.log"
Sep 09 16:38:54.679806 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:54.679783    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgcjl_561d9dd7-04c3-42dc-bcf6-2c92fd61706b/console-operator/1.log"
Sep 09 16:38:54.744321 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:54.742191    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9"]
Sep 09 16:38:54.744321 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:54.742873    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4614ac8a-fb05-42dd-b974-caded7f1c235" containerName="test-curl-6954-140011489900272"
Sep 09 16:38:54.744321 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:54.742891    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4614ac8a-fb05-42dd-b974-caded7f1c235" containerName="test-curl-6954-140011489900272"
Sep 09 16:38:54.744321 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:54.743040    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b6f2f29-47de-463b-a23c-ff71b5b74b4d" containerName="main"
Sep 09 16:38:54.744321 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:54.743051    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b6f2f29-47de-463b-a23c-ff71b5b74b4d" containerName="main"
Sep 09 16:38:54.744321 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:54.743149    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4614ac8a-fb05-42dd-b974-caded7f1c235" containerName="test-curl-6954-140011489900272"
Sep 09 16:38:54.744321 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:54.743164    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b6f2f29-47de-463b-a23c-ff71b5b74b4d" containerName="main"
Sep 09 16:38:54.746946 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:54.746927    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9"
Sep 09 16:38:54.752098 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:54.750473    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-22795bd0-serving-cert\""
Sep 09 16:38:54.756916 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:54.756890    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9"]
Sep 09 16:38:54.900285 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:54.900247    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d6a925-baf0-472f-8813-75e11a1ba8ce-openshift-service-ca\") pod \"maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9\" (UID: \"e3d6a925-baf0-472f-8813-75e11a1ba8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9"
Sep 09 16:38:54.900285 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:54.900288    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3d6a925-baf0-472f-8813-75e11a1ba8ce-maas-api-tls\") pod \"maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9\" (UID: \"e3d6a925-baf0-472f-8813-75e11a1ba8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9"
Sep 09 16:38:54.900581 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:54.900479    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3d6a925-baf0-472f-8813-75e11a1ba8ce-metrics-certs\") pod \"maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9\" (UID: \"e3d6a925-baf0-472f-8813-75e11a1ba8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9"
Sep 09 16:38:54.900581 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:54.900519    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sff5j\" (UniqueName: \"kubernetes.io/projected/e3d6a925-baf0-472f-8813-75e11a1ba8ce-kube-api-access-sff5j\") pod \"maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9\" (UID: \"e3d6a925-baf0-472f-8813-75e11a1ba8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9"
Sep 09 16:38:54.939324 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:54.939208    2555 scope.go:117] "RemoveContainer" containerID="bc0088711b23a35d21997d9436a0ae4d59547447966e6e6edd8f7f7c834ca97a"
Sep 09 16:38:55.003410 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.003383    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d6a925-baf0-472f-8813-75e11a1ba8ce-openshift-service-ca\") pod \"maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9\" (UID: \"e3d6a925-baf0-472f-8813-75e11a1ba8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9"
Sep 09 16:38:55.003526 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.003428    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3d6a925-baf0-472f-8813-75e11a1ba8ce-maas-api-tls\") pod \"maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9\" (UID: \"e3d6a925-baf0-472f-8813-75e11a1ba8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9"
Sep 09 16:38:55.003607 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.003589    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3d6a925-baf0-472f-8813-75e11a1ba8ce-metrics-certs\") pod \"maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9\" (UID: \"e3d6a925-baf0-472f-8813-75e11a1ba8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9"
Sep 09 16:38:55.003691 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.003624    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sff5j\" (UniqueName: \"kubernetes.io/projected/e3d6a925-baf0-472f-8813-75e11a1ba8ce-kube-api-access-sff5j\") pod \"maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9\" (UID: \"e3d6a925-baf0-472f-8813-75e11a1ba8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9"
Sep 09 16:38:55.004150 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.004116    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d6a925-baf0-472f-8813-75e11a1ba8ce-openshift-service-ca\") pod \"maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9\" (UID: \"e3d6a925-baf0-472f-8813-75e11a1ba8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9"
Sep 09 16:38:55.006353 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.006326    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3d6a925-baf0-472f-8813-75e11a1ba8ce-metrics-certs\") pod \"maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9\" (UID: \"e3d6a925-baf0-472f-8813-75e11a1ba8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9"
Sep 09 16:38:55.006464 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.006448    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3d6a925-baf0-472f-8813-75e11a1ba8ce-maas-api-tls\") pod \"maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9\" (UID: \"e3d6a925-baf0-472f-8813-75e11a1ba8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9"
Sep 09 16:38:55.011925 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.011902    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sff5j\" (UniqueName: \"kubernetes.io/projected/e3d6a925-baf0-472f-8813-75e11a1ba8ce-kube-api-access-sff5j\") pod \"maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9\" (UID: \"e3d6a925-baf0-472f-8813-75e11a1ba8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9"
Sep 09 16:38:55.060761 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.060736    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9"
Sep 09 16:38:55.332127 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.332097    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9"]
Sep 09 16:38:55.334264 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:38:55.334235    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d6a925_baf0_472f_8813_75e11a1ba8ce.slice/crio-bc8807c88c70aa6cf8275f007965c149ae6027d72c2a9bb19e9361ad3babb09c WatchSource:0}: Error finding container bc8807c88c70aa6cf8275f007965c149ae6027d72c2a9bb19e9361ad3babb09c: Status 404 returned error can't find the container with id bc8807c88c70aa6cf8275f007965c149ae6027d72c2a9bb19e9361ad3babb09c
Sep 09 16:38:55.393171 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.393138    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9"]
Sep 09 16:38:55.396770 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.396753    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9"
Sep 09 16:38:55.399211 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.399189    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-22795bd0-dockercfg-69654\""
Sep 09 16:38:55.399442 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.399428    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-22795bd0\""
Sep 09 16:38:55.406594 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.406573    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9"]
Sep 09 16:38:55.430772 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.430741    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq"]
Sep 09 16:38:55.434372 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.434355    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq"
Sep 09 16:38:55.447562 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.447524    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq"]
Sep 09 16:38:55.456682 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.456655    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9" event={"ID":"e3d6a925-baf0-472f-8813-75e11a1ba8ce","Type":"ContainerStarted","Data":"bc8807c88c70aa6cf8275f007965c149ae6027d72c2a9bb19e9361ad3babb09c"}
Sep 09 16:38:55.508533 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.508496    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a996d66e-b83e-4508-a226-cb72a8fb220a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9\" (UID: \"a996d66e-b83e-4508-a226-cb72a8fb220a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9"
Sep 09 16:38:55.508726 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.508538    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fszw6\" (UniqueName: \"kubernetes.io/projected/a996d66e-b83e-4508-a226-cb72a8fb220a-kube-api-access-fszw6\") pod \"payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9\" (UID: \"a996d66e-b83e-4508-a226-cb72a8fb220a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9"
Sep 09 16:38:55.610022 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.609943    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a996d66e-b83e-4508-a226-cb72a8fb220a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9\" (UID: \"a996d66e-b83e-4508-a226-cb72a8fb220a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9"
Sep 09 16:38:55.610022 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.609980    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fszw6\" (UniqueName: \"kubernetes.io/projected/a996d66e-b83e-4508-a226-cb72a8fb220a-kube-api-access-fszw6\") pod \"payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9\" (UID: \"a996d66e-b83e-4508-a226-cb72a8fb220a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9"
Sep 09 16:38:55.610022 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.610011    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb1331f-f219-4027-bdce-4d93033ea5ec-plugins-config-volume\") pod \"payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq\" (UID: \"1eb1331f-f219-4027-bdce-4d93033ea5ec\") " pod="openshift-ingress/payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq"
Sep 09 16:38:55.610229 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.610056    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvqcv\" (UniqueName: \"kubernetes.io/projected/1eb1331f-f219-4027-bdce-4d93033ea5ec-kube-api-access-kvqcv\") pod \"payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq\" (UID: \"1eb1331f-f219-4027-bdce-4d93033ea5ec\") " pod="openshift-ingress/payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq"
Sep 09 16:38:55.610791 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.610769    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a996d66e-b83e-4508-a226-cb72a8fb220a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9\" (UID: \"a996d66e-b83e-4508-a226-cb72a8fb220a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9"
Sep 09 16:38:55.617484 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.617462    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fszw6\" (UniqueName: \"kubernetes.io/projected/a996d66e-b83e-4508-a226-cb72a8fb220a-kube-api-access-fszw6\") pod \"payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9\" (UID: \"a996d66e-b83e-4508-a226-cb72a8fb220a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9"
Sep 09 16:38:55.710049 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.710008    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9"
Sep 09 16:38:55.711416 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.711379    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvqcv\" (UniqueName: \"kubernetes.io/projected/1eb1331f-f219-4027-bdce-4d93033ea5ec-kube-api-access-kvqcv\") pod \"payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq\" (UID: \"1eb1331f-f219-4027-bdce-4d93033ea5ec\") " pod="openshift-ingress/payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq"
Sep 09 16:38:55.711528 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.711507    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb1331f-f219-4027-bdce-4d93033ea5ec-plugins-config-volume\") pod \"payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq\" (UID: \"1eb1331f-f219-4027-bdce-4d93033ea5ec\") " pod="openshift-ingress/payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq"
Sep 09 16:38:55.712165 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.712147    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb1331f-f219-4027-bdce-4d93033ea5ec-plugins-config-volume\") pod \"payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq\" (UID: \"1eb1331f-f219-4027-bdce-4d93033ea5ec\") " pod="openshift-ingress/payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq"
Sep 09 16:38:55.719101 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.719078    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvqcv\" (UniqueName: \"kubernetes.io/projected/1eb1331f-f219-4027-bdce-4d93033ea5ec-kube-api-access-kvqcv\") pod \"payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq\" (UID: \"1eb1331f-f219-4027-bdce-4d93033ea5ec\") " pod="openshift-ingress/payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq"
Sep 09 16:38:55.748253 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.748220    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq"
Sep 09 16:38:55.981688 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:55.981659    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9"]
Sep 09 16:38:55.982725 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:38:55.982702    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda996d66e_b83e_4508_a226_cb72a8fb220a.slice/crio-286e0ffbea13928c2ec242a28ed9d3028c0f80a5051f33fad5776bc1297b9971 WatchSource:0}: Error finding container 286e0ffbea13928c2ec242a28ed9d3028c0f80a5051f33fad5776bc1297b9971: Status 404 returned error can't find the container with id 286e0ffbea13928c2ec242a28ed9d3028c0f80a5051f33fad5776bc1297b9971
Sep 09 16:38:56.031132 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:56.031103    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq"]
Sep 09 16:38:56.465510 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:56.465469    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9" event={"ID":"e3d6a925-baf0-472f-8813-75e11a1ba8ce","Type":"ContainerStarted","Data":"286936415cdfa486061059c9905f5f39623bba6137fc046c5caa096577fc256e"}
Sep 09 16:38:56.465712 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:56.465589    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9"
Sep 09 16:38:56.467183 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:56.467159    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9" event={"ID":"a996d66e-b83e-4508-a226-cb72a8fb220a","Type":"ContainerStarted","Data":"00bac8083b28edf2bbed2b3097a4c8624f76d6c62c2ce85c6645de3983015210"}
Sep 09 16:38:56.467325 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:56.467217    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9" event={"ID":"a996d66e-b83e-4508-a226-cb72a8fb220a","Type":"ContainerStarted","Data":"286e0ffbea13928c2ec242a28ed9d3028c0f80a5051f33fad5776bc1297b9971"}
Sep 09 16:38:56.467325 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:56.467247    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9"
Sep 09 16:38:56.469078 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:56.469051    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq" event={"ID":"1eb1331f-f219-4027-bdce-4d93033ea5ec","Type":"ContainerStarted","Data":"ea86a33de03a00feb1bc73bfdc7566eea72f70160994d6693c3928cd14669b16"}
Sep 09 16:38:56.469184 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:56.469096    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq" event={"ID":"1eb1331f-f219-4027-bdce-4d93033ea5ec","Type":"ContainerStarted","Data":"c46599e5059c5c73eb722427b8fb1a8a59931a9951b199c8ccb9bd1cb433c991"}
Sep 09 16:38:56.469184 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:56.469124    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq"
Sep 09 16:38:56.481975 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:56.481940    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9" podStartSLOduration=2.005327192 podStartE2EDuration="2.481928009s" totalImagesPullingTime="476.600817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:38:56.480686368 +0000 UTC m=+1502.310894429" watchObservedRunningTime="2026-09-09 16:38:56.481928009 +0000 UTC m=+1502.312136067"
Sep 09 16:38:56.496059 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:56.496020    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq" podStartSLOduration=1.496007408 podStartE2EDuration="1.496007408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:38:56.495027245 +0000 UTC m=+1502.325235326" watchObservedRunningTime="2026-09-09 16:38:56.496007408 +0000 UTC m=+1502.326215493"
Sep 09 16:38:56.509854 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:38:56.509813    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9" podStartSLOduration=1.509802082 podStartE2EDuration="1.509802082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:38:56.508772441 +0000 UTC m=+1502.338980505" watchObservedRunningTime="2026-09-09 16:38:56.509802082 +0000 UTC m=+1502.340010173"
Sep 09 16:39:02.097818 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:02.097773    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:39:02.481172 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:02.481092    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9"
Sep 09 16:39:07.477916 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:07.477876    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9"
Sep 09 16:39:07.478375 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:07.477949    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq"
Sep 09 16:39:18.103989 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:18.103945    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:39:31.892612 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:31.892580    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:39:36.896880 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:36.896840    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"]
Sep 09 16:39:36.901399 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:36.901370    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:39:36.904821 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:36.904098    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-22795bd0\"/\"kube-root-ca.crt\""
Sep 09 16:39:36.904821 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:36.904487    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-22795bd0\"/\"openshift-service-ca.crt\""
Sep 09 16:39:36.905646 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:36.905627    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-22795bd0\"/\"e2e-lifecycle-model-22795bd0-kserve-self-signed-certs\""
Sep 09 16:39:36.906670 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:36.906648    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-22795bd0\"/\"default-dockercfg-7w5xf\""
Sep 09 16:39:36.907083 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:36.907065    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"]
Sep 09 16:39:37.006530 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:37.006486    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-home\") pod \"e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp\" (UID: \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\") " pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:39:37.006710 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:37.006575    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-tmp-dir\") pod \"e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp\" (UID: \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\") " pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:39:37.006710 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:37.006623    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-tls-certs\") pod \"e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp\" (UID: \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\") " pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:39:37.006710 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:37.006640    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mzgw\" (UniqueName: \"kubernetes.io/projected/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-kube-api-access-4mzgw\") pod \"e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp\" (UID: \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\") " pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:39:37.006836 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:37.006726    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-model-cache\") pod \"e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp\" (UID: \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\") " pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:39:37.006836 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:37.006802    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-dshm\") pod \"e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp\" (UID: \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\") " pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:39:37.107696 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:37.107650    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-model-cache\") pod \"e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp\" (UID: \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\") " pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:39:37.107861 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:37.107710    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-dshm\") pod \"e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp\" (UID: \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\") " pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:39:37.107861 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:37.107755    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-home\") pod \"e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp\" (UID: \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\") " pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:39:37.107861 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:37.107817    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-tmp-dir\") pod \"e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp\" (UID: \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\") " pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:39:37.108034 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:37.107859    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-tls-certs\") pod \"e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp\" (UID: \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\") " pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:39:37.108034 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:37.107886    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mzgw\" (UniqueName: \"kubernetes.io/projected/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-kube-api-access-4mzgw\") pod \"e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp\" (UID: \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\") " pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:39:37.108155 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:37.108109    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-model-cache\") pod \"e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp\" (UID: \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\") " pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:39:37.108215 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:37.108195    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-tmp-dir\") pod \"e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp\" (UID: \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\") " pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:39:37.108265 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:37.108241    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-home\") pod \"e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp\" (UID: \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\") " pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:39:37.110001 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:37.109974    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-dshm\") pod \"e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp\" (UID: \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\") " pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:39:37.110375 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:37.110356    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-tls-certs\") pod \"e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp\" (UID: \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\") " pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:39:37.115545 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:37.115520    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mzgw\" (UniqueName: \"kubernetes.io/projected/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-kube-api-access-4mzgw\") pod \"e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp\" (UID: \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\") " pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:39:37.214007 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:37.213921    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:39:37.488159 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:37.488118    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"]
Sep 09 16:39:37.736187 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:37.736083    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp" event={"ID":"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0","Type":"ContainerStarted","Data":"0f2bb6ec8e26cc6d5fef975049da6afa15ebff45c91622957864a4b40b7aae46"}
Sep 09 16:39:37.736187 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:37.736136    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp" event={"ID":"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0","Type":"ContainerStarted","Data":"ec50b13716fa6b923391726c8b97012f5954888252a68a52dcd50a097533ba35"}
Sep 09 16:39:37.752594 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:37.752545    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp" podStartSLOduration=1.752530767 podStartE2EDuration="1.752530767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:39:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:39:37.750850616 +0000 UTC m=+1543.581058675" watchObservedRunningTime="2026-09-09 16:39:37.752530767 +0000 UTC m=+1543.582738863"
Sep 09 16:39:47.214372 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:47.214330    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:39:47.214773 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:47.214385    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:39:47.226950 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:47.226918    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:39:47.808758 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:47.808724    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:39:50.190581 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:39:50.190549    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:40:08.891399 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:08.891359    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:40:22.334709 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.334674    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9"]
Sep 09 16:40:22.335183 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.334923    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9" podUID="e3d6a925-baf0-472f-8813-75e11a1ba8ce" containerName="maas-api" containerID="cri-o://286936415cdfa486061059c9905f5f39623bba6137fc046c5caa096577fc256e" gracePeriod=30
Sep 09 16:40:22.387717 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.387603    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq"]
Sep 09 16:40:22.387886 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.387836    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq" podUID="1eb1331f-f219-4027-bdce-4d93033ea5ec" containerName="payload-processing" containerID="cri-o://ea86a33de03a00feb1bc73bfdc7566eea72f70160994d6693c3928cd14669b16" gracePeriod=30
Sep 09 16:40:22.391652 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.391626    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9"]
Sep 09 16:40:22.391988 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.391956    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9" podUID="a996d66e-b83e-4508-a226-cb72a8fb220a" containerName="payload-pre-processing" containerID="cri-o://00bac8083b28edf2bbed2b3097a4c8624f76d6c62c2ce85c6645de3983015210" gracePeriod=30
Sep 09 16:40:22.475903 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.475871    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9" podUID="e3d6a925-baf0-472f-8813-75e11a1ba8ce" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.88:8443/health\": dial tcp 10.134.0.88:8443: connect: connection refused"
Sep 09 16:40:22.669227 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.669190    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9"
Sep 09 16:40:22.701076 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.701054    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq"
Sep 09 16:40:22.708108 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.708091    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9"
Sep 09 16:40:22.745455 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.745431    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sff5j\" (UniqueName: \"kubernetes.io/projected/e3d6a925-baf0-472f-8813-75e11a1ba8ce-kube-api-access-sff5j\") pod \"e3d6a925-baf0-472f-8813-75e11a1ba8ce\" (UID: \"e3d6a925-baf0-472f-8813-75e11a1ba8ce\") "
Sep 09 16:40:22.745589 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.745471    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d6a925-baf0-472f-8813-75e11a1ba8ce-openshift-service-ca\") pod \"e3d6a925-baf0-472f-8813-75e11a1ba8ce\" (UID: \"e3d6a925-baf0-472f-8813-75e11a1ba8ce\") "
Sep 09 16:40:22.745589 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.745537    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3d6a925-baf0-472f-8813-75e11a1ba8ce-metrics-certs\") pod \"e3d6a925-baf0-472f-8813-75e11a1ba8ce\" (UID: \"e3d6a925-baf0-472f-8813-75e11a1ba8ce\") "
Sep 09 16:40:22.745707 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.745656    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3d6a925-baf0-472f-8813-75e11a1ba8ce-maas-api-tls\") pod \"e3d6a925-baf0-472f-8813-75e11a1ba8ce\" (UID: \"e3d6a925-baf0-472f-8813-75e11a1ba8ce\") "
Sep 09 16:40:22.745926 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.745814    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3d6a925-baf0-472f-8813-75e11a1ba8ce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e3d6a925-baf0-472f-8813-75e11a1ba8ce" (UID: "e3d6a925-baf0-472f-8813-75e11a1ba8ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:40:22.746036 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.745961    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d6a925-baf0-472f-8813-75e11a1ba8ce-openshift-service-ca\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:40:22.747625 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.747598    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3d6a925-baf0-472f-8813-75e11a1ba8ce-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e3d6a925-baf0-472f-8813-75e11a1ba8ce" (UID: "e3d6a925-baf0-472f-8813-75e11a1ba8ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:40:22.747904 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.747883    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3d6a925-baf0-472f-8813-75e11a1ba8ce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e3d6a925-baf0-472f-8813-75e11a1ba8ce" (UID: "e3d6a925-baf0-472f-8813-75e11a1ba8ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:40:22.747963 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.747903    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d6a925-baf0-472f-8813-75e11a1ba8ce-kube-api-access-sff5j" (OuterVolumeSpecName: "kube-api-access-sff5j") pod "e3d6a925-baf0-472f-8813-75e11a1ba8ce" (UID: "e3d6a925-baf0-472f-8813-75e11a1ba8ce"). InnerVolumeSpecName "kube-api-access-sff5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:40:22.846534 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.846453    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvqcv\" (UniqueName: \"kubernetes.io/projected/1eb1331f-f219-4027-bdce-4d93033ea5ec-kube-api-access-kvqcv\") pod \"1eb1331f-f219-4027-bdce-4d93033ea5ec\" (UID: \"1eb1331f-f219-4027-bdce-4d93033ea5ec\") "
Sep 09 16:40:22.846681 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.846583    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a996d66e-b83e-4508-a226-cb72a8fb220a-plugins-config-volume\") pod \"a996d66e-b83e-4508-a226-cb72a8fb220a\" (UID: \"a996d66e-b83e-4508-a226-cb72a8fb220a\") "
Sep 09 16:40:22.846681 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.846624    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fszw6\" (UniqueName: \"kubernetes.io/projected/a996d66e-b83e-4508-a226-cb72a8fb220a-kube-api-access-fszw6\") pod \"a996d66e-b83e-4508-a226-cb72a8fb220a\" (UID: \"a996d66e-b83e-4508-a226-cb72a8fb220a\") "
Sep 09 16:40:22.846681 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.846676    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb1331f-f219-4027-bdce-4d93033ea5ec-plugins-config-volume\") pod \"1eb1331f-f219-4027-bdce-4d93033ea5ec\" (UID: \"1eb1331f-f219-4027-bdce-4d93033ea5ec\") "
Sep 09 16:40:22.846975 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.846945    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a996d66e-b83e-4508-a226-cb72a8fb220a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a996d66e-b83e-4508-a226-cb72a8fb220a" (UID: "a996d66e-b83e-4508-a226-cb72a8fb220a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:40:22.847062 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.846985    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3d6a925-baf0-472f-8813-75e11a1ba8ce-metrics-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:40:22.847062 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.847004    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3d6a925-baf0-472f-8813-75e11a1ba8ce-maas-api-tls\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:40:22.847062 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.847017    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sff5j\" (UniqueName: \"kubernetes.io/projected/e3d6a925-baf0-472f-8813-75e11a1ba8ce-kube-api-access-sff5j\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:40:22.847188 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.847160    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb1331f-f219-4027-bdce-4d93033ea5ec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1eb1331f-f219-4027-bdce-4d93033ea5ec" (UID: "1eb1331f-f219-4027-bdce-4d93033ea5ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:40:22.848660 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.848638    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a996d66e-b83e-4508-a226-cb72a8fb220a-kube-api-access-fszw6" (OuterVolumeSpecName: "kube-api-access-fszw6") pod "a996d66e-b83e-4508-a226-cb72a8fb220a" (UID: "a996d66e-b83e-4508-a226-cb72a8fb220a"). InnerVolumeSpecName "kube-api-access-fszw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:40:22.848709 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.848646    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb1331f-f219-4027-bdce-4d93033ea5ec-kube-api-access-kvqcv" (OuterVolumeSpecName: "kube-api-access-kvqcv") pod "1eb1331f-f219-4027-bdce-4d93033ea5ec" (UID: "1eb1331f-f219-4027-bdce-4d93033ea5ec"). InnerVolumeSpecName "kube-api-access-kvqcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:40:22.947567 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.947535    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a996d66e-b83e-4508-a226-cb72a8fb220a-plugins-config-volume\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:40:22.947567 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.947561    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fszw6\" (UniqueName: \"kubernetes.io/projected/a996d66e-b83e-4508-a226-cb72a8fb220a-kube-api-access-fszw6\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:40:22.947740 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.947575    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb1331f-f219-4027-bdce-4d93033ea5ec-plugins-config-volume\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:40:22.947740 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:22.947586    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvqcv\" (UniqueName: \"kubernetes.io/projected/1eb1331f-f219-4027-bdce-4d93033ea5ec-kube-api-access-kvqcv\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:40:23.007190 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.007150    2555 generic.go:356] "Generic (PLEG): container finished" podID="1eb1331f-f219-4027-bdce-4d93033ea5ec" containerID="ea86a33de03a00feb1bc73bfdc7566eea72f70160994d6693c3928cd14669b16" exitCode=0
Sep 09 16:40:23.007399 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.007193    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq"
Sep 09 16:40:23.007399 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.007231    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq" event={"ID":"1eb1331f-f219-4027-bdce-4d93033ea5ec","Type":"ContainerDied","Data":"ea86a33de03a00feb1bc73bfdc7566eea72f70160994d6693c3928cd14669b16"}
Sep 09 16:40:23.007399 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.007287    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq" event={"ID":"1eb1331f-f219-4027-bdce-4d93033ea5ec","Type":"ContainerDied","Data":"c46599e5059c5c73eb722427b8fb1a8a59931a9951b199c8ccb9bd1cb433c991"}
Sep 09 16:40:23.007399 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.007335    2555 scope.go:117] "RemoveContainer" containerID="ea86a33de03a00feb1bc73bfdc7566eea72f70160994d6693c3928cd14669b16"
Sep 09 16:40:23.008725 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.008701    2555 generic.go:356] "Generic (PLEG): container finished" podID="e3d6a925-baf0-472f-8813-75e11a1ba8ce" containerID="286936415cdfa486061059c9905f5f39623bba6137fc046c5caa096577fc256e" exitCode=0
Sep 09 16:40:23.008905 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.008734    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9" event={"ID":"e3d6a925-baf0-472f-8813-75e11a1ba8ce","Type":"ContainerDied","Data":"286936415cdfa486061059c9905f5f39623bba6137fc046c5caa096577fc256e"}
Sep 09 16:40:23.008905 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.008749    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9"
Sep 09 16:40:23.008905 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.008784    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9" event={"ID":"e3d6a925-baf0-472f-8813-75e11a1ba8ce","Type":"ContainerDied","Data":"bc8807c88c70aa6cf8275f007965c149ae6027d72c2a9bb19e9361ad3babb09c"}
Sep 09 16:40:23.011363 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.011336    2555 generic.go:356] "Generic (PLEG): container finished" podID="a996d66e-b83e-4508-a226-cb72a8fb220a" containerID="00bac8083b28edf2bbed2b3097a4c8624f76d6c62c2ce85c6645de3983015210" exitCode=0
Sep 09 16:40:23.011489 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.011422    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9" event={"ID":"a996d66e-b83e-4508-a226-cb72a8fb220a","Type":"ContainerDied","Data":"00bac8083b28edf2bbed2b3097a4c8624f76d6c62c2ce85c6645de3983015210"}
Sep 09 16:40:23.011489 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.011462    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9" event={"ID":"a996d66e-b83e-4508-a226-cb72a8fb220a","Type":"ContainerDied","Data":"286e0ffbea13928c2ec242a28ed9d3028c0f80a5051f33fad5776bc1297b9971"}
Sep 09 16:40:23.011602 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.011526    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9"
Sep 09 16:40:23.021587 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.021413    2555 scope.go:117] "RemoveContainer" containerID="ea86a33de03a00feb1bc73bfdc7566eea72f70160994d6693c3928cd14669b16"
Sep 09 16:40:23.021805 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:40:23.021785    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea86a33de03a00feb1bc73bfdc7566eea72f70160994d6693c3928cd14669b16\": container with ID starting with ea86a33de03a00feb1bc73bfdc7566eea72f70160994d6693c3928cd14669b16 not found: ID does not exist" containerID="ea86a33de03a00feb1bc73bfdc7566eea72f70160994d6693c3928cd14669b16"
Sep 09 16:40:23.021879 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.021815    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea86a33de03a00feb1bc73bfdc7566eea72f70160994d6693c3928cd14669b16"} err="failed to get container status \"ea86a33de03a00feb1bc73bfdc7566eea72f70160994d6693c3928cd14669b16\": rpc error: code = NotFound desc = could not find container \"ea86a33de03a00feb1bc73bfdc7566eea72f70160994d6693c3928cd14669b16\": container with ID starting with ea86a33de03a00feb1bc73bfdc7566eea72f70160994d6693c3928cd14669b16 not found: ID does not exist"
Sep 09 16:40:23.021879 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.021838    2555 scope.go:117] "RemoveContainer" containerID="286936415cdfa486061059c9905f5f39623bba6137fc046c5caa096577fc256e"
Sep 09 16:40:23.030839 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.030820    2555 scope.go:117] "RemoveContainer" containerID="286936415cdfa486061059c9905f5f39623bba6137fc046c5caa096577fc256e"
Sep 09 16:40:23.031096 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:40:23.031080    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"286936415cdfa486061059c9905f5f39623bba6137fc046c5caa096577fc256e\": container with ID starting with 286936415cdfa486061059c9905f5f39623bba6137fc046c5caa096577fc256e not found: ID does not exist" containerID="286936415cdfa486061059c9905f5f39623bba6137fc046c5caa096577fc256e"
Sep 09 16:40:23.031151 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.031103    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"286936415cdfa486061059c9905f5f39623bba6137fc046c5caa096577fc256e"} err="failed to get container status \"286936415cdfa486061059c9905f5f39623bba6137fc046c5caa096577fc256e\": rpc error: code = NotFound desc = could not find container \"286936415cdfa486061059c9905f5f39623bba6137fc046c5caa096577fc256e\": container with ID starting with 286936415cdfa486061059c9905f5f39623bba6137fc046c5caa096577fc256e not found: ID does not exist"
Sep 09 16:40:23.031151 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.031118    2555 scope.go:117] "RemoveContainer" containerID="00bac8083b28edf2bbed2b3097a4c8624f76d6c62c2ce85c6645de3983015210"
Sep 09 16:40:23.040395 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.040371    2555 scope.go:117] "RemoveContainer" containerID="00bac8083b28edf2bbed2b3097a4c8624f76d6c62c2ce85c6645de3983015210"
Sep 09 16:40:23.040508 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.040444    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9"]
Sep 09 16:40:23.040730 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:40:23.040706    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00bac8083b28edf2bbed2b3097a4c8624f76d6c62c2ce85c6645de3983015210\": container with ID starting with 00bac8083b28edf2bbed2b3097a4c8624f76d6c62c2ce85c6645de3983015210 not found: ID does not exist" containerID="00bac8083b28edf2bbed2b3097a4c8624f76d6c62c2ce85c6645de3983015210"
Sep 09 16:40:23.040806 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.040733    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00bac8083b28edf2bbed2b3097a4c8624f76d6c62c2ce85c6645de3983015210"} err="failed to get container status \"00bac8083b28edf2bbed2b3097a4c8624f76d6c62c2ce85c6645de3983015210\": rpc error: code = NotFound desc = could not find container \"00bac8083b28edf2bbed2b3097a4c8624f76d6c62c2ce85c6645de3983015210\": container with ID starting with 00bac8083b28edf2bbed2b3097a4c8624f76d6c62c2ce85c6645de3983015210 not found: ID does not exist"
Sep 09 16:40:23.044502 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.044478    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-22795bd0-769bd879d5-9hnq9"]
Sep 09 16:40:23.055886 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.055860    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq"]
Sep 09 16:40:23.059185 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.059162    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-22795bd0-65c7d5f8f7-wvrlq"]
Sep 09 16:40:23.077121 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.077096    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9"]
Sep 09 16:40:23.080078 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:23.080057    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-22795bd0-69f8f9b846-cx9c9"]
Sep 09 16:40:24.608398 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:24.608364    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1eb1331f-f219-4027-bdce-4d93033ea5ec" path="/var/lib/kubelet/pods/1eb1331f-f219-4027-bdce-4d93033ea5ec/volumes"
Sep 09 16:40:24.608906 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:24.608885    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a996d66e-b83e-4508-a226-cb72a8fb220a" path="/var/lib/kubelet/pods/a996d66e-b83e-4508-a226-cb72a8fb220a/volumes"
Sep 09 16:40:24.609370 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:24.609349    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3d6a925-baf0-472f-8813-75e11a1ba8ce" path="/var/lib/kubelet/pods/e3d6a925-baf0-472f-8813-75e11a1ba8ce/volumes"
Sep 09 16:40:26.500938 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:26.500906    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:40:30.868546 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:30.868511    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"]
Sep 09 16:40:42.690070 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:42.690037    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:40:45.888512 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:45.888467    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp" podUID="840e9a3a-9ad3-4550-9ec3-d3794fb63fc0" containerName="main" containerID="cri-o://0f2bb6ec8e26cc6d5fef975049da6afa15ebff45c91622957864a4b40b7aae46" gracePeriod=45
Sep 09 16:40:46.145872 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.145806    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:40:46.152197 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.152165    2555 generic.go:356] "Generic (PLEG): container finished" podID="840e9a3a-9ad3-4550-9ec3-d3794fb63fc0" containerID="0f2bb6ec8e26cc6d5fef975049da6afa15ebff45c91622957864a4b40b7aae46" exitCode=0
Sep 09 16:40:46.152346 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.152208    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"
Sep 09 16:40:46.152346 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.152242    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp" event={"ID":"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0","Type":"ContainerDied","Data":"0f2bb6ec8e26cc6d5fef975049da6afa15ebff45c91622957864a4b40b7aae46"}
Sep 09 16:40:46.152346 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.152282    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp" event={"ID":"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0","Type":"ContainerDied","Data":"ec50b13716fa6b923391726c8b97012f5954888252a68a52dcd50a097533ba35"}
Sep 09 16:40:46.152346 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.152328    2555 scope.go:117] "RemoveContainer" containerID="0f2bb6ec8e26cc6d5fef975049da6afa15ebff45c91622957864a4b40b7aae46"
Sep 09 16:40:46.160668 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.160653    2555 scope.go:117] "RemoveContainer" containerID="0f2bb6ec8e26cc6d5fef975049da6afa15ebff45c91622957864a4b40b7aae46"
Sep 09 16:40:46.161015 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:40:46.160995    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f2bb6ec8e26cc6d5fef975049da6afa15ebff45c91622957864a4b40b7aae46\": container with ID starting with 0f2bb6ec8e26cc6d5fef975049da6afa15ebff45c91622957864a4b40b7aae46 not found: ID does not exist" containerID="0f2bb6ec8e26cc6d5fef975049da6afa15ebff45c91622957864a4b40b7aae46"
Sep 09 16:40:46.161084 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.161026    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f2bb6ec8e26cc6d5fef975049da6afa15ebff45c91622957864a4b40b7aae46"} err="failed to get container status \"0f2bb6ec8e26cc6d5fef975049da6afa15ebff45c91622957864a4b40b7aae46\": rpc error: code = NotFound desc = could not find container \"0f2bb6ec8e26cc6d5fef975049da6afa15ebff45c91622957864a4b40b7aae46\": container with ID starting with 0f2bb6ec8e26cc6d5fef975049da6afa15ebff45c91622957864a4b40b7aae46 not found: ID does not exist"
Sep 09 16:40:46.275308 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.275259    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-tmp-dir\") pod \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\" (UID: \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\") "
Sep 09 16:40:46.275499 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.275328    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-home\") pod \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\" (UID: \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\") "
Sep 09 16:40:46.275499 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.275408    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-tls-certs\") pod \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\" (UID: \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\") "
Sep 09 16:40:46.275499 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.275430    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mzgw\" (UniqueName: \"kubernetes.io/projected/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-kube-api-access-4mzgw\") pod \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\" (UID: \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\") "
Sep 09 16:40:46.275499 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.275451    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-model-cache\") pod \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\" (UID: \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\") "
Sep 09 16:40:46.275499 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.275495    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-dshm\") pod \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\" (UID: \"840e9a3a-9ad3-4550-9ec3-d3794fb63fc0\") "
Sep 09 16:40:46.275757 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.275596    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "840e9a3a-9ad3-4550-9ec3-d3794fb63fc0" (UID: "840e9a3a-9ad3-4550-9ec3-d3794fb63fc0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:40:46.275757 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.275612    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-home" (OuterVolumeSpecName: "home") pod "840e9a3a-9ad3-4550-9ec3-d3794fb63fc0" (UID: "840e9a3a-9ad3-4550-9ec3-d3794fb63fc0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:40:46.275757 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.275716    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-model-cache" (OuterVolumeSpecName: "model-cache") pod "840e9a3a-9ad3-4550-9ec3-d3794fb63fc0" (UID: "840e9a3a-9ad3-4550-9ec3-d3794fb63fc0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:40:46.276195 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.275892    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-tmp-dir\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:40:46.276195 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.275914    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-home\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:40:46.276195 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.275926    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-model-cache\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:40:46.277853 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.277824    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-dshm" (OuterVolumeSpecName: "dshm") pod "840e9a3a-9ad3-4550-9ec3-d3794fb63fc0" (UID: "840e9a3a-9ad3-4550-9ec3-d3794fb63fc0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:40:46.277853 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.277827    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-kube-api-access-4mzgw" (OuterVolumeSpecName: "kube-api-access-4mzgw") pod "840e9a3a-9ad3-4550-9ec3-d3794fb63fc0" (UID: "840e9a3a-9ad3-4550-9ec3-d3794fb63fc0"). InnerVolumeSpecName "kube-api-access-4mzgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:40:46.278003 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.277931    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "840e9a3a-9ad3-4550-9ec3-d3794fb63fc0" (UID: "840e9a3a-9ad3-4550-9ec3-d3794fb63fc0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:40:46.377153 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.377117    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-tls-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:40:46.377153 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.377146    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mzgw\" (UniqueName: \"kubernetes.io/projected/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-kube-api-access-4mzgw\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:40:46.377153 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.377157    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0-dshm\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:40:46.473975 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.473947    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"]
Sep 09 16:40:46.479636 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.479609    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-22795bd0/e2e-lifecycle-model-22795bd0-kserve-86d7f45f-qqdkp"]
Sep 09 16:40:46.608747 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:46.608710    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="840e9a3a-9ad3-4550-9ec3-d3794fb63fc0" path="/var/lib/kubelet/pods/840e9a3a-9ad3-4550-9ec3-d3794fb63fc0/volumes"
Sep 09 16:40:58.507355 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:40:58.507319    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:41:00.555428 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:00.555393    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-6957-5ab4ec"]
Sep 09 16:41:00.555810 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:00.555768    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3d6a925-baf0-472f-8813-75e11a1ba8ce" containerName="maas-api"
Sep 09 16:41:00.555810 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:00.555779    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3d6a925-baf0-472f-8813-75e11a1ba8ce" containerName="maas-api"
Sep 09 16:41:00.555810 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:00.555788    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a996d66e-b83e-4508-a226-cb72a8fb220a" containerName="payload-pre-processing"
Sep 09 16:41:00.555810 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:00.555793    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a996d66e-b83e-4508-a226-cb72a8fb220a" containerName="payload-pre-processing"
Sep 09 16:41:00.555810 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:00.555803    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eb1331f-f219-4027-bdce-4d93033ea5ec" containerName="payload-processing"
Sep 09 16:41:00.555810 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:00.555809    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eb1331f-f219-4027-bdce-4d93033ea5ec" containerName="payload-processing"
Sep 09 16:41:00.556012 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:00.555824    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="840e9a3a-9ad3-4550-9ec3-d3794fb63fc0" containerName="main"
Sep 09 16:41:00.556012 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:00.555830    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="840e9a3a-9ad3-4550-9ec3-d3794fb63fc0" containerName="main"
Sep 09 16:41:00.556012 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:00.555892    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="840e9a3a-9ad3-4550-9ec3-d3794fb63fc0" containerName="main"
Sep 09 16:41:00.556012 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:00.555900    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a996d66e-b83e-4508-a226-cb72a8fb220a" containerName="payload-pre-processing"
Sep 09 16:41:00.556012 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:00.555910    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3d6a925-baf0-472f-8813-75e11a1ba8ce" containerName="maas-api"
Sep 09 16:41:00.556012 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:00.555915    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eb1331f-f219-4027-bdce-4d93033ea5ec" containerName="payload-processing"
Sep 09 16:41:00.559241 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:00.559217    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-6957-5ab4ec"
Sep 09 16:41:00.561554 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:00.561521    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-npvh9\""
Sep 09 16:41:00.562733 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:00.562712    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-6957-5ab4ec"]
Sep 09 16:41:00.709882 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:00.709843    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpxrn\" (UniqueName: \"kubernetes.io/projected/8d5be055-f3c0-4dda-a806-98f9d58b38fe-kube-api-access-wpxrn\") pod \"mt-curl-6957-5ab4ec\" (UID: \"8d5be055-f3c0-4dda-a806-98f9d58b38fe\") " pod="opendatahub/mt-curl-6957-5ab4ec"
Sep 09 16:41:00.811120 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:00.811003    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpxrn\" (UniqueName: \"kubernetes.io/projected/8d5be055-f3c0-4dda-a806-98f9d58b38fe-kube-api-access-wpxrn\") pod \"mt-curl-6957-5ab4ec\" (UID: \"8d5be055-f3c0-4dda-a806-98f9d58b38fe\") " pod="opendatahub/mt-curl-6957-5ab4ec"
Sep 09 16:41:00.819051 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:00.819021    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpxrn\" (UniqueName: \"kubernetes.io/projected/8d5be055-f3c0-4dda-a806-98f9d58b38fe-kube-api-access-wpxrn\") pod \"mt-curl-6957-5ab4ec\" (UID: \"8d5be055-f3c0-4dda-a806-98f9d58b38fe\") " pod="opendatahub/mt-curl-6957-5ab4ec"
Sep 09 16:41:00.869865 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:00.869828    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-6957-5ab4ec"
Sep 09 16:41:01.140577 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:01.140543    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-6957-5ab4ec"]
Sep 09 16:41:01.142008 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:41:01.141969    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d5be055_f3c0_4dda_a806_98f9d58b38fe.slice/crio-1425edc52dd07760ae8f08414a8a8c64f742ef47e5c937d370bd237a5bdb7b64 WatchSource:0}: Error finding container 1425edc52dd07760ae8f08414a8a8c64f742ef47e5c937d370bd237a5bdb7b64: Status 404 returned error can't find the container with id 1425edc52dd07760ae8f08414a8a8c64f742ef47e5c937d370bd237a5bdb7b64
Sep 09 16:41:01.254098 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:01.254048    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-6957-5ab4ec" event={"ID":"8d5be055-f3c0-4dda-a806-98f9d58b38fe","Type":"ContainerStarted","Data":"1425edc52dd07760ae8f08414a8a8c64f742ef47e5c937d370bd237a5bdb7b64"}
Sep 09 16:41:02.268734 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:02.268697    2555 generic.go:356] "Generic (PLEG): container finished" podID="8d5be055-f3c0-4dda-a806-98f9d58b38fe" containerID="6330ef6c52728b18ab8594a18c3da329b9eee9214d8e7c6a973594e2ce175b7b" exitCode=0
Sep 09 16:41:02.269106 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:02.268784    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-6957-5ab4ec" event={"ID":"8d5be055-f3c0-4dda-a806-98f9d58b38fe","Type":"ContainerDied","Data":"6330ef6c52728b18ab8594a18c3da329b9eee9214d8e7c6a973594e2ce175b7b"}
Sep 09 16:41:03.405561 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:03.405539    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-6957-5ab4ec"
Sep 09 16:41:03.435591 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:03.435568    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-6957-5ab4ec_8d5be055-f3c0-4dda-a806-98f9d58b38fe/mt-curl-6957-5ab4ec/0.log"
Sep 09 16:41:03.459246 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:03.459222    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-6957-5ab4ec"]
Sep 09 16:41:03.461605 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:03.461586    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-6957-5ab4ec"]
Sep 09 16:41:03.538281 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:03.538176    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpxrn\" (UniqueName: \"kubernetes.io/projected/8d5be055-f3c0-4dda-a806-98f9d58b38fe-kube-api-access-wpxrn\") pod \"8d5be055-f3c0-4dda-a806-98f9d58b38fe\" (UID: \"8d5be055-f3c0-4dda-a806-98f9d58b38fe\") "
Sep 09 16:41:03.540310 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:03.540267    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d5be055-f3c0-4dda-a806-98f9d58b38fe-kube-api-access-wpxrn" (OuterVolumeSpecName: "kube-api-access-wpxrn") pod "8d5be055-f3c0-4dda-a806-98f9d58b38fe" (UID: "8d5be055-f3c0-4dda-a806-98f9d58b38fe"). InnerVolumeSpecName "kube-api-access-wpxrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:41:03.639280 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:03.639237    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpxrn\" (UniqueName: \"kubernetes.io/projected/8d5be055-f3c0-4dda-a806-98f9d58b38fe-kube-api-access-wpxrn\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:41:04.281772 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:04.281743    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-6957-5ab4ec"
Sep 09 16:41:04.281950 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:04.281781    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1425edc52dd07760ae8f08414a8a8c64f742ef47e5c937d370bd237a5bdb7b64"
Sep 09 16:41:04.291585 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:41:04.291553    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-6957-5ab4ec\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="8d5be055-f3c0-4dda-a806-98f9d58b38fe" pod="opendatahub/mt-curl-6957-5ab4ec"
Sep 09 16:41:04.606541 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:41:04.606456    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-6957-5ab4ec\" is forbidden: User \"system:node:ip-10-0-141-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-202.ec2.internal' and this object" podUID="8d5be055-f3c0-4dda-a806-98f9d58b38fe" pod="opendatahub/mt-curl-6957-5ab4ec"
Sep 09 16:41:04.608488 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:04.608463    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d5be055-f3c0-4dda-a806-98f9d58b38fe" path="/var/lib/kubelet/pods/8d5be055-f3c0-4dda-a806-98f9d58b38fe/volumes"
Sep 09 16:41:04.898451 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:04.898371    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"]
Sep 09 16:41:11.598555 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.598518    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"]
Sep 09 16:41:11.598953 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.598939    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d5be055-f3c0-4dda-a806-98f9d58b38fe" containerName="mt-curl-6957-5ab4ec"
Sep 09 16:41:11.598997 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.598955    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d5be055-f3c0-4dda-a806-98f9d58b38fe" containerName="mt-curl-6957-5ab4ec"
Sep 09 16:41:11.599032 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.599021    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d5be055-f3c0-4dda-a806-98f9d58b38fe" containerName="mt-curl-6957-5ab4ec"
Sep 09 16:41:11.602406 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.602385    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:41:11.602947 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.602926    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d1baecf-0643-40f9-bbae-882fae1c12a1-model-cache\") pod \"sub-test-model-331bf8-kserve-789b898c57-zwg6d\" (UID: \"0d1baecf-0643-40f9-bbae-882fae1c12a1\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:41:11.603037 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.602962    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d1baecf-0643-40f9-bbae-882fae1c12a1-tmp-dir\") pod \"sub-test-model-331bf8-kserve-789b898c57-zwg6d\" (UID: \"0d1baecf-0643-40f9-bbae-882fae1c12a1\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:41:11.603037 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.602979    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d1baecf-0643-40f9-bbae-882fae1c12a1-home\") pod \"sub-test-model-331bf8-kserve-789b898c57-zwg6d\" (UID: \"0d1baecf-0643-40f9-bbae-882fae1c12a1\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:41:11.603110 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.603073    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d1baecf-0643-40f9-bbae-882fae1c12a1-dshm\") pod \"sub-test-model-331bf8-kserve-789b898c57-zwg6d\" (UID: \"0d1baecf-0643-40f9-bbae-882fae1c12a1\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:41:11.603144 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.603122    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdt7g\" (UniqueName: \"kubernetes.io/projected/0d1baecf-0643-40f9-bbae-882fae1c12a1-kube-api-access-kdt7g\") pod \"sub-test-model-331bf8-kserve-789b898c57-zwg6d\" (UID: \"0d1baecf-0643-40f9-bbae-882fae1c12a1\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:41:11.603182 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.603159    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d1baecf-0643-40f9-bbae-882fae1c12a1-tls-certs\") pod \"sub-test-model-331bf8-kserve-789b898c57-zwg6d\" (UID: \"0d1baecf-0643-40f9-bbae-882fae1c12a1\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:41:11.604958 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.604936    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-331bf8\"/\"sub-test-model-331bf8-kserve-self-signed-certs\""
Sep 09 16:41:11.610758 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.610738    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"]
Sep 09 16:41:11.703862 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.703827    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d1baecf-0643-40f9-bbae-882fae1c12a1-dshm\") pod \"sub-test-model-331bf8-kserve-789b898c57-zwg6d\" (UID: \"0d1baecf-0643-40f9-bbae-882fae1c12a1\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:41:11.704047 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.703903    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdt7g\" (UniqueName: \"kubernetes.io/projected/0d1baecf-0643-40f9-bbae-882fae1c12a1-kube-api-access-kdt7g\") pod \"sub-test-model-331bf8-kserve-789b898c57-zwg6d\" (UID: \"0d1baecf-0643-40f9-bbae-882fae1c12a1\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:41:11.704047 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.703940    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d1baecf-0643-40f9-bbae-882fae1c12a1-tls-certs\") pod \"sub-test-model-331bf8-kserve-789b898c57-zwg6d\" (UID: \"0d1baecf-0643-40f9-bbae-882fae1c12a1\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:41:11.704047 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.703975    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d1baecf-0643-40f9-bbae-882fae1c12a1-model-cache\") pod \"sub-test-model-331bf8-kserve-789b898c57-zwg6d\" (UID: \"0d1baecf-0643-40f9-bbae-882fae1c12a1\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:41:11.704047 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.704033    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d1baecf-0643-40f9-bbae-882fae1c12a1-tmp-dir\") pod \"sub-test-model-331bf8-kserve-789b898c57-zwg6d\" (UID: \"0d1baecf-0643-40f9-bbae-882fae1c12a1\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:41:11.704286 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.704060    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d1baecf-0643-40f9-bbae-882fae1c12a1-home\") pod \"sub-test-model-331bf8-kserve-789b898c57-zwg6d\" (UID: \"0d1baecf-0643-40f9-bbae-882fae1c12a1\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:41:11.704438 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.704411    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d1baecf-0643-40f9-bbae-882fae1c12a1-model-cache\") pod \"sub-test-model-331bf8-kserve-789b898c57-zwg6d\" (UID: \"0d1baecf-0643-40f9-bbae-882fae1c12a1\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:41:11.704499 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.704464    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d1baecf-0643-40f9-bbae-882fae1c12a1-tmp-dir\") pod \"sub-test-model-331bf8-kserve-789b898c57-zwg6d\" (UID: \"0d1baecf-0643-40f9-bbae-882fae1c12a1\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:41:11.704562 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.704493    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d1baecf-0643-40f9-bbae-882fae1c12a1-home\") pod \"sub-test-model-331bf8-kserve-789b898c57-zwg6d\" (UID: \"0d1baecf-0643-40f9-bbae-882fae1c12a1\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:41:11.706178 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.706153    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d1baecf-0643-40f9-bbae-882fae1c12a1-dshm\") pod \"sub-test-model-331bf8-kserve-789b898c57-zwg6d\" (UID: \"0d1baecf-0643-40f9-bbae-882fae1c12a1\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:41:11.706354 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.706312    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d1baecf-0643-40f9-bbae-882fae1c12a1-tls-certs\") pod \"sub-test-model-331bf8-kserve-789b898c57-zwg6d\" (UID: \"0d1baecf-0643-40f9-bbae-882fae1c12a1\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:41:11.711381 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.711364    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdt7g\" (UniqueName: \"kubernetes.io/projected/0d1baecf-0643-40f9-bbae-882fae1c12a1-kube-api-access-kdt7g\") pod \"sub-test-model-331bf8-kserve-789b898c57-zwg6d\" (UID: \"0d1baecf-0643-40f9-bbae-882fae1c12a1\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:41:11.912496 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:11.912417    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:41:12.194771 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:12.194747    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"]
Sep 09 16:41:12.196258 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:41:12.196229    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d1baecf_0643_40f9_bbae_882fae1c12a1.slice/crio-23445ed2169007e164f8875c18cd1accca01149176bea24bbfdbbed55abc379b WatchSource:0}: Error finding container 23445ed2169007e164f8875c18cd1accca01149176bea24bbfdbbed55abc379b: Status 404 returned error can't find the container with id 23445ed2169007e164f8875c18cd1accca01149176bea24bbfdbbed55abc379b
Sep 09 16:41:12.328393 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:12.328338    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d" event={"ID":"0d1baecf-0643-40f9-bbae-882fae1c12a1","Type":"ContainerStarted","Data":"9f42d979ed3b6194cd971574dfca1620f7f2257c4ddd660505676f5be403eef0"}
Sep 09 16:41:12.328546 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:12.328407    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d" event={"ID":"0d1baecf-0643-40f9-bbae-882fae1c12a1","Type":"ContainerStarted","Data":"23445ed2169007e164f8875c18cd1accca01149176bea24bbfdbbed55abc379b"}
Sep 09 16:41:12.344036 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:12.343991    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d" podStartSLOduration=1.343974199 podStartE2EDuration="1.343974199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:41:12.342968556 +0000 UTC m=+1638.173176627" watchObservedRunningTime="2026-09-09 16:41:12.343974199 +0000 UTC m=+1638.174182257"
Sep 09 16:41:16.139917 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:16.139880    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:41:19.917785 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:19.917746    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc" podUID="d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7" containerName="main" containerID="cri-o://9fd8f287dc8353b1208b45ec81cb338341b128923bab9d571b427ea646d24158" gracePeriod=45
Sep 09 16:41:20.176449 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.176385    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:41:20.276631 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.276592    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85t2j\" (UniqueName: \"kubernetes.io/projected/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-kube-api-access-85t2j\") pod \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\" (UID: \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\") "
Sep 09 16:41:20.276815 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.276666    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-model-cache\") pod \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\" (UID: \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\") "
Sep 09 16:41:20.276815 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.276742    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-dshm\") pod \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\" (UID: \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\") "
Sep 09 16:41:20.276815 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.276783    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-tls-certs\") pod \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\" (UID: \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\") "
Sep 09 16:41:20.276815 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.276811    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-tmp-dir\") pod \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\" (UID: \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\") "
Sep 09 16:41:20.277031 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.276847    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-home\") pod \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\" (UID: \"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7\") "
Sep 09 16:41:20.277031 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.276963    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-model-cache" (OuterVolumeSpecName: "model-cache") pod "d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7" (UID: "d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:41:20.277216 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.277193    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-home" (OuterVolumeSpecName: "home") pod "d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7" (UID: "d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:41:20.277318 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.277210    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7" (UID: "d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:41:20.277394 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.277337    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-tmp-dir\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:41:20.277394 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.277358    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-home\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:41:20.277394 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.277371    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-model-cache\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:41:20.279067 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.279035    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-dshm" (OuterVolumeSpecName: "dshm") pod "d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7" (UID: "d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:41:20.279178 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.279109    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-kube-api-access-85t2j" (OuterVolumeSpecName: "kube-api-access-85t2j") pod "d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7" (UID: "d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7"). InnerVolumeSpecName "kube-api-access-85t2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:41:20.279339 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.279288    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7" (UID: "d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:41:20.377871 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.377849    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85t2j\" (UniqueName: \"kubernetes.io/projected/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-kube-api-access-85t2j\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:41:20.377871 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.377871    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-dshm\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:41:20.378053 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.377880    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7-tls-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:41:20.378691 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.378666    2555 generic.go:356] "Generic (PLEG): container finished" podID="d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7" containerID="9fd8f287dc8353b1208b45ec81cb338341b128923bab9d571b427ea646d24158" exitCode=0
Sep 09 16:41:20.378744 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.378713    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"
Sep 09 16:41:20.378777 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.378747    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc" event={"ID":"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7","Type":"ContainerDied","Data":"9fd8f287dc8353b1208b45ec81cb338341b128923bab9d571b427ea646d24158"}
Sep 09 16:41:20.378815 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.378792    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc" event={"ID":"d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7","Type":"ContainerDied","Data":"cb586121af2bc10165b1fee31e51a1cc184cebdc30813cd66bd0a7629a46d65e"}
Sep 09 16:41:20.378815 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.378811    2555 scope.go:117] "RemoveContainer" containerID="9fd8f287dc8353b1208b45ec81cb338341b128923bab9d571b427ea646d24158"
Sep 09 16:41:20.388033 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.388016    2555 scope.go:117] "RemoveContainer" containerID="9fd8f287dc8353b1208b45ec81cb338341b128923bab9d571b427ea646d24158"
Sep 09 16:41:20.388346 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:41:20.388271    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fd8f287dc8353b1208b45ec81cb338341b128923bab9d571b427ea646d24158\": container with ID starting with 9fd8f287dc8353b1208b45ec81cb338341b128923bab9d571b427ea646d24158 not found: ID does not exist" containerID="9fd8f287dc8353b1208b45ec81cb338341b128923bab9d571b427ea646d24158"
Sep 09 16:41:20.388402 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.388356    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fd8f287dc8353b1208b45ec81cb338341b128923bab9d571b427ea646d24158"} err="failed to get container status \"9fd8f287dc8353b1208b45ec81cb338341b128923bab9d571b427ea646d24158\": rpc error: code = NotFound desc = could not find container \"9fd8f287dc8353b1208b45ec81cb338341b128923bab9d571b427ea646d24158\": container with ID starting with 9fd8f287dc8353b1208b45ec81cb338341b128923bab9d571b427ea646d24158 not found: ID does not exist"
Sep 09 16:41:20.399438 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.399416    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"]
Sep 09 16:41:20.405104 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.405084    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-331bf8/auth-test-model-331bf8-kserve-6b49f79659-74jdc"]
Sep 09 16:41:20.608220 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:20.608178    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7" path="/var/lib/kubelet/pods/d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7/volumes"
Sep 09 16:41:21.913173 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:21.913142    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:41:21.913532 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:21.913200    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:41:21.925532 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:21.925507    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:41:22.403523 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:22.403489    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:41:46.500936 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.500901    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"]
Sep 09 16:41:46.501537 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.501366    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7" containerName="main"
Sep 09 16:41:46.501537 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.501380    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7" containerName="main"
Sep 09 16:41:46.501537 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.501442    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d06cfd5b-3d5b-4d1b-8f8d-f69be17d30c7" containerName="main"
Sep 09 16:41:46.506614 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.506592    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:41:46.511143 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.510829    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-50fcbc\"/\"default-dockercfg-qltth\""
Sep 09 16:41:46.511315 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.511264    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-50fcbc\"/\"openshift-service-ca.crt\""
Sep 09 16:41:46.512118 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.511454    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-50fcbc\"/\"sub-test-model-50fcbc-kserve-self-signed-certs\""
Sep 09 16:41:46.512118 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.511574    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-50fcbc\"/\"kube-root-ca.crt\""
Sep 09 16:41:46.515174 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.515140    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"]
Sep 09 16:41:46.627614 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.627584    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6211c979-e5c2-4e15-b1a1-e0dadfad13be-dshm\") pod \"sub-test-model-50fcbc-kserve-7cf449988f-jqps4\" (UID: \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:41:46.627614 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.627618    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6211c979-e5c2-4e15-b1a1-e0dadfad13be-tls-certs\") pod \"sub-test-model-50fcbc-kserve-7cf449988f-jqps4\" (UID: \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:41:46.627819 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.627646    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6211c979-e5c2-4e15-b1a1-e0dadfad13be-model-cache\") pod \"sub-test-model-50fcbc-kserve-7cf449988f-jqps4\" (UID: \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:41:46.627819 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.627784    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clcll\" (UniqueName: \"kubernetes.io/projected/6211c979-e5c2-4e15-b1a1-e0dadfad13be-kube-api-access-clcll\") pod \"sub-test-model-50fcbc-kserve-7cf449988f-jqps4\" (UID: \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:41:46.627886 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.627823    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6211c979-e5c2-4e15-b1a1-e0dadfad13be-home\") pod \"sub-test-model-50fcbc-kserve-7cf449988f-jqps4\" (UID: \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:41:46.627920 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.627884    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6211c979-e5c2-4e15-b1a1-e0dadfad13be-tmp-dir\") pod \"sub-test-model-50fcbc-kserve-7cf449988f-jqps4\" (UID: \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:41:46.728716 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.728683    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6211c979-e5c2-4e15-b1a1-e0dadfad13be-dshm\") pod \"sub-test-model-50fcbc-kserve-7cf449988f-jqps4\" (UID: \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:41:46.728716 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.728716    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6211c979-e5c2-4e15-b1a1-e0dadfad13be-tls-certs\") pod \"sub-test-model-50fcbc-kserve-7cf449988f-jqps4\" (UID: \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:41:46.728919 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.728739    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6211c979-e5c2-4e15-b1a1-e0dadfad13be-model-cache\") pod \"sub-test-model-50fcbc-kserve-7cf449988f-jqps4\" (UID: \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:41:46.728919 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.728811    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clcll\" (UniqueName: \"kubernetes.io/projected/6211c979-e5c2-4e15-b1a1-e0dadfad13be-kube-api-access-clcll\") pod \"sub-test-model-50fcbc-kserve-7cf449988f-jqps4\" (UID: \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:41:46.728919 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.728837    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6211c979-e5c2-4e15-b1a1-e0dadfad13be-home\") pod \"sub-test-model-50fcbc-kserve-7cf449988f-jqps4\" (UID: \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:41:46.728919 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.728882    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6211c979-e5c2-4e15-b1a1-e0dadfad13be-tmp-dir\") pod \"sub-test-model-50fcbc-kserve-7cf449988f-jqps4\" (UID: \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:41:46.729235 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.729206    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6211c979-e5c2-4e15-b1a1-e0dadfad13be-model-cache\") pod \"sub-test-model-50fcbc-kserve-7cf449988f-jqps4\" (UID: \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:41:46.729389 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.729243    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6211c979-e5c2-4e15-b1a1-e0dadfad13be-tmp-dir\") pod \"sub-test-model-50fcbc-kserve-7cf449988f-jqps4\" (UID: \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:41:46.729389 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.729323    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6211c979-e5c2-4e15-b1a1-e0dadfad13be-home\") pod \"sub-test-model-50fcbc-kserve-7cf449988f-jqps4\" (UID: \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:41:46.730892 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.730873    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6211c979-e5c2-4e15-b1a1-e0dadfad13be-dshm\") pod \"sub-test-model-50fcbc-kserve-7cf449988f-jqps4\" (UID: \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:41:46.731176 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.731155    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6211c979-e5c2-4e15-b1a1-e0dadfad13be-tls-certs\") pod \"sub-test-model-50fcbc-kserve-7cf449988f-jqps4\" (UID: \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:41:46.736305 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.736271    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clcll\" (UniqueName: \"kubernetes.io/projected/6211c979-e5c2-4e15-b1a1-e0dadfad13be-kube-api-access-clcll\") pod \"sub-test-model-50fcbc-kserve-7cf449988f-jqps4\" (UID: \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:41:46.818485 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:46.818394    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:41:47.088677 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:47.088649    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"]
Sep 09 16:41:47.090998 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:41:47.090969    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6211c979_e5c2_4e15_b1a1_e0dadfad13be.slice/crio-7ca421c70f3278c937614278bb6bdfc2e33782fa06c2226190cc5b2d244569ff WatchSource:0}: Error finding container 7ca421c70f3278c937614278bb6bdfc2e33782fa06c2226190cc5b2d244569ff: Status 404 returned error can't find the container with id 7ca421c70f3278c937614278bb6bdfc2e33782fa06c2226190cc5b2d244569ff
Sep 09 16:41:47.537696 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:47.537658    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4" event={"ID":"6211c979-e5c2-4e15-b1a1-e0dadfad13be","Type":"ContainerStarted","Data":"9cd46e8651c9805acef4225d9f4add6697a4fdc2b769c1c6a08dc15da1b48212"}
Sep 09 16:41:47.537696 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:47.537701    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4" event={"ID":"6211c979-e5c2-4e15-b1a1-e0dadfad13be","Type":"ContainerStarted","Data":"7ca421c70f3278c937614278bb6bdfc2e33782fa06c2226190cc5b2d244569ff"}
Sep 09 16:41:47.560127 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:47.560081    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4" podStartSLOduration=1.560066473 podStartE2EDuration="1.560066473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:41:47.55846288 +0000 UTC m=+1673.388670936" watchObservedRunningTime="2026-09-09 16:41:47.560066473 +0000 UTC m=+1673.390274531"
Sep 09 16:41:50.502624 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:50.502589    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:41:56.819518 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:56.819483    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:41:56.819971 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:56.819534    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:41:56.831736 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:56.831707    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:41:57.611165 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:41:57.611134    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:42:14.500251 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:14.500217    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:42:19.141474 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.141440    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk"]
Sep 09 16:42:19.145184 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.145163    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk"
Sep 09 16:42:19.147780 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.147756    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-14e637-serving-cert\""
Sep 09 16:42:19.156685 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.156659    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk"]
Sep 09 16:42:19.232181 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.232148    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v5dp\" (UniqueName: \"kubernetes.io/projected/2e9b4b90-4f61-4aa7-943e-f6601e24c61c-kube-api-access-6v5dp\") pod \"maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk\" (UID: \"2e9b4b90-4f61-4aa7-943e-f6601e24c61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk"
Sep 09 16:42:19.232181 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.232189    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e9b4b90-4f61-4aa7-943e-f6601e24c61c-metrics-certs\") pod \"maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk\" (UID: \"2e9b4b90-4f61-4aa7-943e-f6601e24c61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk"
Sep 09 16:42:19.232533 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.232372    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9b4b90-4f61-4aa7-943e-f6601e24c61c-openshift-service-ca\") pod \"maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk\" (UID: \"2e9b4b90-4f61-4aa7-943e-f6601e24c61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk"
Sep 09 16:42:19.232533 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.232412    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e9b4b90-4f61-4aa7-943e-f6601e24c61c-maas-api-tls\") pod \"maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk\" (UID: \"2e9b4b90-4f61-4aa7-943e-f6601e24c61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk"
Sep 09 16:42:19.334508 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.334207    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v5dp\" (UniqueName: \"kubernetes.io/projected/2e9b4b90-4f61-4aa7-943e-f6601e24c61c-kube-api-access-6v5dp\") pod \"maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk\" (UID: \"2e9b4b90-4f61-4aa7-943e-f6601e24c61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk"
Sep 09 16:42:19.334508 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.334262    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e9b4b90-4f61-4aa7-943e-f6601e24c61c-metrics-certs\") pod \"maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk\" (UID: \"2e9b4b90-4f61-4aa7-943e-f6601e24c61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk"
Sep 09 16:42:19.334508 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.334391    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9b4b90-4f61-4aa7-943e-f6601e24c61c-openshift-service-ca\") pod \"maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk\" (UID: \"2e9b4b90-4f61-4aa7-943e-f6601e24c61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk"
Sep 09 16:42:19.334508 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.334421    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e9b4b90-4f61-4aa7-943e-f6601e24c61c-maas-api-tls\") pod \"maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk\" (UID: \"2e9b4b90-4f61-4aa7-943e-f6601e24c61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk"
Sep 09 16:42:19.336807 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.336713    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9b4b90-4f61-4aa7-943e-f6601e24c61c-openshift-service-ca\") pod \"maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk\" (UID: \"2e9b4b90-4f61-4aa7-943e-f6601e24c61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk"
Sep 09 16:42:19.338054 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.338028    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e9b4b90-4f61-4aa7-943e-f6601e24c61c-maas-api-tls\") pod \"maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk\" (UID: \"2e9b4b90-4f61-4aa7-943e-f6601e24c61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk"
Sep 09 16:42:19.339347 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.339325    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e9b4b90-4f61-4aa7-943e-f6601e24c61c-metrics-certs\") pod \"maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk\" (UID: \"2e9b4b90-4f61-4aa7-943e-f6601e24c61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk"
Sep 09 16:42:19.342529 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.342510    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v5dp\" (UniqueName: \"kubernetes.io/projected/2e9b4b90-4f61-4aa7-943e-f6601e24c61c-kube-api-access-6v5dp\") pod \"maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk\" (UID: \"2e9b4b90-4f61-4aa7-943e-f6601e24c61c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk"
Sep 09 16:42:19.457129 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.457055    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk"
Sep 09 16:42:19.734665 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.734634    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk"]
Sep 09 16:42:19.735773 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:42:19.735747    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9b4b90_4f61_4aa7_943e_f6601e24c61c.slice/crio-1b4b9f89430a2c030b6fcf5548b0f8a6096b3cafe5f7507646841972b7460b7a WatchSource:0}: Error finding container 1b4b9f89430a2c030b6fcf5548b0f8a6096b3cafe5f7507646841972b7460b7a: Status 404 returned error can't find the container with id 1b4b9f89430a2c030b6fcf5548b0f8a6096b3cafe5f7507646841972b7460b7a
Sep 09 16:42:19.737420 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.737402    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 16:42:19.838419 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.838376    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz"]
Sep 09 16:42:19.846915 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.846890    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz"
Sep 09 16:42:19.849620 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.849593    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-14e637\""
Sep 09 16:42:19.849853 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.849834    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-14e637-dockercfg-rfdz6\""
Sep 09 16:42:19.851353 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.851322    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz"]
Sep 09 16:42:19.944941 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.944904    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gjgn\" (UniqueName: \"kubernetes.io/projected/616c6d67-a842-4d6f-bcb3-efc0c73719fc-kube-api-access-9gjgn\") pod \"payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz\" (UID: \"616c6d67-a842-4d6f-bcb3-efc0c73719fc\") " pod="openshift-ingress/payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz"
Sep 09 16:42:19.945115 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:19.944958    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616c6d67-a842-4d6f-bcb3-efc0c73719fc-plugins-config-volume\") pod \"payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz\" (UID: \"616c6d67-a842-4d6f-bcb3-efc0c73719fc\") " pod="openshift-ingress/payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz"
Sep 09 16:42:20.046018 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:20.045978    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gjgn\" (UniqueName: \"kubernetes.io/projected/616c6d67-a842-4d6f-bcb3-efc0c73719fc-kube-api-access-9gjgn\") pod \"payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz\" (UID: \"616c6d67-a842-4d6f-bcb3-efc0c73719fc\") " pod="openshift-ingress/payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz"
Sep 09 16:42:20.046214 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:20.046030    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616c6d67-a842-4d6f-bcb3-efc0c73719fc-plugins-config-volume\") pod \"payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz\" (UID: \"616c6d67-a842-4d6f-bcb3-efc0c73719fc\") " pod="openshift-ingress/payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz"
Sep 09 16:42:20.046813 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:20.046791    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616c6d67-a842-4d6f-bcb3-efc0c73719fc-plugins-config-volume\") pod \"payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz\" (UID: \"616c6d67-a842-4d6f-bcb3-efc0c73719fc\") " pod="openshift-ingress/payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz"
Sep 09 16:42:20.054189 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:20.054159    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gjgn\" (UniqueName: \"kubernetes.io/projected/616c6d67-a842-4d6f-bcb3-efc0c73719fc-kube-api-access-9gjgn\") pod \"payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz\" (UID: \"616c6d67-a842-4d6f-bcb3-efc0c73719fc\") " pod="openshift-ingress/payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz"
Sep 09 16:42:20.159501 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:20.159463    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz"
Sep 09 16:42:20.430134 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:20.430108    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz"]
Sep 09 16:42:20.431653 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:42:20.431618    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod616c6d67_a842_4d6f_bcb3_efc0c73719fc.slice/crio-294c9e1b81d806591b9a365377807121de3f20372d80de5e668330b3769aa9bb WatchSource:0}: Error finding container 294c9e1b81d806591b9a365377807121de3f20372d80de5e668330b3769aa9bb: Status 404 returned error can't find the container with id 294c9e1b81d806591b9a365377807121de3f20372d80de5e668330b3769aa9bb
Sep 09 16:42:20.741876 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:20.741842    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz" event={"ID":"616c6d67-a842-4d6f-bcb3-efc0c73719fc","Type":"ContainerStarted","Data":"e47ddb1ea52ded4e5199bda4605ae50aa3b84da826e40185b41f961bd361a710"}
Sep 09 16:42:20.742073 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:20.741908    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz" event={"ID":"616c6d67-a842-4d6f-bcb3-efc0c73719fc","Type":"ContainerStarted","Data":"294c9e1b81d806591b9a365377807121de3f20372d80de5e668330b3769aa9bb"}
Sep 09 16:42:20.743818 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:20.743785    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk" event={"ID":"2e9b4b90-4f61-4aa7-943e-f6601e24c61c","Type":"ContainerStarted","Data":"5c8a2844c99854479278924742f06a50bdeb35859c21a09f4155050965ab5b1e"}
Sep 09 16:42:20.743963 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:20.743890    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk"
Sep 09 16:42:20.743963 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:20.743919    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk" event={"ID":"2e9b4b90-4f61-4aa7-943e-f6601e24c61c","Type":"ContainerStarted","Data":"1b4b9f89430a2c030b6fcf5548b0f8a6096b3cafe5f7507646841972b7460b7a"}
Sep 09 16:42:20.760933 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:20.760885    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz" podStartSLOduration=1.760871544 podStartE2EDuration="1.760871544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:42:20.758458535 +0000 UTC m=+1706.588666625" watchObservedRunningTime="2026-09-09 16:42:20.760871544 +0000 UTC m=+1706.591079655"
Sep 09 16:42:20.779011 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:20.778961    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk" podStartSLOduration=1.20392612 podStartE2EDuration="1.778943018s" totalImagesPullingTime="575.016898ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:42:20.775826026 +0000 UTC m=+1706.606034085" watchObservedRunningTime="2026-09-09 16:42:20.778943018 +0000 UTC m=+1706.609151078"
Sep 09 16:42:21.750546 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:21.750520    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz"
Sep 09 16:42:26.756781 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:26.756752    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk"
Sep 09 16:42:32.192056 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:32.192024    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:42:32.759069 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:32.759039    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz"
Sep 09 16:42:43.357260 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:43.357225    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"]
Sep 09 16:42:44.304414 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:44.304381    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"]
Sep 09 16:42:51.000707 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:51.000677    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:42:54.433869 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:54.433834    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc"]
Sep 09 16:42:54.438690 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:54.438669    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc"
Sep 09 16:42:54.441388 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:54.441365    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8cb67b-dockercfg-hbc8j\""
Sep 09 16:42:54.441621 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:54.441382    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8cb67b\""
Sep 09 16:42:54.447123 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:54.447096    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc"]
Sep 09 16:42:54.582471 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:54.582433    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh8x5\" (UniqueName: \"kubernetes.io/projected/fa0e6ab2-4fbd-4669-8d3c-38191b9e756f-kube-api-access-jh8x5\") pod \"payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc\" (UID: \"fa0e6ab2-4fbd-4669-8d3c-38191b9e756f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc"
Sep 09 16:42:54.582631 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:54.582547    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa0e6ab2-4fbd-4669-8d3c-38191b9e756f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc\" (UID: \"fa0e6ab2-4fbd-4669-8d3c-38191b9e756f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc"
Sep 09 16:42:54.683639 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:54.683593    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa0e6ab2-4fbd-4669-8d3c-38191b9e756f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc\" (UID: \"fa0e6ab2-4fbd-4669-8d3c-38191b9e756f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc"
Sep 09 16:42:54.683840 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:54.683696    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh8x5\" (UniqueName: \"kubernetes.io/projected/fa0e6ab2-4fbd-4669-8d3c-38191b9e756f-kube-api-access-jh8x5\") pod \"payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc\" (UID: \"fa0e6ab2-4fbd-4669-8d3c-38191b9e756f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc"
Sep 09 16:42:54.686105 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:54.686052    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8cb67b\""
Sep 09 16:42:54.691468 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:54.691449    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh8x5\" (UniqueName: \"kubernetes.io/projected/fa0e6ab2-4fbd-4669-8d3c-38191b9e756f-kube-api-access-jh8x5\") pod \"payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc\" (UID: \"fa0e6ab2-4fbd-4669-8d3c-38191b9e756f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc"
Sep 09 16:42:54.694967 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:54.694943    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa0e6ab2-4fbd-4669-8d3c-38191b9e756f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc\" (UID: \"fa0e6ab2-4fbd-4669-8d3c-38191b9e756f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc"
Sep 09 16:42:54.755015 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:54.754987    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8cb67b-dockercfg-hbc8j\""
Sep 09 16:42:54.763189 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:54.763166    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc"
Sep 09 16:42:55.032607 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:42:55.032574    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0e6ab2_4fbd_4669_8d3c_38191b9e756f.slice/crio-d9d119a631ca997fe5161e4a958f2eae93bd4663f9623104d585725154dc5731 WatchSource:0}: Error finding container d9d119a631ca997fe5161e4a958f2eae93bd4663f9623104d585725154dc5731: Status 404 returned error can't find the container with id d9d119a631ca997fe5161e4a958f2eae93bd4663f9623104d585725154dc5731
Sep 09 16:42:55.032856 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:55.032830    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc"]
Sep 09 16:42:55.962498 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:55.962453    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc" event={"ID":"fa0e6ab2-4fbd-4669-8d3c-38191b9e756f","Type":"ContainerStarted","Data":"62ce16fd41e1ee769c1ac9d19478cf2fa2532c3b473108b4de569e0072ebe597"}
Sep 09 16:42:55.962871 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:55.962623    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc"
Sep 09 16:42:55.962871 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:55.962664    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc" event={"ID":"fa0e6ab2-4fbd-4669-8d3c-38191b9e756f","Type":"ContainerStarted","Data":"d9d119a631ca997fe5161e4a958f2eae93bd4663f9623104d585725154dc5731"}
Sep 09 16:42:55.979511 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:55.979468    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc" podStartSLOduration=1.979454315 podStartE2EDuration="1.979454315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:42:55.976819207 +0000 UTC m=+1741.807027267" watchObservedRunningTime="2026-09-09 16:42:55.979454315 +0000 UTC m=+1741.809662374"
Sep 09 16:42:58.377084 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.377039    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d" podUID="0d1baecf-0643-40f9-bbae-882fae1c12a1" containerName="main" containerID="cri-o://9f42d979ed3b6194cd971574dfca1620f7f2257c4ddd660505676f5be403eef0" gracePeriod=45
Sep 09 16:42:58.638037 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.637970    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:42:58.725237 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.725209    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d1baecf-0643-40f9-bbae-882fae1c12a1-tls-certs\") pod \"0d1baecf-0643-40f9-bbae-882fae1c12a1\" (UID: \"0d1baecf-0643-40f9-bbae-882fae1c12a1\") "
Sep 09 16:42:58.725420 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.725313    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d1baecf-0643-40f9-bbae-882fae1c12a1-tmp-dir\") pod \"0d1baecf-0643-40f9-bbae-882fae1c12a1\" (UID: \"0d1baecf-0643-40f9-bbae-882fae1c12a1\") "
Sep 09 16:42:58.725420 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.725350    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d1baecf-0643-40f9-bbae-882fae1c12a1-model-cache\") pod \"0d1baecf-0643-40f9-bbae-882fae1c12a1\" (UID: \"0d1baecf-0643-40f9-bbae-882fae1c12a1\") "
Sep 09 16:42:58.725420 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.725370    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdt7g\" (UniqueName: \"kubernetes.io/projected/0d1baecf-0643-40f9-bbae-882fae1c12a1-kube-api-access-kdt7g\") pod \"0d1baecf-0643-40f9-bbae-882fae1c12a1\" (UID: \"0d1baecf-0643-40f9-bbae-882fae1c12a1\") "
Sep 09 16:42:58.725420 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.725417    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d1baecf-0643-40f9-bbae-882fae1c12a1-home\") pod \"0d1baecf-0643-40f9-bbae-882fae1c12a1\" (UID: \"0d1baecf-0643-40f9-bbae-882fae1c12a1\") "
Sep 09 16:42:58.725664 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.725457    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d1baecf-0643-40f9-bbae-882fae1c12a1-dshm\") pod \"0d1baecf-0643-40f9-bbae-882fae1c12a1\" (UID: \"0d1baecf-0643-40f9-bbae-882fae1c12a1\") "
Sep 09 16:42:58.725664 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.725570    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1baecf-0643-40f9-bbae-882fae1c12a1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d1baecf-0643-40f9-bbae-882fae1c12a1" (UID: "0d1baecf-0643-40f9-bbae-882fae1c12a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:42:58.725664 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.725640    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1baecf-0643-40f9-bbae-882fae1c12a1-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d1baecf-0643-40f9-bbae-882fae1c12a1" (UID: "0d1baecf-0643-40f9-bbae-882fae1c12a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:42:58.725819 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.725748    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1baecf-0643-40f9-bbae-882fae1c12a1-home" (OuterVolumeSpecName: "home") pod "0d1baecf-0643-40f9-bbae-882fae1c12a1" (UID: "0d1baecf-0643-40f9-bbae-882fae1c12a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:42:58.725865 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.725853    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d1baecf-0643-40f9-bbae-882fae1c12a1-tmp-dir\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:42:58.725924 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.725872    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d1baecf-0643-40f9-bbae-882fae1c12a1-model-cache\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:42:58.725924 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.725884    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d1baecf-0643-40f9-bbae-882fae1c12a1-home\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:42:58.727452 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.727424    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d1baecf-0643-40f9-bbae-882fae1c12a1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d1baecf-0643-40f9-bbae-882fae1c12a1" (UID: "0d1baecf-0643-40f9-bbae-882fae1c12a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:42:58.727558 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.727483    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d1baecf-0643-40f9-bbae-882fae1c12a1-kube-api-access-kdt7g" (OuterVolumeSpecName: "kube-api-access-kdt7g") pod "0d1baecf-0643-40f9-bbae-882fae1c12a1" (UID: "0d1baecf-0643-40f9-bbae-882fae1c12a1"). InnerVolumeSpecName "kube-api-access-kdt7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:42:58.727558 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.727488    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1baecf-0643-40f9-bbae-882fae1c12a1-dshm" (OuterVolumeSpecName: "dshm") pod "0d1baecf-0643-40f9-bbae-882fae1c12a1" (UID: "0d1baecf-0643-40f9-bbae-882fae1c12a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:42:58.827214 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.827182    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdt7g\" (UniqueName: \"kubernetes.io/projected/0d1baecf-0643-40f9-bbae-882fae1c12a1-kube-api-access-kdt7g\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:42:58.827214 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.827209    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d1baecf-0643-40f9-bbae-882fae1c12a1-dshm\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:42:58.827214 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.827218    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d1baecf-0643-40f9-bbae-882fae1c12a1-tls-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:42:58.983212 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.983123    2555 generic.go:356] "Generic (PLEG): container finished" podID="0d1baecf-0643-40f9-bbae-882fae1c12a1" containerID="9f42d979ed3b6194cd971574dfca1620f7f2257c4ddd660505676f5be403eef0" exitCode=0
Sep 09 16:42:58.983212 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.983166    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"
Sep 09 16:42:58.983212 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.983174    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d" event={"ID":"0d1baecf-0643-40f9-bbae-882fae1c12a1","Type":"ContainerDied","Data":"9f42d979ed3b6194cd971574dfca1620f7f2257c4ddd660505676f5be403eef0"}
Sep 09 16:42:58.983212 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.983205    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d" event={"ID":"0d1baecf-0643-40f9-bbae-882fae1c12a1","Type":"ContainerDied","Data":"23445ed2169007e164f8875c18cd1accca01149176bea24bbfdbbed55abc379b"}
Sep 09 16:42:58.983538 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.983222    2555 scope.go:117] "RemoveContainer" containerID="9f42d979ed3b6194cd971574dfca1620f7f2257c4ddd660505676f5be403eef0"
Sep 09 16:42:58.993176 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.993157    2555 scope.go:117] "RemoveContainer" containerID="9f42d979ed3b6194cd971574dfca1620f7f2257c4ddd660505676f5be403eef0"
Sep 09 16:42:58.993494 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:42:58.993473    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f42d979ed3b6194cd971574dfca1620f7f2257c4ddd660505676f5be403eef0\": container with ID starting with 9f42d979ed3b6194cd971574dfca1620f7f2257c4ddd660505676f5be403eef0 not found: ID does not exist" containerID="9f42d979ed3b6194cd971574dfca1620f7f2257c4ddd660505676f5be403eef0"
Sep 09 16:42:58.993553 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:58.993501    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f42d979ed3b6194cd971574dfca1620f7f2257c4ddd660505676f5be403eef0"} err="failed to get container status \"9f42d979ed3b6194cd971574dfca1620f7f2257c4ddd660505676f5be403eef0\": rpc error: code = NotFound desc = could not find container \"9f42d979ed3b6194cd971574dfca1620f7f2257c4ddd660505676f5be403eef0\": container with ID starting with 9f42d979ed3b6194cd971574dfca1620f7f2257c4ddd660505676f5be403eef0 not found: ID does not exist"
Sep 09 16:42:59.005575 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.005552    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"]
Sep 09 16:42:59.009006 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.008983    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-331bf8/sub-test-model-331bf8-kserve-789b898c57-zwg6d"]
Sep 09 16:42:59.324072 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.324033    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4" podUID="6211c979-e5c2-4e15-b1a1-e0dadfad13be" containerName="main" containerID="cri-o://9cd46e8651c9805acef4225d9f4add6697a4fdc2b769c1c6a08dc15da1b48212" gracePeriod=45
Sep 09 16:42:59.593739 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.593668    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"]
Sep 09 16:42:59.594328 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.594311    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d1baecf-0643-40f9-bbae-882fae1c12a1" containerName="main"
Sep 09 16:42:59.594371 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.594330    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d1baecf-0643-40f9-bbae-882fae1c12a1" containerName="main"
Sep 09 16:42:59.594455 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.594444    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d1baecf-0643-40f9-bbae-882fae1c12a1" containerName="main"
Sep 09 16:42:59.597498 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.597476    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:42:59.598335 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.598287    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:42:59.601974 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.601955    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-331bf8\"/\"default-dockercfg-9jrw7\""
Sep 09 16:42:59.602417 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.601961    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-331bf8\"/\"kube-root-ca.crt\""
Sep 09 16:42:59.602417 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.601964    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-331bf8\"/\"openshift-service-ca.crt\""
Sep 09 16:42:59.602417 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.601964    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-331bf8\"/\"rate-test-model-331bf8-kserve-self-signed-certs\""
Sep 09 16:42:59.607641 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.607107    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"]
Sep 09 16:42:59.734133 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.734099    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6211c979-e5c2-4e15-b1a1-e0dadfad13be-dshm\") pod \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\" (UID: \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\") "
Sep 09 16:42:59.734338 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.734146    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6211c979-e5c2-4e15-b1a1-e0dadfad13be-model-cache\") pod \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\" (UID: \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\") "
Sep 09 16:42:59.734338 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.734163    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6211c979-e5c2-4e15-b1a1-e0dadfad13be-home\") pod \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\" (UID: \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\") "
Sep 09 16:42:59.734338 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.734218    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6211c979-e5c2-4e15-b1a1-e0dadfad13be-tls-certs\") pod \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\" (UID: \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\") "
Sep 09 16:42:59.734338 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.734254    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clcll\" (UniqueName: \"kubernetes.io/projected/6211c979-e5c2-4e15-b1a1-e0dadfad13be-kube-api-access-clcll\") pod \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\" (UID: \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\") "
Sep 09 16:42:59.734338 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.734281    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6211c979-e5c2-4e15-b1a1-e0dadfad13be-tmp-dir\") pod \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\" (UID: \"6211c979-e5c2-4e15-b1a1-e0dadfad13be\") "
Sep 09 16:42:59.734625 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.734482    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-home\") pod \"rate-test-model-331bf8-kserve-6577cf7bff-rwbgx\" (UID: \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:42:59.734625 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.734513    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6211c979-e5c2-4e15-b1a1-e0dadfad13be-home" (OuterVolumeSpecName: "home") pod "6211c979-e5c2-4e15-b1a1-e0dadfad13be" (UID: "6211c979-e5c2-4e15-b1a1-e0dadfad13be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:42:59.734625 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.734555    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-model-cache\") pod \"rate-test-model-331bf8-kserve-6577cf7bff-rwbgx\" (UID: \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:42:59.734625 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.734581    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6211c979-e5c2-4e15-b1a1-e0dadfad13be-model-cache" (OuterVolumeSpecName: "model-cache") pod "6211c979-e5c2-4e15-b1a1-e0dadfad13be" (UID: "6211c979-e5c2-4e15-b1a1-e0dadfad13be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:42:59.734625 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.734616    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-tls-certs\") pod \"rate-test-model-331bf8-kserve-6577cf7bff-rwbgx\" (UID: \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:42:59.734867 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.734719    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-tmp-dir\") pod \"rate-test-model-331bf8-kserve-6577cf7bff-rwbgx\" (UID: \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:42:59.734867 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.734743    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6211c979-e5c2-4e15-b1a1-e0dadfad13be-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6211c979-e5c2-4e15-b1a1-e0dadfad13be" (UID: "6211c979-e5c2-4e15-b1a1-e0dadfad13be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:42:59.734867 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.734770    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-dshm\") pod \"rate-test-model-331bf8-kserve-6577cf7bff-rwbgx\" (UID: \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:42:59.734867 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.734834    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhgqb\" (UniqueName: \"kubernetes.io/projected/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-kube-api-access-rhgqb\") pod \"rate-test-model-331bf8-kserve-6577cf7bff-rwbgx\" (UID: \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:42:59.735026 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.734887    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6211c979-e5c2-4e15-b1a1-e0dadfad13be-model-cache\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:42:59.735026 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.734902    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6211c979-e5c2-4e15-b1a1-e0dadfad13be-home\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:42:59.735026 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.734915    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6211c979-e5c2-4e15-b1a1-e0dadfad13be-tmp-dir\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:42:59.737008 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.736984    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6211c979-e5c2-4e15-b1a1-e0dadfad13be-dshm" (OuterVolumeSpecName: "dshm") pod "6211c979-e5c2-4e15-b1a1-e0dadfad13be" (UID: "6211c979-e5c2-4e15-b1a1-e0dadfad13be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:42:59.737093 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.737005    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6211c979-e5c2-4e15-b1a1-e0dadfad13be-kube-api-access-clcll" (OuterVolumeSpecName: "kube-api-access-clcll") pod "6211c979-e5c2-4e15-b1a1-e0dadfad13be" (UID: "6211c979-e5c2-4e15-b1a1-e0dadfad13be"). InnerVolumeSpecName "kube-api-access-clcll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:42:59.737093 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.737054    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6211c979-e5c2-4e15-b1a1-e0dadfad13be-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6211c979-e5c2-4e15-b1a1-e0dadfad13be" (UID: "6211c979-e5c2-4e15-b1a1-e0dadfad13be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:42:59.836361 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.836322    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-tmp-dir\") pod \"rate-test-model-331bf8-kserve-6577cf7bff-rwbgx\" (UID: \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:42:59.836556 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.836387    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-dshm\") pod \"rate-test-model-331bf8-kserve-6577cf7bff-rwbgx\" (UID: \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:42:59.836556 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.836440    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhgqb\" (UniqueName: \"kubernetes.io/projected/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-kube-api-access-rhgqb\") pod \"rate-test-model-331bf8-kserve-6577cf7bff-rwbgx\" (UID: \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:42:59.836556 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.836476    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-home\") pod \"rate-test-model-331bf8-kserve-6577cf7bff-rwbgx\" (UID: \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:42:59.836556 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.836527    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-model-cache\") pod \"rate-test-model-331bf8-kserve-6577cf7bff-rwbgx\" (UID: \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:42:59.836734 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.836569    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-tls-certs\") pod \"rate-test-model-331bf8-kserve-6577cf7bff-rwbgx\" (UID: \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:42:59.836734 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.836663    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6211c979-e5c2-4e15-b1a1-e0dadfad13be-dshm\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:42:59.836734 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.836678    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6211c979-e5c2-4e15-b1a1-e0dadfad13be-tls-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:42:59.836734 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.836692    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clcll\" (UniqueName: \"kubernetes.io/projected/6211c979-e5c2-4e15-b1a1-e0dadfad13be-kube-api-access-clcll\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:42:59.836876 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.836758    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-tmp-dir\") pod \"rate-test-model-331bf8-kserve-6577cf7bff-rwbgx\" (UID: \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:42:59.836911 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.836881    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-home\") pod \"rate-test-model-331bf8-kserve-6577cf7bff-rwbgx\" (UID: \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:42:59.836972 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.836956    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-model-cache\") pod \"rate-test-model-331bf8-kserve-6577cf7bff-rwbgx\" (UID: \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:42:59.838608 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.838590    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-dshm\") pod \"rate-test-model-331bf8-kserve-6577cf7bff-rwbgx\" (UID: \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:42:59.838885 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.838868    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-tls-certs\") pod \"rate-test-model-331bf8-kserve-6577cf7bff-rwbgx\" (UID: \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:42:59.844186 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.844113    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhgqb\" (UniqueName: \"kubernetes.io/projected/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-kube-api-access-rhgqb\") pod \"rate-test-model-331bf8-kserve-6577cf7bff-rwbgx\" (UID: \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\") " pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:42:59.910333 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.910276    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:42:59.996377 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.996346    2555 generic.go:356] "Generic (PLEG): container finished" podID="6211c979-e5c2-4e15-b1a1-e0dadfad13be" containerID="9cd46e8651c9805acef4225d9f4add6697a4fdc2b769c1c6a08dc15da1b48212" exitCode=0
Sep 09 16:42:59.996565 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.996404    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"
Sep 09 16:42:59.996662 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.996433    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4" event={"ID":"6211c979-e5c2-4e15-b1a1-e0dadfad13be","Type":"ContainerDied","Data":"9cd46e8651c9805acef4225d9f4add6697a4fdc2b769c1c6a08dc15da1b48212"}
Sep 09 16:42:59.996724 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.996675    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4" event={"ID":"6211c979-e5c2-4e15-b1a1-e0dadfad13be","Type":"ContainerDied","Data":"7ca421c70f3278c937614278bb6bdfc2e33782fa06c2226190cc5b2d244569ff"}
Sep 09 16:42:59.996724 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:42:59.996706    2555 scope.go:117] "RemoveContainer" containerID="9cd46e8651c9805acef4225d9f4add6697a4fdc2b769c1c6a08dc15da1b48212"
Sep 09 16:43:00.009455 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:00.009432    2555 scope.go:117] "RemoveContainer" containerID="9cd46e8651c9805acef4225d9f4add6697a4fdc2b769c1c6a08dc15da1b48212"
Sep 09 16:43:00.009734 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:43:00.009701    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cd46e8651c9805acef4225d9f4add6697a4fdc2b769c1c6a08dc15da1b48212\": container with ID starting with 9cd46e8651c9805acef4225d9f4add6697a4fdc2b769c1c6a08dc15da1b48212 not found: ID does not exist" containerID="9cd46e8651c9805acef4225d9f4add6697a4fdc2b769c1c6a08dc15da1b48212"
Sep 09 16:43:00.009801 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:00.009746    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cd46e8651c9805acef4225d9f4add6697a4fdc2b769c1c6a08dc15da1b48212"} err="failed to get container status \"9cd46e8651c9805acef4225d9f4add6697a4fdc2b769c1c6a08dc15da1b48212\": rpc error: code = NotFound desc = could not find container \"9cd46e8651c9805acef4225d9f4add6697a4fdc2b769c1c6a08dc15da1b48212\": container with ID starting with 9cd46e8651c9805acef4225d9f4add6697a4fdc2b769c1c6a08dc15da1b48212 not found: ID does not exist"
Sep 09 16:43:00.025139 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:00.025113    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"]
Sep 09 16:43:00.027561 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:00.027542    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-50fcbc/sub-test-model-50fcbc-kserve-7cf449988f-jqps4"]
Sep 09 16:43:00.187565 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:00.187539    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"]
Sep 09 16:43:00.188186 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:43:00.188166    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b3b9fba_4c3b_4ab2_b388_560dfd58e04b.slice/crio-17ae6f9f1bf649e12a9790d956a084b4b75f0459f7351e31ae36d6da5b0bf315 WatchSource:0}: Error finding container 17ae6f9f1bf649e12a9790d956a084b4b75f0459f7351e31ae36d6da5b0bf315: Status 404 returned error can't find the container with id 17ae6f9f1bf649e12a9790d956a084b4b75f0459f7351e31ae36d6da5b0bf315
Sep 09 16:43:00.609007 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:00.608970    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d1baecf-0643-40f9-bbae-882fae1c12a1" path="/var/lib/kubelet/pods/0d1baecf-0643-40f9-bbae-882fae1c12a1/volumes"
Sep 09 16:43:00.609415 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:00.609400    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6211c979-e5c2-4e15-b1a1-e0dadfad13be" path="/var/lib/kubelet/pods/6211c979-e5c2-4e15-b1a1-e0dadfad13be/volumes"
Sep 09 16:43:01.012853 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:01.012809    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx" event={"ID":"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b","Type":"ContainerStarted","Data":"2e406c35bb01b16716f3433457c45d2a8fd2a285ea43f8cb620181899365d649"}
Sep 09 16:43:01.013025 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:01.012863    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx" event={"ID":"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b","Type":"ContainerStarted","Data":"17ae6f9f1bf649e12a9790d956a084b4b75f0459f7351e31ae36d6da5b0bf315"}
Sep 09 16:43:01.032103 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:01.032050    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx" podStartSLOduration=2.03203592 podStartE2EDuration="2.03203592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:43:01.028977077 +0000 UTC m=+1746.859185132" watchObservedRunningTime="2026-09-09 16:43:01.03203592 +0000 UTC m=+1746.862243979"
Sep 09 16:43:06.095538 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:06.095494    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:43:06.973971 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:06.973943    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc"
Sep 09 16:43:09.910725 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:09.910691    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:43:09.913083 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:09.910742    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:43:09.923845 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:09.923818    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:43:10.082442 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:10.082410    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:43:39.365840 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:39.365804    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk"]
Sep 09 16:43:39.366419 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:39.366048    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk" podUID="2e9b4b90-4f61-4aa7-943e-f6601e24c61c" containerName="maas-api" containerID="cri-o://5c8a2844c99854479278924742f06a50bdeb35859c21a09f4155050965ab5b1e" gracePeriod=30
Sep 09 16:43:39.416304 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:39.416257    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz"]
Sep 09 16:43:39.416754 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:39.416703    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz" podUID="616c6d67-a842-4d6f-bcb3-efc0c73719fc" containerName="payload-processing" containerID="cri-o://e47ddb1ea52ded4e5199bda4605ae50aa3b84da826e40185b41f961bd361a710" gracePeriod=30
Sep 09 16:43:39.681041 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:39.681009    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk"
Sep 09 16:43:39.712153 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:39.712130    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz"
Sep 09 16:43:39.824573 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:39.824534    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e9b4b90-4f61-4aa7-943e-f6601e24c61c-metrics-certs\") pod \"2e9b4b90-4f61-4aa7-943e-f6601e24c61c\" (UID: \"2e9b4b90-4f61-4aa7-943e-f6601e24c61c\") "
Sep 09 16:43:39.824759 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:39.824596    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gjgn\" (UniqueName: \"kubernetes.io/projected/616c6d67-a842-4d6f-bcb3-efc0c73719fc-kube-api-access-9gjgn\") pod \"616c6d67-a842-4d6f-bcb3-efc0c73719fc\" (UID: \"616c6d67-a842-4d6f-bcb3-efc0c73719fc\") "
Sep 09 16:43:39.824759 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:39.824672    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e9b4b90-4f61-4aa7-943e-f6601e24c61c-maas-api-tls\") pod \"2e9b4b90-4f61-4aa7-943e-f6601e24c61c\" (UID: \"2e9b4b90-4f61-4aa7-943e-f6601e24c61c\") "
Sep 09 16:43:39.824759 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:39.824707    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v5dp\" (UniqueName: \"kubernetes.io/projected/2e9b4b90-4f61-4aa7-943e-f6601e24c61c-kube-api-access-6v5dp\") pod \"2e9b4b90-4f61-4aa7-943e-f6601e24c61c\" (UID: \"2e9b4b90-4f61-4aa7-943e-f6601e24c61c\") "
Sep 09 16:43:39.824759 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:39.824741    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9b4b90-4f61-4aa7-943e-f6601e24c61c-openshift-service-ca\") pod \"2e9b4b90-4f61-4aa7-943e-f6601e24c61c\" (UID: \"2e9b4b90-4f61-4aa7-943e-f6601e24c61c\") "
Sep 09 16:43:39.824986 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:39.824898    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616c6d67-a842-4d6f-bcb3-efc0c73719fc-plugins-config-volume\") pod \"616c6d67-a842-4d6f-bcb3-efc0c73719fc\" (UID: \"616c6d67-a842-4d6f-bcb3-efc0c73719fc\") "
Sep 09 16:43:39.825206 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:39.825180    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e9b4b90-4f61-4aa7-943e-f6601e24c61c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2e9b4b90-4f61-4aa7-943e-f6601e24c61c" (UID: "2e9b4b90-4f61-4aa7-943e-f6601e24c61c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:43:39.825342 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:39.825287    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/616c6d67-a842-4d6f-bcb3-efc0c73719fc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "616c6d67-a842-4d6f-bcb3-efc0c73719fc" (UID: "616c6d67-a842-4d6f-bcb3-efc0c73719fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:43:39.825445 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:39.825429    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9b4b90-4f61-4aa7-943e-f6601e24c61c-openshift-service-ca\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:43:39.826915 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:39.826887    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e9b4b90-4f61-4aa7-943e-f6601e24c61c-kube-api-access-6v5dp" (OuterVolumeSpecName: "kube-api-access-6v5dp") pod "2e9b4b90-4f61-4aa7-943e-f6601e24c61c" (UID: "2e9b4b90-4f61-4aa7-943e-f6601e24c61c"). InnerVolumeSpecName "kube-api-access-6v5dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:43:39.826980 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:39.826899    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616c6d67-a842-4d6f-bcb3-efc0c73719fc-kube-api-access-9gjgn" (OuterVolumeSpecName: "kube-api-access-9gjgn") pod "616c6d67-a842-4d6f-bcb3-efc0c73719fc" (UID: "616c6d67-a842-4d6f-bcb3-efc0c73719fc"). InnerVolumeSpecName "kube-api-access-9gjgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:43:39.827176 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:39.827155    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e9b4b90-4f61-4aa7-943e-f6601e24c61c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2e9b4b90-4f61-4aa7-943e-f6601e24c61c" (UID: "2e9b4b90-4f61-4aa7-943e-f6601e24c61c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:43:39.827176 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:39.827172    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e9b4b90-4f61-4aa7-943e-f6601e24c61c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2e9b4b90-4f61-4aa7-943e-f6601e24c61c" (UID: "2e9b4b90-4f61-4aa7-943e-f6601e24c61c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:43:39.926560 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:39.926481    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gjgn\" (UniqueName: \"kubernetes.io/projected/616c6d67-a842-4d6f-bcb3-efc0c73719fc-kube-api-access-9gjgn\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:43:39.926560 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:39.926512    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e9b4b90-4f61-4aa7-943e-f6601e24c61c-maas-api-tls\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:43:39.926560 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:39.926522    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v5dp\" (UniqueName: \"kubernetes.io/projected/2e9b4b90-4f61-4aa7-943e-f6601e24c61c-kube-api-access-6v5dp\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:43:39.926560 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:39.926530    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616c6d67-a842-4d6f-bcb3-efc0c73719fc-plugins-config-volume\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:43:39.926560 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:39.926539    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e9b4b90-4f61-4aa7-943e-f6601e24c61c-metrics-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:43:40.254963 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:40.254928    2555 generic.go:356] "Generic (PLEG): container finished" podID="616c6d67-a842-4d6f-bcb3-efc0c73719fc" containerID="e47ddb1ea52ded4e5199bda4605ae50aa3b84da826e40185b41f961bd361a710" exitCode=0
Sep 09 16:43:40.255148 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:40.254976    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz"
Sep 09 16:43:40.255148 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:40.255021    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz" event={"ID":"616c6d67-a842-4d6f-bcb3-efc0c73719fc","Type":"ContainerDied","Data":"e47ddb1ea52ded4e5199bda4605ae50aa3b84da826e40185b41f961bd361a710"}
Sep 09 16:43:40.255148 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:40.255077    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz" event={"ID":"616c6d67-a842-4d6f-bcb3-efc0c73719fc","Type":"ContainerDied","Data":"294c9e1b81d806591b9a365377807121de3f20372d80de5e668330b3769aa9bb"}
Sep 09 16:43:40.255148 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:40.255108    2555 scope.go:117] "RemoveContainer" containerID="e47ddb1ea52ded4e5199bda4605ae50aa3b84da826e40185b41f961bd361a710"
Sep 09 16:43:40.256437 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:40.256410    2555 generic.go:356] "Generic (PLEG): container finished" podID="2e9b4b90-4f61-4aa7-943e-f6601e24c61c" containerID="5c8a2844c99854479278924742f06a50bdeb35859c21a09f4155050965ab5b1e" exitCode=0
Sep 09 16:43:40.256538 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:40.256453    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk"
Sep 09 16:43:40.256538 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:40.256499    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk" event={"ID":"2e9b4b90-4f61-4aa7-943e-f6601e24c61c","Type":"ContainerDied","Data":"5c8a2844c99854479278924742f06a50bdeb35859c21a09f4155050965ab5b1e"}
Sep 09 16:43:40.256653 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:40.256547    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk" event={"ID":"2e9b4b90-4f61-4aa7-943e-f6601e24c61c","Type":"ContainerDied","Data":"1b4b9f89430a2c030b6fcf5548b0f8a6096b3cafe5f7507646841972b7460b7a"}
Sep 09 16:43:40.266021 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:40.266006    2555 scope.go:117] "RemoveContainer" containerID="e47ddb1ea52ded4e5199bda4605ae50aa3b84da826e40185b41f961bd361a710"
Sep 09 16:43:40.266381 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:43:40.266352    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e47ddb1ea52ded4e5199bda4605ae50aa3b84da826e40185b41f961bd361a710\": container with ID starting with e47ddb1ea52ded4e5199bda4605ae50aa3b84da826e40185b41f961bd361a710 not found: ID does not exist" containerID="e47ddb1ea52ded4e5199bda4605ae50aa3b84da826e40185b41f961bd361a710"
Sep 09 16:43:40.266479 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:40.266389    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e47ddb1ea52ded4e5199bda4605ae50aa3b84da826e40185b41f961bd361a710"} err="failed to get container status \"e47ddb1ea52ded4e5199bda4605ae50aa3b84da826e40185b41f961bd361a710\": rpc error: code = NotFound desc = could not find container \"e47ddb1ea52ded4e5199bda4605ae50aa3b84da826e40185b41f961bd361a710\": container with ID starting with e47ddb1ea52ded4e5199bda4605ae50aa3b84da826e40185b41f961bd361a710 not found: ID does not exist"
Sep 09 16:43:40.266479 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:40.266413    2555 scope.go:117] "RemoveContainer" containerID="5c8a2844c99854479278924742f06a50bdeb35859c21a09f4155050965ab5b1e"
Sep 09 16:43:40.275168 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:40.275150    2555 scope.go:117] "RemoveContainer" containerID="5c8a2844c99854479278924742f06a50bdeb35859c21a09f4155050965ab5b1e"
Sep 09 16:43:40.275441 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:43:40.275423    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c8a2844c99854479278924742f06a50bdeb35859c21a09f4155050965ab5b1e\": container with ID starting with 5c8a2844c99854479278924742f06a50bdeb35859c21a09f4155050965ab5b1e not found: ID does not exist" containerID="5c8a2844c99854479278924742f06a50bdeb35859c21a09f4155050965ab5b1e"
Sep 09 16:43:40.275498 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:40.275447    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c8a2844c99854479278924742f06a50bdeb35859c21a09f4155050965ab5b1e"} err="failed to get container status \"5c8a2844c99854479278924742f06a50bdeb35859c21a09f4155050965ab5b1e\": rpc error: code = NotFound desc = could not find container \"5c8a2844c99854479278924742f06a50bdeb35859c21a09f4155050965ab5b1e\": container with ID starting with 5c8a2844c99854479278924742f06a50bdeb35859c21a09f4155050965ab5b1e not found: ID does not exist"
Sep 09 16:43:40.280896 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:40.280874    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk"]
Sep 09 16:43:40.286235 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:40.286211    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-14e637-7dbf87bd47-5jxpk"]
Sep 09 16:43:40.295580 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:40.295552    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz"]
Sep 09 16:43:40.299277 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:40.299251    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-14e637-5b7cbc4cc9-r97fz"]
Sep 09 16:43:40.609748 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:40.609673    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e9b4b90-4f61-4aa7-943e-f6601e24c61c" path="/var/lib/kubelet/pods/2e9b4b90-4f61-4aa7-943e-f6601e24c61c/volumes"
Sep 09 16:43:40.610103 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:40.610069    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="616c6d67-a842-4d6f-bcb3-efc0c73719fc" path="/var/lib/kubelet/pods/616c6d67-a842-4d6f-bcb3-efc0c73719fc/volumes"
Sep 09 16:43:42.698962 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:42.698928    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:43:47.691724 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.691647    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"]
Sep 09 16:43:47.692100 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.692084    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6211c979-e5c2-4e15-b1a1-e0dadfad13be" containerName="main"
Sep 09 16:43:47.692144 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.692103    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6211c979-e5c2-4e15-b1a1-e0dadfad13be" containerName="main"
Sep 09 16:43:47.692144 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.692112    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e9b4b90-4f61-4aa7-943e-f6601e24c61c" containerName="maas-api"
Sep 09 16:43:47.692144 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.692117    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e9b4b90-4f61-4aa7-943e-f6601e24c61c" containerName="maas-api"
Sep 09 16:43:47.692144 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.692134    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="616c6d67-a842-4d6f-bcb3-efc0c73719fc" containerName="payload-processing"
Sep 09 16:43:47.692144 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.692140    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="616c6d67-a842-4d6f-bcb3-efc0c73719fc" containerName="payload-processing"
Sep 09 16:43:47.692348 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.692224    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="616c6d67-a842-4d6f-bcb3-efc0c73719fc" containerName="payload-processing"
Sep 09 16:43:47.692348 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.692233    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e9b4b90-4f61-4aa7-943e-f6601e24c61c" containerName="maas-api"
Sep 09 16:43:47.692348 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.692242    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6211c979-e5c2-4e15-b1a1-e0dadfad13be" containerName="main"
Sep 09 16:43:47.696785 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.696767    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:43:47.699416 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.699394    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-50fcbc\"/\"kube-root-ca.crt\""
Sep 09 16:43:47.700746 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.700724    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-50fcbc\"/\"openshift-service-ca.crt\""
Sep 09 16:43:47.700952 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.700932    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-50fcbc\"/\"default-dockercfg-qltth\""
Sep 09 16:43:47.701092 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.700979    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-50fcbc\"/\"rate-test-model-50fcbc-kserve-self-signed-certs\""
Sep 09 16:43:47.705182 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.705157    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"]
Sep 09 16:43:47.809100 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.809060    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtdvv\" (UniqueName: \"kubernetes.io/projected/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-kube-api-access-wtdvv\") pod \"rate-test-model-50fcbc-kserve-59d7d9ccf-5275x\" (UID: \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:43:47.809265 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.809130    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-model-cache\") pod \"rate-test-model-50fcbc-kserve-59d7d9ccf-5275x\" (UID: \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:43:47.809265 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.809182    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-tmp-dir\") pod \"rate-test-model-50fcbc-kserve-59d7d9ccf-5275x\" (UID: \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:43:47.809265 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.809248    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-home\") pod \"rate-test-model-50fcbc-kserve-59d7d9ccf-5275x\" (UID: \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:43:47.809485 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.809330    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-tls-certs\") pod \"rate-test-model-50fcbc-kserve-59d7d9ccf-5275x\" (UID: \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:43:47.809485 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.809364    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-dshm\") pod \"rate-test-model-50fcbc-kserve-59d7d9ccf-5275x\" (UID: \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:43:47.910662 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.910620    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-tls-certs\") pod \"rate-test-model-50fcbc-kserve-59d7d9ccf-5275x\" (UID: \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:43:47.910937 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.910916    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-dshm\") pod \"rate-test-model-50fcbc-kserve-59d7d9ccf-5275x\" (UID: \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:43:47.911127 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.911111    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtdvv\" (UniqueName: \"kubernetes.io/projected/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-kube-api-access-wtdvv\") pod \"rate-test-model-50fcbc-kserve-59d7d9ccf-5275x\" (UID: \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:43:47.911244 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.911230    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-model-cache\") pod \"rate-test-model-50fcbc-kserve-59d7d9ccf-5275x\" (UID: \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:43:47.911403 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.911388    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-tmp-dir\") pod \"rate-test-model-50fcbc-kserve-59d7d9ccf-5275x\" (UID: \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:43:47.911568 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.911549    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-home\") pod \"rate-test-model-50fcbc-kserve-59d7d9ccf-5275x\" (UID: \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:43:47.912168 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.912141    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-model-cache\") pod \"rate-test-model-50fcbc-kserve-59d7d9ccf-5275x\" (UID: \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:43:47.912168 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.912156    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-tmp-dir\") pod \"rate-test-model-50fcbc-kserve-59d7d9ccf-5275x\" (UID: \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:43:47.912524 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.912499    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-home\") pod \"rate-test-model-50fcbc-kserve-59d7d9ccf-5275x\" (UID: \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:43:47.914349 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.914321    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-tls-certs\") pod \"rate-test-model-50fcbc-kserve-59d7d9ccf-5275x\" (UID: \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:43:47.914439 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.914404    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-dshm\") pod \"rate-test-model-50fcbc-kserve-59d7d9ccf-5275x\" (UID: \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:43:47.919499 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:47.919478    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtdvv\" (UniqueName: \"kubernetes.io/projected/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-kube-api-access-wtdvv\") pod \"rate-test-model-50fcbc-kserve-59d7d9ccf-5275x\" (UID: \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:43:48.007552 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:48.007523    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:43:48.282641 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:48.282613    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"]
Sep 09 16:43:48.284261 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:43:48.284234    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1ec7651_c7b5_41c4_b3cd_5f8fce1bcdae.slice/crio-debc76f8f2b838ed179e46018a57c5a34be4eb66e453d523794e651f3abbe790 WatchSource:0}: Error finding container debc76f8f2b838ed179e46018a57c5a34be4eb66e453d523794e651f3abbe790: Status 404 returned error can't find the container with id debc76f8f2b838ed179e46018a57c5a34be4eb66e453d523794e651f3abbe790
Sep 09 16:43:48.309525 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:48.309497    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x" event={"ID":"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae","Type":"ContainerStarted","Data":"debc76f8f2b838ed179e46018a57c5a34be4eb66e453d523794e651f3abbe790"}
Sep 09 16:43:49.317273 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:49.317240    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x" event={"ID":"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae","Type":"ContainerStarted","Data":"381d3b3635911782285339e023973ab93dd69029a187fb31c0df4dd3fee14aa9"}
Sep 09 16:43:49.333924 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:49.333875    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x" podStartSLOduration=2.3338624709999998 podStartE2EDuration="2.333862471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:43:49.332491885 +0000 UTC m=+1795.162699948" watchObservedRunningTime="2026-09-09 16:43:49.333862471 +0000 UTC m=+1795.164070578"
Sep 09 16:43:54.714132 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:54.714101    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgcjl_561d9dd7-04c3-42dc-bcf6-2c92fd61706b/console-operator/1.log"
Sep 09 16:43:54.722231 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:54.722207    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgcjl_561d9dd7-04c3-42dc-bcf6-2c92fd61706b/console-operator/1.log"
Sep 09 16:43:58.008491 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:58.008455    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:43:58.008885 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:58.008504    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:43:58.021759 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:58.021729    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:43:58.381211 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:58.381123    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:43:59.890581 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:43:59.890538    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:44:01.678458 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:01.678427    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc"]
Sep 09 16:44:01.678904 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:01.678674    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc" podUID="fa0e6ab2-4fbd-4669-8d3c-38191b9e756f" containerName="payload-pre-processing" containerID="cri-o://62ce16fd41e1ee769c1ac9d19478cf2fa2532c3b473108b4de569e0072ebe597" gracePeriod=30
Sep 09 16:44:01.946841 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:01.946768    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc"
Sep 09 16:44:02.045737 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:02.045699    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa0e6ab2-4fbd-4669-8d3c-38191b9e756f-plugins-config-volume\") pod \"fa0e6ab2-4fbd-4669-8d3c-38191b9e756f\" (UID: \"fa0e6ab2-4fbd-4669-8d3c-38191b9e756f\") "
Sep 09 16:44:02.045911 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:02.045800    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jh8x5\" (UniqueName: \"kubernetes.io/projected/fa0e6ab2-4fbd-4669-8d3c-38191b9e756f-kube-api-access-jh8x5\") pod \"fa0e6ab2-4fbd-4669-8d3c-38191b9e756f\" (UID: \"fa0e6ab2-4fbd-4669-8d3c-38191b9e756f\") "
Sep 09 16:44:02.046190 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:02.046166    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0e6ab2-4fbd-4669-8d3c-38191b9e756f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fa0e6ab2-4fbd-4669-8d3c-38191b9e756f" (UID: "fa0e6ab2-4fbd-4669-8d3c-38191b9e756f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:44:02.047924 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:02.047900    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0e6ab2-4fbd-4669-8d3c-38191b9e756f-kube-api-access-jh8x5" (OuterVolumeSpecName: "kube-api-access-jh8x5") pod "fa0e6ab2-4fbd-4669-8d3c-38191b9e756f" (UID: "fa0e6ab2-4fbd-4669-8d3c-38191b9e756f"). InnerVolumeSpecName "kube-api-access-jh8x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:44:02.147376 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:02.147335    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh8x5\" (UniqueName: \"kubernetes.io/projected/fa0e6ab2-4fbd-4669-8d3c-38191b9e756f-kube-api-access-jh8x5\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:02.147376 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:02.147367    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa0e6ab2-4fbd-4669-8d3c-38191b9e756f-plugins-config-volume\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:02.393620 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:02.393586    2555 generic.go:356] "Generic (PLEG): container finished" podID="fa0e6ab2-4fbd-4669-8d3c-38191b9e756f" containerID="62ce16fd41e1ee769c1ac9d19478cf2fa2532c3b473108b4de569e0072ebe597" exitCode=0
Sep 09 16:44:02.393819 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:02.393630    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc"
Sep 09 16:44:02.393819 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:02.393669    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc" event={"ID":"fa0e6ab2-4fbd-4669-8d3c-38191b9e756f","Type":"ContainerDied","Data":"62ce16fd41e1ee769c1ac9d19478cf2fa2532c3b473108b4de569e0072ebe597"}
Sep 09 16:44:02.393819 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:02.393735    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc" event={"ID":"fa0e6ab2-4fbd-4669-8d3c-38191b9e756f","Type":"ContainerDied","Data":"d9d119a631ca997fe5161e4a958f2eae93bd4663f9623104d585725154dc5731"}
Sep 09 16:44:02.393819 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:02.393750    2555 scope.go:117] "RemoveContainer" containerID="62ce16fd41e1ee769c1ac9d19478cf2fa2532c3b473108b4de569e0072ebe597"
Sep 09 16:44:02.403118 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:02.403100    2555 scope.go:117] "RemoveContainer" containerID="62ce16fd41e1ee769c1ac9d19478cf2fa2532c3b473108b4de569e0072ebe597"
Sep 09 16:44:02.403400 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:44:02.403383    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ce16fd41e1ee769c1ac9d19478cf2fa2532c3b473108b4de569e0072ebe597\": container with ID starting with 62ce16fd41e1ee769c1ac9d19478cf2fa2532c3b473108b4de569e0072ebe597 not found: ID does not exist" containerID="62ce16fd41e1ee769c1ac9d19478cf2fa2532c3b473108b4de569e0072ebe597"
Sep 09 16:44:02.403454 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:02.403409    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ce16fd41e1ee769c1ac9d19478cf2fa2532c3b473108b4de569e0072ebe597"} err="failed to get container status \"62ce16fd41e1ee769c1ac9d19478cf2fa2532c3b473108b4de569e0072ebe597\": rpc error: code = NotFound desc = could not find container \"62ce16fd41e1ee769c1ac9d19478cf2fa2532c3b473108b4de569e0072ebe597\": container with ID starting with 62ce16fd41e1ee769c1ac9d19478cf2fa2532c3b473108b4de569e0072ebe597 not found: ID does not exist"
Sep 09 16:44:02.415473 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:02.415449    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc"]
Sep 09 16:44:02.419338 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:02.419319    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8cb67b-5d7c949bf4-x74fc"]
Sep 09 16:44:02.609860 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:02.609820    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa0e6ab2-4fbd-4669-8d3c-38191b9e756f" path="/var/lib/kubelet/pods/fa0e6ab2-4fbd-4669-8d3c-38191b9e756f/volumes"
Sep 09 16:44:16.092774 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:16.092734    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:44:19.492187 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:19.492150    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28"]
Sep 09 16:44:19.492716 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:19.492448    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28" podUID="f63431c3-a6e0-4c22-b7b5-2facc77d6e63" containerName="maas-api" containerID="cri-o://21781cf50affdc305e5809277be911f11a6361959923d585352146fcb09ae4c1" gracePeriod=30
Sep 09 16:44:19.734326 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:19.734287    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28"
Sep 09 16:44:19.817170 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:19.817069    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f63431c3-a6e0-4c22-b7b5-2facc77d6e63-openshift-service-ca\") pod \"f63431c3-a6e0-4c22-b7b5-2facc77d6e63\" (UID: \"f63431c3-a6e0-4c22-b7b5-2facc77d6e63\") "
Sep 09 16:44:19.817170 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:19.817138    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9jms\" (UniqueName: \"kubernetes.io/projected/f63431c3-a6e0-4c22-b7b5-2facc77d6e63-kube-api-access-p9jms\") pod \"f63431c3-a6e0-4c22-b7b5-2facc77d6e63\" (UID: \"f63431c3-a6e0-4c22-b7b5-2facc77d6e63\") "
Sep 09 16:44:19.817483 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:19.817251    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f63431c3-a6e0-4c22-b7b5-2facc77d6e63-maas-api-tls\") pod \"f63431c3-a6e0-4c22-b7b5-2facc77d6e63\" (UID: \"f63431c3-a6e0-4c22-b7b5-2facc77d6e63\") "
Sep 09 16:44:19.817483 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:19.817316    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f63431c3-a6e0-4c22-b7b5-2facc77d6e63-metrics-certs\") pod \"f63431c3-a6e0-4c22-b7b5-2facc77d6e63\" (UID: \"f63431c3-a6e0-4c22-b7b5-2facc77d6e63\") "
Sep 09 16:44:19.817623 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:19.817478    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f63431c3-a6e0-4c22-b7b5-2facc77d6e63-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f63431c3-a6e0-4c22-b7b5-2facc77d6e63" (UID: "f63431c3-a6e0-4c22-b7b5-2facc77d6e63"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:44:19.817730 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:19.817708    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f63431c3-a6e0-4c22-b7b5-2facc77d6e63-openshift-service-ca\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:19.819392 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:19.819365    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f63431c3-a6e0-4c22-b7b5-2facc77d6e63-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f63431c3-a6e0-4c22-b7b5-2facc77d6e63" (UID: "f63431c3-a6e0-4c22-b7b5-2facc77d6e63"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:44:19.819497 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:19.819390    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f63431c3-a6e0-4c22-b7b5-2facc77d6e63-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f63431c3-a6e0-4c22-b7b5-2facc77d6e63" (UID: "f63431c3-a6e0-4c22-b7b5-2facc77d6e63"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:44:19.819497 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:19.819450    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f63431c3-a6e0-4c22-b7b5-2facc77d6e63-kube-api-access-p9jms" (OuterVolumeSpecName: "kube-api-access-p9jms") pod "f63431c3-a6e0-4c22-b7b5-2facc77d6e63" (UID: "f63431c3-a6e0-4c22-b7b5-2facc77d6e63"). InnerVolumeSpecName "kube-api-access-p9jms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:44:19.918701 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:19.918661    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9jms\" (UniqueName: \"kubernetes.io/projected/f63431c3-a6e0-4c22-b7b5-2facc77d6e63-kube-api-access-p9jms\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:19.918701 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:19.918702    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f63431c3-a6e0-4c22-b7b5-2facc77d6e63-maas-api-tls\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:19.918970 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:19.918720    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f63431c3-a6e0-4c22-b7b5-2facc77d6e63-metrics-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:20.516707 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:20.516667    2555 generic.go:356] "Generic (PLEG): container finished" podID="f63431c3-a6e0-4c22-b7b5-2facc77d6e63" containerID="21781cf50affdc305e5809277be911f11a6361959923d585352146fcb09ae4c1" exitCode=0
Sep 09 16:44:20.517111 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:20.516731    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28"
Sep 09 16:44:20.517111 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:20.516753    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28" event={"ID":"f63431c3-a6e0-4c22-b7b5-2facc77d6e63","Type":"ContainerDied","Data":"21781cf50affdc305e5809277be911f11a6361959923d585352146fcb09ae4c1"}
Sep 09 16:44:20.517111 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:20.516832    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28" event={"ID":"f63431c3-a6e0-4c22-b7b5-2facc77d6e63","Type":"ContainerDied","Data":"036c1df7d13119da6f69b491dd5dcd8ac02d8a51c2af8d1a3416c916336ca967"}
Sep 09 16:44:20.517111 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:20.516857    2555 scope.go:117] "RemoveContainer" containerID="21781cf50affdc305e5809277be911f11a6361959923d585352146fcb09ae4c1"
Sep 09 16:44:20.527206 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:20.527188    2555 scope.go:117] "RemoveContainer" containerID="21781cf50affdc305e5809277be911f11a6361959923d585352146fcb09ae4c1"
Sep 09 16:44:20.527490 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:44:20.527469    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21781cf50affdc305e5809277be911f11a6361959923d585352146fcb09ae4c1\": container with ID starting with 21781cf50affdc305e5809277be911f11a6361959923d585352146fcb09ae4c1 not found: ID does not exist" containerID="21781cf50affdc305e5809277be911f11a6361959923d585352146fcb09ae4c1"
Sep 09 16:44:20.527537 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:20.527500    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21781cf50affdc305e5809277be911f11a6361959923d585352146fcb09ae4c1"} err="failed to get container status \"21781cf50affdc305e5809277be911f11a6361959923d585352146fcb09ae4c1\": rpc error: code = NotFound desc = could not find container \"21781cf50affdc305e5809277be911f11a6361959923d585352146fcb09ae4c1\": container with ID starting with 21781cf50affdc305e5809277be911f11a6361959923d585352146fcb09ae4c1 not found: ID does not exist"
Sep 09 16:44:20.539579 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:20.539558    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28"]
Sep 09 16:44:20.547203 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:20.547179    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2bdf45-74b4d9dfd5-xgf28"]
Sep 09 16:44:20.609212 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:20.609184    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f63431c3-a6e0-4c22-b7b5-2facc77d6e63" path="/var/lib/kubelet/pods/f63431c3-a6e0-4c22-b7b5-2facc77d6e63/volumes"
Sep 09 16:44:26.025353 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:26.025320    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"]
Sep 09 16:44:26.025824 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:26.025566    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p" podUID="eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" containerName="istio-proxy" containerID="cri-o://bee8fe89f7d4a9e87955522a9eb1906774b710f7a73e02cc1920bf71043ed003" gracePeriod=30
Sep 09 16:44:29.389390 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:29.389356    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:44:30.134785 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:30.134752    2555 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 16:44:30.134978 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:30.134821    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p" podUID="eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 16:44:31.288660 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.288595    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:44:31.326592 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.326562    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-workload-socket\") pod \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") "
Sep 09 16:44:31.326761 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.326606    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istio-token\") pod \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") "
Sep 09 16:44:31.326761 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.326638    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-workload-certs\") pod \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") "
Sep 09 16:44:31.326761 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.326755    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istio-podinfo\") pod \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") "
Sep 09 16:44:31.326918 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.326791    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-credential-socket\") pod \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") "
Sep 09 16:44:31.326918 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.326813    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istio-data\") pod \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") "
Sep 09 16:44:31.326918 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.326839    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istiod-ca-cert\") pod \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") "
Sep 09 16:44:31.326918 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.326850    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" (UID: "eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:44:31.326918 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.326871    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istio-envoy\") pod \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") "
Sep 09 16:44:31.326918 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.326915    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96rds\" (UniqueName: \"kubernetes.io/projected/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-kube-api-access-96rds\") pod \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\" (UID: \"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a\") "
Sep 09 16:44:31.327234 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.326917    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" (UID: "eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:44:31.327234 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.327152    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istio-data" (OuterVolumeSpecName: "istio-data") pod "eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" (UID: "eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:44:31.327383 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.327280    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" (UID: "eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:44:31.327383 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.327350    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istio-data\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:31.327383 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.327359    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" (UID: "eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:44:31.327383 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.327370    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-workload-socket\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:31.327591 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.327385    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-workload-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:31.329230 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.329198    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" (UID: "eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 16:44:31.329230 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.329208    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-kube-api-access-96rds" (OuterVolumeSpecName: "kube-api-access-96rds") pod "eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" (UID: "eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a"). InnerVolumeSpecName "kube-api-access-96rds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:44:31.329428 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.329382    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" (UID: "eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:44:31.329491 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.329450    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istio-token" (OuterVolumeSpecName: "istio-token") pod "eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" (UID: "eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:44:31.428003 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.427971    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istio-token\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:31.428003 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.427997    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istio-podinfo\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:31.428003 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.428005    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-credential-socket\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:31.428222 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.428015    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istiod-ca-cert\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:31.428222 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.428023    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-istio-envoy\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:31.428222 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.428031    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96rds\" (UniqueName: \"kubernetes.io/projected/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a-kube-api-access-96rds\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:31.589096 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.589016    2555 generic.go:356] "Generic (PLEG): container finished" podID="eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" containerID="bee8fe89f7d4a9e87955522a9eb1906774b710f7a73e02cc1920bf71043ed003" exitCode=0
Sep 09 16:44:31.589096 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.589066    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"
Sep 09 16:44:31.589275 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.589096    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p" event={"ID":"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a","Type":"ContainerDied","Data":"bee8fe89f7d4a9e87955522a9eb1906774b710f7a73e02cc1920bf71043ed003"}
Sep 09 16:44:31.589275 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.589147    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p" event={"ID":"eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a","Type":"ContainerDied","Data":"7c6035acafcbbc984eb258552825ad454d9656cd15978f76c646fdc42846b92c"}
Sep 09 16:44:31.589275 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.589169    2555 scope.go:117] "RemoveContainer" containerID="bee8fe89f7d4a9e87955522a9eb1906774b710f7a73e02cc1920bf71043ed003"
Sep 09 16:44:31.599288 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.599272    2555 scope.go:117] "RemoveContainer" containerID="ea4fb7d9f27929664db7a48562105da8c65654d0c1510f6f41ceba0ed5efdfd3"
Sep 09 16:44:31.607587 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.607540    2555 scope.go:117] "RemoveContainer" containerID="bee8fe89f7d4a9e87955522a9eb1906774b710f7a73e02cc1920bf71043ed003"
Sep 09 16:44:31.607949 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:44:31.607923    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bee8fe89f7d4a9e87955522a9eb1906774b710f7a73e02cc1920bf71043ed003\": container with ID starting with bee8fe89f7d4a9e87955522a9eb1906774b710f7a73e02cc1920bf71043ed003 not found: ID does not exist" containerID="bee8fe89f7d4a9e87955522a9eb1906774b710f7a73e02cc1920bf71043ed003"
Sep 09 16:44:31.608035 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.607961    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bee8fe89f7d4a9e87955522a9eb1906774b710f7a73e02cc1920bf71043ed003"} err="failed to get container status \"bee8fe89f7d4a9e87955522a9eb1906774b710f7a73e02cc1920bf71043ed003\": rpc error: code = NotFound desc = could not find container \"bee8fe89f7d4a9e87955522a9eb1906774b710f7a73e02cc1920bf71043ed003\": container with ID starting with bee8fe89f7d4a9e87955522a9eb1906774b710f7a73e02cc1920bf71043ed003 not found: ID does not exist"
Sep 09 16:44:31.608035 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.607984    2555 scope.go:117] "RemoveContainer" containerID="ea4fb7d9f27929664db7a48562105da8c65654d0c1510f6f41ceba0ed5efdfd3"
Sep 09 16:44:31.608266 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:44:31.608245    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea4fb7d9f27929664db7a48562105da8c65654d0c1510f6f41ceba0ed5efdfd3\": container with ID starting with ea4fb7d9f27929664db7a48562105da8c65654d0c1510f6f41ceba0ed5efdfd3 not found: ID does not exist" containerID="ea4fb7d9f27929664db7a48562105da8c65654d0c1510f6f41ceba0ed5efdfd3"
Sep 09 16:44:31.608326 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.608277    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea4fb7d9f27929664db7a48562105da8c65654d0c1510f6f41ceba0ed5efdfd3"} err="failed to get container status \"ea4fb7d9f27929664db7a48562105da8c65654d0c1510f6f41ceba0ed5efdfd3\": rpc error: code = NotFound desc = could not find container \"ea4fb7d9f27929664db7a48562105da8c65654d0c1510f6f41ceba0ed5efdfd3\": container with ID starting with ea4fb7d9f27929664db7a48562105da8c65654d0c1510f6f41ceba0ed5efdfd3 not found: ID does not exist"
Sep 09 16:44:31.611108 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.611090    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"]
Sep 09 16:44:31.614925 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:31.614908    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2bdf45-openshift-default-676c964b5b-txc9p"]
Sep 09 16:44:32.607731 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:32.607697    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" path="/var/lib/kubelet/pods/eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a/volumes"
Sep 09 16:44:33.690634 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:33.690602    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"]
Sep 09 16:44:34.782578 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:34.782549    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"]
Sep 09 16:44:36.862071 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:36.862037    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d"]
Sep 09 16:44:36.862533 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:36.862261    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d" podUID="258b7a1e-a5b1-4041-9e22-4d618e367bce" containerName="payload-pre-processing" containerID="cri-o://20797f56bdd8d0060673559eac370b1774a0b25db5e80dbe351b8cec3010f5e8" gracePeriod=30
Sep 09 16:44:37.131048 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:37.131027    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d"
Sep 09 16:44:37.187182 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:37.187152    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/258b7a1e-a5b1-4041-9e22-4d618e367bce-plugins-config-volume\") pod \"258b7a1e-a5b1-4041-9e22-4d618e367bce\" (UID: \"258b7a1e-a5b1-4041-9e22-4d618e367bce\") "
Sep 09 16:44:37.187387 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:37.187206    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwhrz\" (UniqueName: \"kubernetes.io/projected/258b7a1e-a5b1-4041-9e22-4d618e367bce-kube-api-access-qwhrz\") pod \"258b7a1e-a5b1-4041-9e22-4d618e367bce\" (UID: \"258b7a1e-a5b1-4041-9e22-4d618e367bce\") "
Sep 09 16:44:37.187574 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:37.187552    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/258b7a1e-a5b1-4041-9e22-4d618e367bce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "258b7a1e-a5b1-4041-9e22-4d618e367bce" (UID: "258b7a1e-a5b1-4041-9e22-4d618e367bce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:44:37.189244 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:37.189217    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/258b7a1e-a5b1-4041-9e22-4d618e367bce-kube-api-access-qwhrz" (OuterVolumeSpecName: "kube-api-access-qwhrz") pod "258b7a1e-a5b1-4041-9e22-4d618e367bce" (UID: "258b7a1e-a5b1-4041-9e22-4d618e367bce"). InnerVolumeSpecName "kube-api-access-qwhrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:44:37.288390 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:37.288355    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwhrz\" (UniqueName: \"kubernetes.io/projected/258b7a1e-a5b1-4041-9e22-4d618e367bce-kube-api-access-qwhrz\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:37.288390 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:37.288392    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/258b7a1e-a5b1-4041-9e22-4d618e367bce-plugins-config-volume\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:37.626127 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:37.626093    2555 generic.go:356] "Generic (PLEG): container finished" podID="258b7a1e-a5b1-4041-9e22-4d618e367bce" containerID="20797f56bdd8d0060673559eac370b1774a0b25db5e80dbe351b8cec3010f5e8" exitCode=0
Sep 09 16:44:37.626327 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:37.626135    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d"
Sep 09 16:44:37.626327 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:37.626180    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d" event={"ID":"258b7a1e-a5b1-4041-9e22-4d618e367bce","Type":"ContainerDied","Data":"20797f56bdd8d0060673559eac370b1774a0b25db5e80dbe351b8cec3010f5e8"}
Sep 09 16:44:37.626327 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:37.626233    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d" event={"ID":"258b7a1e-a5b1-4041-9e22-4d618e367bce","Type":"ContainerDied","Data":"111e81722b62a3ce5d92ed37cdf0dbfd4717b9ac67bc56ecebc3961a01a08ff1"}
Sep 09 16:44:37.626327 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:37.626257    2555 scope.go:117] "RemoveContainer" containerID="20797f56bdd8d0060673559eac370b1774a0b25db5e80dbe351b8cec3010f5e8"
Sep 09 16:44:37.642253 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:37.642229    2555 scope.go:117] "RemoveContainer" containerID="20797f56bdd8d0060673559eac370b1774a0b25db5e80dbe351b8cec3010f5e8"
Sep 09 16:44:37.642580 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:44:37.642552    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20797f56bdd8d0060673559eac370b1774a0b25db5e80dbe351b8cec3010f5e8\": container with ID starting with 20797f56bdd8d0060673559eac370b1774a0b25db5e80dbe351b8cec3010f5e8 not found: ID does not exist" containerID="20797f56bdd8d0060673559eac370b1774a0b25db5e80dbe351b8cec3010f5e8"
Sep 09 16:44:37.642657 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:37.642582    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20797f56bdd8d0060673559eac370b1774a0b25db5e80dbe351b8cec3010f5e8"} err="failed to get container status \"20797f56bdd8d0060673559eac370b1774a0b25db5e80dbe351b8cec3010f5e8\": rpc error: code = NotFound desc = could not find container \"20797f56bdd8d0060673559eac370b1774a0b25db5e80dbe351b8cec3010f5e8\": container with ID starting with 20797f56bdd8d0060673559eac370b1774a0b25db5e80dbe351b8cec3010f5e8 not found: ID does not exist"
Sep 09 16:44:37.650669 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:37.650647    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d"]
Sep 09 16:44:37.653826 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:37.653804    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1d59dd-5c958fcb66-jkb7d"]
Sep 09 16:44:38.608306 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:38.608264    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="258b7a1e-a5b1-4041-9e22-4d618e367bce" path="/var/lib/kubelet/pods/258b7a1e-a5b1-4041-9e22-4d618e367bce/volumes"
Sep 09 16:44:39.099612 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:39.099573    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:44:41.714115 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.714077    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz"]
Sep 09 16:44:41.714632 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.714617    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" containerName="istio-proxy"
Sep 09 16:44:41.714676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.714635    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" containerName="istio-proxy"
Sep 09 16:44:41.714676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.714656    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="258b7a1e-a5b1-4041-9e22-4d618e367bce" containerName="payload-pre-processing"
Sep 09 16:44:41.714676 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.714665    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="258b7a1e-a5b1-4041-9e22-4d618e367bce" containerName="payload-pre-processing"
Sep 09 16:44:41.714773 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.714678    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0e6ab2-4fbd-4669-8d3c-38191b9e756f" containerName="payload-pre-processing"
Sep 09 16:44:41.714773 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.714687    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0e6ab2-4fbd-4669-8d3c-38191b9e756f" containerName="payload-pre-processing"
Sep 09 16:44:41.714773 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.714706    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" containerName="istio-proxy"
Sep 09 16:44:41.714773 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.714714    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" containerName="istio-proxy"
Sep 09 16:44:41.714773 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.714738    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f63431c3-a6e0-4c22-b7b5-2facc77d6e63" containerName="maas-api"
Sep 09 16:44:41.714773 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.714746    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f63431c3-a6e0-4c22-b7b5-2facc77d6e63" containerName="maas-api"
Sep 09 16:44:41.715249 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.714823    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f63431c3-a6e0-4c22-b7b5-2facc77d6e63" containerName="maas-api"
Sep 09 16:44:41.715249 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.714837    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" containerName="istio-proxy"
Sep 09 16:44:41.715249 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.714845    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa0e6ab2-4fbd-4669-8d3c-38191b9e756f" containerName="payload-pre-processing"
Sep 09 16:44:41.715249 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.714854    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="eadb9d1f-2b00-4442-a24e-e8e6f6dcc15a" containerName="istio-proxy"
Sep 09 16:44:41.715249 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.714863    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="258b7a1e-a5b1-4041-9e22-4d618e367bce" containerName="payload-pre-processing"
Sep 09 16:44:41.719430 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.719410    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz"
Sep 09 16:44:41.722210 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.722183    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-61f0f9-dockercfg-vw8z2\""
Sep 09 16:44:41.722326 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.722226    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-61f0f9\""
Sep 09 16:44:41.727427 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.727405    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz"]
Sep 09 16:44:41.750175 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.750147    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56"]
Sep 09 16:44:41.754111 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.754085    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56"
Sep 09 16:44:41.765503 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.765426    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56"]
Sep 09 16:44:41.830541 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.830508    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10d7d61-e121-4188-8891-f94995e314ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz\" (UID: \"a10d7d61-e121-4188-8891-f94995e314ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz"
Sep 09 16:44:41.830701 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.830547    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4vqp\" (UniqueName: \"kubernetes.io/projected/6c6a8a6e-3e08-4b92-b544-d025ae443124-kube-api-access-k4vqp\") pod \"payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56\" (UID: \"6c6a8a6e-3e08-4b92-b544-d025ae443124\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56"
Sep 09 16:44:41.830701 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.830649    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dt6c\" (UniqueName: \"kubernetes.io/projected/a10d7d61-e121-4188-8891-f94995e314ed-kube-api-access-9dt6c\") pod \"payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz\" (UID: \"a10d7d61-e121-4188-8891-f94995e314ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz"
Sep 09 16:44:41.830701 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.830675    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c6a8a6e-3e08-4b92-b544-d025ae443124-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56\" (UID: \"6c6a8a6e-3e08-4b92-b544-d025ae443124\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56"
Sep 09 16:44:41.931689 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.931651    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dt6c\" (UniqueName: \"kubernetes.io/projected/a10d7d61-e121-4188-8891-f94995e314ed-kube-api-access-9dt6c\") pod \"payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz\" (UID: \"a10d7d61-e121-4188-8891-f94995e314ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz"
Sep 09 16:44:41.931870 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.931706    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c6a8a6e-3e08-4b92-b544-d025ae443124-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56\" (UID: \"6c6a8a6e-3e08-4b92-b544-d025ae443124\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56"
Sep 09 16:44:41.931870 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.931798    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10d7d61-e121-4188-8891-f94995e314ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz\" (UID: \"a10d7d61-e121-4188-8891-f94995e314ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz"
Sep 09 16:44:41.931870 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.931816    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4vqp\" (UniqueName: \"kubernetes.io/projected/6c6a8a6e-3e08-4b92-b544-d025ae443124-kube-api-access-k4vqp\") pod \"payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56\" (UID: \"6c6a8a6e-3e08-4b92-b544-d025ae443124\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56"
Sep 09 16:44:41.932473 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.932449    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c6a8a6e-3e08-4b92-b544-d025ae443124-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56\" (UID: \"6c6a8a6e-3e08-4b92-b544-d025ae443124\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56"
Sep 09 16:44:41.932576 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.932530    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10d7d61-e121-4188-8891-f94995e314ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz\" (UID: \"a10d7d61-e121-4188-8891-f94995e314ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz"
Sep 09 16:44:41.939221 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.939198    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dt6c\" (UniqueName: \"kubernetes.io/projected/a10d7d61-e121-4188-8891-f94995e314ed-kube-api-access-9dt6c\") pod \"payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz\" (UID: \"a10d7d61-e121-4188-8891-f94995e314ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz"
Sep 09 16:44:41.939345 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:41.939285    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4vqp\" (UniqueName: \"kubernetes.io/projected/6c6a8a6e-3e08-4b92-b544-d025ae443124-kube-api-access-k4vqp\") pod \"payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56\" (UID: \"6c6a8a6e-3e08-4b92-b544-d025ae443124\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56"
Sep 09 16:44:42.032127 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:42.032097    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz"
Sep 09 16:44:42.066286 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:42.066252    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56"
Sep 09 16:44:42.312828 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:42.312802    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz"]
Sep 09 16:44:42.313840 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:44:42.313816    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10d7d61_e121_4188_8891_f94995e314ed.slice/crio-f1d6d7c37e8975b768aeff18829e8a36e3c208d6745b86e5d9a883e308efe1aa WatchSource:0}: Error finding container f1d6d7c37e8975b768aeff18829e8a36e3c208d6745b86e5d9a883e308efe1aa: Status 404 returned error can't find the container with id f1d6d7c37e8975b768aeff18829e8a36e3c208d6745b86e5d9a883e308efe1aa
Sep 09 16:44:42.347568 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:42.347539    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56"]
Sep 09 16:44:42.348559 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:44:42.348530    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6a8a6e_3e08_4b92_b544_d025ae443124.slice/crio-07749f571b192e249b4d20c58f0278bf4c6b899c26acdfbbcb46228528f56906 WatchSource:0}: Error finding container 07749f571b192e249b4d20c58f0278bf4c6b899c26acdfbbcb46228528f56906: Status 404 returned error can't find the container with id 07749f571b192e249b4d20c58f0278bf4c6b899c26acdfbbcb46228528f56906
Sep 09 16:44:42.660768 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:42.660671    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56" event={"ID":"6c6a8a6e-3e08-4b92-b544-d025ae443124","Type":"ContainerStarted","Data":"7a2af65f6f6c164ec03b3a81868323059759d48a24a040c36433ce21bbe6d222"}
Sep 09 16:44:42.660768 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:42.660735    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56" event={"ID":"6c6a8a6e-3e08-4b92-b544-d025ae443124","Type":"ContainerStarted","Data":"07749f571b192e249b4d20c58f0278bf4c6b899c26acdfbbcb46228528f56906"}
Sep 09 16:44:42.660768 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:42.660763    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56"
Sep 09 16:44:42.662781 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:42.662751    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz" event={"ID":"a10d7d61-e121-4188-8891-f94995e314ed","Type":"ContainerStarted","Data":"6c795f0aaaa7f8624f8087a491e24000dc1f1d9bcdc33609ad06f496293754a1"}
Sep 09 16:44:42.662910 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:42.662809    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz"
Sep 09 16:44:42.662910 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:42.662834    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz" event={"ID":"a10d7d61-e121-4188-8891-f94995e314ed","Type":"ContainerStarted","Data":"f1d6d7c37e8975b768aeff18829e8a36e3c208d6745b86e5d9a883e308efe1aa"}
Sep 09 16:44:42.675731 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:42.675675    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56" podStartSLOduration=1.675660988 podStartE2EDuration="1.675660988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:44:42.674632159 +0000 UTC m=+1848.504840216" watchObservedRunningTime="2026-09-09 16:44:42.675660988 +0000 UTC m=+1848.505869061"
Sep 09 16:44:42.689394 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:42.689355    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz" podStartSLOduration=1.689344062 podStartE2EDuration="1.689344062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:44:42.687868811 +0000 UTC m=+1848.518076868" watchObservedRunningTime="2026-09-09 16:44:42.689344062 +0000 UTC m=+1848.519552119"
Sep 09 16:44:48.709659 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:48.709598    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx" podUID="9b3b9fba-4c3b-4ab2-b388-560dfd58e04b" containerName="main" containerID="cri-o://2e406c35bb01b16716f3433457c45d2a8fd2a285ea43f8cb620181899365d649" gracePeriod=45
Sep 09 16:44:48.963339 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:48.963276    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:44:49.096006 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.095967    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-home\") pod \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\" (UID: \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\") "
Sep 09 16:44:49.096173 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.096017    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhgqb\" (UniqueName: \"kubernetes.io/projected/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-kube-api-access-rhgqb\") pod \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\" (UID: \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\") "
Sep 09 16:44:49.096173 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.096078    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-tmp-dir\") pod \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\" (UID: \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\") "
Sep 09 16:44:49.096173 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.096127    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-model-cache\") pod \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\" (UID: \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\") "
Sep 09 16:44:49.096173 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.096161    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-tls-certs\") pod \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\" (UID: \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\") "
Sep 09 16:44:49.096413 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.096194    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-dshm\") pod \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\" (UID: \"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b\") "
Sep 09 16:44:49.096413 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.096267    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-home" (OuterVolumeSpecName: "home") pod "9b3b9fba-4c3b-4ab2-b388-560dfd58e04b" (UID: "9b3b9fba-4c3b-4ab2-b388-560dfd58e04b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:44:49.096522 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.096427    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b3b9fba-4c3b-4ab2-b388-560dfd58e04b" (UID: "9b3b9fba-4c3b-4ab2-b388-560dfd58e04b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:44:49.096522 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.096475    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b3b9fba-4c3b-4ab2-b388-560dfd58e04b" (UID: "9b3b9fba-4c3b-4ab2-b388-560dfd58e04b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:44:49.096658 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.096636    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-home\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:49.096658 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.096658    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-tmp-dir\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:49.096810 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.096671    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-model-cache\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:49.098263 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.098231    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-dshm" (OuterVolumeSpecName: "dshm") pod "9b3b9fba-4c3b-4ab2-b388-560dfd58e04b" (UID: "9b3b9fba-4c3b-4ab2-b388-560dfd58e04b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:44:49.098368 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.098258    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-kube-api-access-rhgqb" (OuterVolumeSpecName: "kube-api-access-rhgqb") pod "9b3b9fba-4c3b-4ab2-b388-560dfd58e04b" (UID: "9b3b9fba-4c3b-4ab2-b388-560dfd58e04b"). InnerVolumeSpecName "kube-api-access-rhgqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:44:49.098368 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.098236    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b3b9fba-4c3b-4ab2-b388-560dfd58e04b" (UID: "9b3b9fba-4c3b-4ab2-b388-560dfd58e04b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:44:49.197877 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.197845    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-tls-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:49.197877 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.197871    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-dshm\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:49.197877 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.197880    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhgqb\" (UniqueName: \"kubernetes.io/projected/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b-kube-api-access-rhgqb\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:49.704195 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.704156    2555 generic.go:356] "Generic (PLEG): container finished" podID="9b3b9fba-4c3b-4ab2-b388-560dfd58e04b" containerID="2e406c35bb01b16716f3433457c45d2a8fd2a285ea43f8cb620181899365d649" exitCode=0
Sep 09 16:44:49.704430 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.704201    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"
Sep 09 16:44:49.704430 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.704245    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx" event={"ID":"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b","Type":"ContainerDied","Data":"2e406c35bb01b16716f3433457c45d2a8fd2a285ea43f8cb620181899365d649"}
Sep 09 16:44:49.704430 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.704324    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx" event={"ID":"9b3b9fba-4c3b-4ab2-b388-560dfd58e04b","Type":"ContainerDied","Data":"17ae6f9f1bf649e12a9790d956a084b4b75f0459f7351e31ae36d6da5b0bf315"}
Sep 09 16:44:49.704430 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.704349    2555 scope.go:117] "RemoveContainer" containerID="2e406c35bb01b16716f3433457c45d2a8fd2a285ea43f8cb620181899365d649"
Sep 09 16:44:49.714433 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.714232    2555 scope.go:117] "RemoveContainer" containerID="2e406c35bb01b16716f3433457c45d2a8fd2a285ea43f8cb620181899365d649"
Sep 09 16:44:49.714663 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:44:49.714528    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e406c35bb01b16716f3433457c45d2a8fd2a285ea43f8cb620181899365d649\": container with ID starting with 2e406c35bb01b16716f3433457c45d2a8fd2a285ea43f8cb620181899365d649 not found: ID does not exist" containerID="2e406c35bb01b16716f3433457c45d2a8fd2a285ea43f8cb620181899365d649"
Sep 09 16:44:49.714663 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.714554    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e406c35bb01b16716f3433457c45d2a8fd2a285ea43f8cb620181899365d649"} err="failed to get container status \"2e406c35bb01b16716f3433457c45d2a8fd2a285ea43f8cb620181899365d649\": rpc error: code = NotFound desc = could not find container \"2e406c35bb01b16716f3433457c45d2a8fd2a285ea43f8cb620181899365d649\": container with ID starting with 2e406c35bb01b16716f3433457c45d2a8fd2a285ea43f8cb620181899365d649 not found: ID does not exist"
Sep 09 16:44:49.726897 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.726872    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"]
Sep 09 16:44:49.732186 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.732161    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-331bf8/rate-test-model-331bf8-kserve-6577cf7bff-rwbgx"]
Sep 09 16:44:49.801504 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:49.801472    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x" podUID="a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae" containerName="main" containerID="cri-o://381d3b3635911782285339e023973ab93dd69029a187fb31c0df4dd3fee14aa9" gracePeriod=45
Sep 09 16:44:50.050094 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.050073    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:44:50.105355 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.105326    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-dshm\") pod \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\" (UID: \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\") "
Sep 09 16:44:50.105495 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.105387    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtdvv\" (UniqueName: \"kubernetes.io/projected/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-kube-api-access-wtdvv\") pod \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\" (UID: \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\") "
Sep 09 16:44:50.105495 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.105432    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-tmp-dir\") pod \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\" (UID: \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\") "
Sep 09 16:44:50.105495 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.105475    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-model-cache\") pod \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\" (UID: \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\") "
Sep 09 16:44:50.105648 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.105497    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-home\") pod \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\" (UID: \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\") "
Sep 09 16:44:50.105648 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.105540    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-tls-certs\") pod \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\" (UID: \"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae\") "
Sep 09 16:44:50.105774 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.105743    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-home" (OuterVolumeSpecName: "home") pod "a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae" (UID: "a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:44:50.105852 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.105774    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-model-cache" (OuterVolumeSpecName: "model-cache") pod "a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae" (UID: "a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:44:50.105852 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.105732    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae" (UID: "a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:44:50.105954 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.105909    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-tmp-dir\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:50.105954 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.105931    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-model-cache\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:50.105954 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.105943    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-home\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:50.107476 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.107448    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-kube-api-access-wtdvv" (OuterVolumeSpecName: "kube-api-access-wtdvv") pod "a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae" (UID: "a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae"). InnerVolumeSpecName "kube-api-access-wtdvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:44:50.107568 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.107477    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-dshm" (OuterVolumeSpecName: "dshm") pod "a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae" (UID: "a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:44:50.107994 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.107969    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae" (UID: "a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:44:50.207157 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.207127    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-dshm\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:50.207157 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.207152    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtdvv\" (UniqueName: \"kubernetes.io/projected/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-kube-api-access-wtdvv\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:50.207157 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.207162    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae-tls-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:44:50.607683 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.607652    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b3b9fba-4c3b-4ab2-b388-560dfd58e04b" path="/var/lib/kubelet/pods/9b3b9fba-4c3b-4ab2-b388-560dfd58e04b/volumes"
Sep 09 16:44:50.712485 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.712454    2555 generic.go:356] "Generic (PLEG): container finished" podID="a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae" containerID="381d3b3635911782285339e023973ab93dd69029a187fb31c0df4dd3fee14aa9" exitCode=0
Sep 09 16:44:50.712644 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.712494    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"
Sep 09 16:44:50.712644 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.712533    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x" event={"ID":"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae","Type":"ContainerDied","Data":"381d3b3635911782285339e023973ab93dd69029a187fb31c0df4dd3fee14aa9"}
Sep 09 16:44:50.712644 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.712569    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x" event={"ID":"a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae","Type":"ContainerDied","Data":"debc76f8f2b838ed179e46018a57c5a34be4eb66e453d523794e651f3abbe790"}
Sep 09 16:44:50.712644 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.712588    2555 scope.go:117] "RemoveContainer" containerID="381d3b3635911782285339e023973ab93dd69029a187fb31c0df4dd3fee14aa9"
Sep 09 16:44:50.721846 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.721637    2555 scope.go:117] "RemoveContainer" containerID="381d3b3635911782285339e023973ab93dd69029a187fb31c0df4dd3fee14aa9"
Sep 09 16:44:50.722052 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:44:50.721875    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"381d3b3635911782285339e023973ab93dd69029a187fb31c0df4dd3fee14aa9\": container with ID starting with 381d3b3635911782285339e023973ab93dd69029a187fb31c0df4dd3fee14aa9 not found: ID does not exist" containerID="381d3b3635911782285339e023973ab93dd69029a187fb31c0df4dd3fee14aa9"
Sep 09 16:44:50.722052 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.721900    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"381d3b3635911782285339e023973ab93dd69029a187fb31c0df4dd3fee14aa9"} err="failed to get container status \"381d3b3635911782285339e023973ab93dd69029a187fb31c0df4dd3fee14aa9\": rpc error: code = NotFound desc = could not find container \"381d3b3635911782285339e023973ab93dd69029a187fb31c0df4dd3fee14aa9\": container with ID starting with 381d3b3635911782285339e023973ab93dd69029a187fb31c0df4dd3fee14aa9 not found: ID does not exist"
Sep 09 16:44:50.733697 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.733674    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"]
Sep 09 16:44:50.738547 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:50.738520    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-50fcbc/rate-test-model-50fcbc-kserve-59d7d9ccf-5275x"]
Sep 09 16:44:52.185903 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:52.185873    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:44:52.611498 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:52.610626    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae" path="/var/lib/kubelet/pods/a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae/volumes"
Sep 09 16:44:53.670715 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:53.670682    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz"
Sep 09 16:44:53.671104 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:53.670833    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56"
Sep 09 16:44:55.065999 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:55.065967    2555 scope.go:117] "RemoveContainer" containerID="8d180a9e961218e4aca67fea8e869920332578e2f68e2df985b3e4eadc484175"
Sep 09 16:44:55.074689 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:44:55.074665    2555 scope.go:117] "RemoveContainer" containerID="0c17a7e4bc72afca1c4be017c858e9d9a9999edf819a2a5e8090c59abf3d4419"
Sep 09 16:45:00.143359 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:00.143326    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-50fcbc-29816205-cvfzp"]
Sep 09 16:45:00.143822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:00.143702    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b3b9fba-4c3b-4ab2-b388-560dfd58e04b" containerName="main"
Sep 09 16:45:00.143822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:00.143714    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b3b9fba-4c3b-4ab2-b388-560dfd58e04b" containerName="main"
Sep 09 16:45:00.143822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:00.143731    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae" containerName="main"
Sep 09 16:45:00.143822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:00.143736    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae" containerName="main"
Sep 09 16:45:00.143822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:00.143814    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b3b9fba-4c3b-4ab2-b388-560dfd58e04b" containerName="main"
Sep 09 16:45:00.143822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:00.143825    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1ec7651-c7b5-41c4-b3cd-5f8fce1bcdae" containerName="main"
Sep 09 16:45:00.146603 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:00.146585    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-50fcbc-29816205-cvfzp"
Sep 09 16:45:00.149184 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:00.149159    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jz5ff\""
Sep 09 16:45:00.156210 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:00.156185    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-50fcbc-29816205-cvfzp"]
Sep 09 16:45:00.159489 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:00.159468    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-50fcbc-29816205-cvfzp"
Sep 09 16:45:00.430403 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:00.430179    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-50fcbc-29816205-cvfzp"]
Sep 09 16:45:00.433103 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:45:00.433073    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0c00ad_1883_44e7_8de5_dd4e396b5b05.slice/crio-fbeed328efa7d36837be045b82e9db6fc77d76807a2d571615f0fc0e089fc24f WatchSource:0}: Error finding container fbeed328efa7d36837be045b82e9db6fc77d76807a2d571615f0fc0e089fc24f: Status 404 returned error can't find the container with id fbeed328efa7d36837be045b82e9db6fc77d76807a2d571615f0fc0e089fc24f
Sep 09 16:45:00.772001 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:00.771964    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-50fcbc-29816205-cvfzp" event={"ID":"ad0c00ad-1883-44e7-8de5-dd4e396b5b05","Type":"ContainerStarted","Data":"fbeed328efa7d36837be045b82e9db6fc77d76807a2d571615f0fc0e089fc24f"}
Sep 09 16:45:03.791440 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:03.791400    2555 generic.go:356] "Generic (PLEG): container finished" podID="ad0c00ad-1883-44e7-8de5-dd4e396b5b05" containerID="38a9ff5e3627e23169b291f8ba389afec79f057daa0cd6c2478ac1041c280b0b" exitCode=0
Sep 09 16:45:03.791835 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:03.791491    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-50fcbc-29816205-cvfzp" event={"ID":"ad0c00ad-1883-44e7-8de5-dd4e396b5b05","Type":"ContainerDied","Data":"38a9ff5e3627e23169b291f8ba389afec79f057daa0cd6c2478ac1041c280b0b"}
Sep 09 16:45:04.935668 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:04.935643    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-50fcbc-29816205-cvfzp"
Sep 09 16:45:05.805024 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:05.804991    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-50fcbc-29816205-cvfzp"
Sep 09 16:45:05.805197 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:05.804990    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-50fcbc-29816205-cvfzp" event={"ID":"ad0c00ad-1883-44e7-8de5-dd4e396b5b05","Type":"ContainerDied","Data":"fbeed328efa7d36837be045b82e9db6fc77d76807a2d571615f0fc0e089fc24f"}
Sep 09 16:45:05.805197 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:05.805112    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbeed328efa7d36837be045b82e9db6fc77d76807a2d571615f0fc0e089fc24f"
Sep 09 16:45:10.368834 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.368800    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq"]
Sep 09 16:45:10.369238 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.369225    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad0c00ad-1883-44e7-8de5-dd4e396b5b05" containerName="cleanup"
Sep 09 16:45:10.369284 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.369239    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad0c00ad-1883-44e7-8de5-dd4e396b5b05" containerName="cleanup"
Sep 09 16:45:10.369353 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.369341    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad0c00ad-1883-44e7-8de5-dd4e396b5b05" containerName="cleanup"
Sep 09 16:45:10.372509 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.372487    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq"
Sep 09 16:45:10.374878 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.374854    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-cd0c91-serving-cert\""
Sep 09 16:45:10.383936 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.383914    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq"]
Sep 09 16:45:10.489626 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.489588    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f555h\" (UniqueName: \"kubernetes.io/projected/56aee8a7-9d83-4ca2-a7e0-e4173caa1070-kube-api-access-f555h\") pod \"maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq\" (UID: \"56aee8a7-9d83-4ca2-a7e0-e4173caa1070\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq"
Sep 09 16:45:10.489626 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.489628    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56aee8a7-9d83-4ca2-a7e0-e4173caa1070-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq\" (UID: \"56aee8a7-9d83-4ca2-a7e0-e4173caa1070\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq"
Sep 09 16:45:10.489850 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.489647    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56aee8a7-9d83-4ca2-a7e0-e4173caa1070-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq\" (UID: \"56aee8a7-9d83-4ca2-a7e0-e4173caa1070\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq"
Sep 09 16:45:10.489850 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.489688    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56aee8a7-9d83-4ca2-a7e0-e4173caa1070-metrics-certs\") pod \"maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq\" (UID: \"56aee8a7-9d83-4ca2-a7e0-e4173caa1070\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq"
Sep 09 16:45:10.590868 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.590833    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f555h\" (UniqueName: \"kubernetes.io/projected/56aee8a7-9d83-4ca2-a7e0-e4173caa1070-kube-api-access-f555h\") pod \"maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq\" (UID: \"56aee8a7-9d83-4ca2-a7e0-e4173caa1070\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq"
Sep 09 16:45:10.590868 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.590872    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56aee8a7-9d83-4ca2-a7e0-e4173caa1070-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq\" (UID: \"56aee8a7-9d83-4ca2-a7e0-e4173caa1070\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq"
Sep 09 16:45:10.591118 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.590895    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56aee8a7-9d83-4ca2-a7e0-e4173caa1070-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq\" (UID: \"56aee8a7-9d83-4ca2-a7e0-e4173caa1070\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq"
Sep 09 16:45:10.591118 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.591024    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56aee8a7-9d83-4ca2-a7e0-e4173caa1070-metrics-certs\") pod \"maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq\" (UID: \"56aee8a7-9d83-4ca2-a7e0-e4173caa1070\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq"
Sep 09 16:45:10.591613 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.591590    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56aee8a7-9d83-4ca2-a7e0-e4173caa1070-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq\" (UID: \"56aee8a7-9d83-4ca2-a7e0-e4173caa1070\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq"
Sep 09 16:45:10.593202 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.593178    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56aee8a7-9d83-4ca2-a7e0-e4173caa1070-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq\" (UID: \"56aee8a7-9d83-4ca2-a7e0-e4173caa1070\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq"
Sep 09 16:45:10.593320 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.593269    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56aee8a7-9d83-4ca2-a7e0-e4173caa1070-metrics-certs\") pod \"maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq\" (UID: \"56aee8a7-9d83-4ca2-a7e0-e4173caa1070\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq"
Sep 09 16:45:10.598024 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.598001    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f555h\" (UniqueName: \"kubernetes.io/projected/56aee8a7-9d83-4ca2-a7e0-e4173caa1070-kube-api-access-f555h\") pod \"maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq\" (UID: \"56aee8a7-9d83-4ca2-a7e0-e4173caa1070\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq"
Sep 09 16:45:10.685646 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.685577    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq"
Sep 09 16:45:10.957997 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.957969    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq"]
Sep 09 16:45:10.959859 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:45:10.959833    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56aee8a7_9d83_4ca2_a7e0_e4173caa1070.slice/crio-2d26b66367b272be2bdc3e78ac30e41f3436e3655d89d1ea8b4a50562967f3f0 WatchSource:0}: Error finding container 2d26b66367b272be2bdc3e78ac30e41f3436e3655d89d1ea8b4a50562967f3f0: Status 404 returned error can't find the container with id 2d26b66367b272be2bdc3e78ac30e41f3436e3655d89d1ea8b4a50562967f3f0
Sep 09 16:45:10.968485 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.968461    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27"]
Sep 09 16:45:10.972985 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.972968    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27"
Sep 09 16:45:10.975514 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.975494    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-cd0c91\""
Sep 09 16:45:10.975609 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.975554    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-cd0c91-dockercfg-r2bqv\""
Sep 09 16:45:10.981157 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:10.981136    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27"]
Sep 09 16:45:11.095758 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:11.095720    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsnjr\" (UniqueName: \"kubernetes.io/projected/d38988e9-be38-4e83-b53e-379ecffd2c3b-kube-api-access-hsnjr\") pod \"payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27\" (UID: \"d38988e9-be38-4e83-b53e-379ecffd2c3b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27"
Sep 09 16:45:11.095934 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:11.095763    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38988e9-be38-4e83-b53e-379ecffd2c3b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27\" (UID: \"d38988e9-be38-4e83-b53e-379ecffd2c3b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27"
Sep 09 16:45:11.197243 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:11.197213    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsnjr\" (UniqueName: \"kubernetes.io/projected/d38988e9-be38-4e83-b53e-379ecffd2c3b-kube-api-access-hsnjr\") pod \"payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27\" (UID: \"d38988e9-be38-4e83-b53e-379ecffd2c3b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27"
Sep 09 16:45:11.197433 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:11.197252    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38988e9-be38-4e83-b53e-379ecffd2c3b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27\" (UID: \"d38988e9-be38-4e83-b53e-379ecffd2c3b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27"
Sep 09 16:45:11.198010 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:11.197989    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38988e9-be38-4e83-b53e-379ecffd2c3b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27\" (UID: \"d38988e9-be38-4e83-b53e-379ecffd2c3b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27"
Sep 09 16:45:11.204694 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:11.204676    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsnjr\" (UniqueName: \"kubernetes.io/projected/d38988e9-be38-4e83-b53e-379ecffd2c3b-kube-api-access-hsnjr\") pod \"payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27\" (UID: \"d38988e9-be38-4e83-b53e-379ecffd2c3b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27"
Sep 09 16:45:11.284967 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:11.284935    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27"
Sep 09 16:45:11.579515 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:11.579486    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27"]
Sep 09 16:45:11.580187 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:45:11.580161    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38988e9_be38_4e83_b53e_379ecffd2c3b.slice/crio-40d34f0cda88137bcac8c820b191dd3ca0efead9ff6b247e5e534e344cc360c2 WatchSource:0}: Error finding container 40d34f0cda88137bcac8c820b191dd3ca0efead9ff6b247e5e534e344cc360c2: Status 404 returned error can't find the container with id 40d34f0cda88137bcac8c820b191dd3ca0efead9ff6b247e5e534e344cc360c2
Sep 09 16:45:11.841168 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:11.841072    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq" event={"ID":"56aee8a7-9d83-4ca2-a7e0-e4173caa1070","Type":"ContainerStarted","Data":"012ac724df3a971525cd49dad1754bedbd3906aa2e71f83688653ddcdfced8b1"}
Sep 09 16:45:11.841168 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:11.841133    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq" event={"ID":"56aee8a7-9d83-4ca2-a7e0-e4173caa1070","Type":"ContainerStarted","Data":"2d26b66367b272be2bdc3e78ac30e41f3436e3655d89d1ea8b4a50562967f3f0"}
Sep 09 16:45:11.841434 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:11.841302    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq"
Sep 09 16:45:11.842795 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:11.842767    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27" event={"ID":"d38988e9-be38-4e83-b53e-379ecffd2c3b","Type":"ContainerStarted","Data":"63c8d0562840ce65895d6a2d4749f2ff579f8d29874aa60efa204717fdb45624"}
Sep 09 16:45:11.842920 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:11.842818    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27" event={"ID":"d38988e9-be38-4e83-b53e-379ecffd2c3b","Type":"ContainerStarted","Data":"40d34f0cda88137bcac8c820b191dd3ca0efead9ff6b247e5e534e344cc360c2"}
Sep 09 16:45:11.842920 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:11.842849    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27"
Sep 09 16:45:11.858596 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:11.858551    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq" podStartSLOduration=1.469787578 podStartE2EDuration="1.858537669s" totalImagesPullingTime="388.750091ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:45:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:45:11.856114663 +0000 UTC m=+1877.686322722" watchObservedRunningTime="2026-09-09 16:45:11.858537669 +0000 UTC m=+1877.688745726"
Sep 09 16:45:11.874975 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:11.874918    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27" podStartSLOduration=1.87489998 podStartE2EDuration="1.87489998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:45:11.874098903 +0000 UTC m=+1877.704306961" watchObservedRunningTime="2026-09-09 16:45:11.87489998 +0000 UTC m=+1877.705108058"
Sep 09 16:45:14.494554 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:14.494522    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:45:17.855204 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:17.855166    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq"
Sep 09 16:45:22.851389 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:22.851327    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27"
Sep 09 16:45:38.594879 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:38.594852    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27_d38988e9-be38-4e83-b53e-379ecffd2c3b/payload-processing/0.log"
Sep 09 16:45:51.989176 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:45:51.989134    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:46:17.994879 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:17.994841    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:46:24.429356 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:24.429319    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56_6c6a8a6e-3e08-4b92-b544-d025ae443124/payload-processing/0.log"
Sep 09 16:46:24.682389 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:24.682286    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27_d38988e9-be38-4e83-b53e-379ecffd2c3b/payload-processing/0.log"
Sep 09 16:46:31.275804 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:31.275771    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9"]
Sep 09 16:46:31.279616 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:31.279573    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9"
Sep 09 16:46:31.282134 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:31.282110    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-89cc2b-serving-cert\""
Sep 09 16:46:31.287848 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:31.287667    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9"]
Sep 09 16:46:31.432004 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:31.431966    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bebda1-3586-4101-bd45-9f7f58b93e90-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9\" (UID: \"c2bebda1-3586-4101-bd45-9f7f58b93e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9"
Sep 09 16:46:31.432187 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:31.432009    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2bebda1-3586-4101-bd45-9f7f58b93e90-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9\" (UID: \"c2bebda1-3586-4101-bd45-9f7f58b93e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9"
Sep 09 16:46:31.432187 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:31.432123    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2bebda1-3586-4101-bd45-9f7f58b93e90-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9\" (UID: \"c2bebda1-3586-4101-bd45-9f7f58b93e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9"
Sep 09 16:46:31.432187 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:31.432176    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc5vn\" (UniqueName: \"kubernetes.io/projected/c2bebda1-3586-4101-bd45-9f7f58b93e90-kube-api-access-jc5vn\") pod \"maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9\" (UID: \"c2bebda1-3586-4101-bd45-9f7f58b93e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9"
Sep 09 16:46:31.532938 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:31.532838    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bebda1-3586-4101-bd45-9f7f58b93e90-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9\" (UID: \"c2bebda1-3586-4101-bd45-9f7f58b93e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9"
Sep 09 16:46:31.532938 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:31.532878    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2bebda1-3586-4101-bd45-9f7f58b93e90-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9\" (UID: \"c2bebda1-3586-4101-bd45-9f7f58b93e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9"
Sep 09 16:46:31.533177 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:31.532958    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2bebda1-3586-4101-bd45-9f7f58b93e90-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9\" (UID: \"c2bebda1-3586-4101-bd45-9f7f58b93e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9"
Sep 09 16:46:31.533177 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:31.533008    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc5vn\" (UniqueName: \"kubernetes.io/projected/c2bebda1-3586-4101-bd45-9f7f58b93e90-kube-api-access-jc5vn\") pod \"maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9\" (UID: \"c2bebda1-3586-4101-bd45-9f7f58b93e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9"
Sep 09 16:46:31.533651 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:31.533629    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bebda1-3586-4101-bd45-9f7f58b93e90-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9\" (UID: \"c2bebda1-3586-4101-bd45-9f7f58b93e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9"
Sep 09 16:46:31.535371 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:31.535350    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2bebda1-3586-4101-bd45-9f7f58b93e90-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9\" (UID: \"c2bebda1-3586-4101-bd45-9f7f58b93e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9"
Sep 09 16:46:31.535531 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:31.535509    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2bebda1-3586-4101-bd45-9f7f58b93e90-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9\" (UID: \"c2bebda1-3586-4101-bd45-9f7f58b93e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9"
Sep 09 16:46:31.540762 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:31.540737    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc5vn\" (UniqueName: \"kubernetes.io/projected/c2bebda1-3586-4101-bd45-9f7f58b93e90-kube-api-access-jc5vn\") pod \"maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9\" (UID: \"c2bebda1-3586-4101-bd45-9f7f58b93e90\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9"
Sep 09 16:46:31.595403 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:31.595374    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9"
Sep 09 16:46:31.877793 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:31.877764    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9"]
Sep 09 16:46:31.879810 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:46:31.879786    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2bebda1_3586_4101_bd45_9f7f58b93e90.slice/crio-3125350c613c8986799bff1d6639ed7ba5b52a14af101d773c8eaf0cc6699639 WatchSource:0}: Error finding container 3125350c613c8986799bff1d6639ed7ba5b52a14af101d773c8eaf0cc6699639: Status 404 returned error can't find the container with id 3125350c613c8986799bff1d6639ed7ba5b52a14af101d773c8eaf0cc6699639
Sep 09 16:46:31.931598 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:31.931567    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc"]
Sep 09 16:46:31.936285 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:31.936268    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc"
Sep 09 16:46:31.938671 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:31.938651    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-89cc2b\""
Sep 09 16:46:31.948836 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:31.948809    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc"]
Sep 09 16:46:32.037778 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:32.037734    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwc7f\" (UniqueName: \"kubernetes.io/projected/e9149dd7-8eac-4cd1-8087-6c166a8989ea-kube-api-access-nwc7f\") pod \"payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc\" (UID: \"e9149dd7-8eac-4cd1-8087-6c166a8989ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc"
Sep 09 16:46:32.037971 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:32.037824    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9149dd7-8eac-4cd1-8087-6c166a8989ea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc\" (UID: \"e9149dd7-8eac-4cd1-8087-6c166a8989ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc"
Sep 09 16:46:32.139480 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:32.139395    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwc7f\" (UniqueName: \"kubernetes.io/projected/e9149dd7-8eac-4cd1-8087-6c166a8989ea-kube-api-access-nwc7f\") pod \"payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc\" (UID: \"e9149dd7-8eac-4cd1-8087-6c166a8989ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc"
Sep 09 16:46:32.139480 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:32.139453    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9149dd7-8eac-4cd1-8087-6c166a8989ea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc\" (UID: \"e9149dd7-8eac-4cd1-8087-6c166a8989ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc"
Sep 09 16:46:32.140117 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:32.140096    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9149dd7-8eac-4cd1-8087-6c166a8989ea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc\" (UID: \"e9149dd7-8eac-4cd1-8087-6c166a8989ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc"
Sep 09 16:46:32.147235 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:32.147208    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwc7f\" (UniqueName: \"kubernetes.io/projected/e9149dd7-8eac-4cd1-8087-6c166a8989ea-kube-api-access-nwc7f\") pod \"payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc\" (UID: \"e9149dd7-8eac-4cd1-8087-6c166a8989ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc"
Sep 09 16:46:32.249967 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:32.249944    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc"
Sep 09 16:46:32.309288 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:32.309254    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9" event={"ID":"c2bebda1-3586-4101-bd45-9f7f58b93e90","Type":"ContainerStarted","Data":"e75364c4c82258f4a7cc8bf8f03c3684c9ca3aae4d4a3d89bb100956f858974c"}
Sep 09 16:46:32.309721 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:32.309530    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9" event={"ID":"c2bebda1-3586-4101-bd45-9f7f58b93e90","Type":"ContainerStarted","Data":"3125350c613c8986799bff1d6639ed7ba5b52a14af101d773c8eaf0cc6699639"}
Sep 09 16:46:32.310074 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:32.310055    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9"
Sep 09 16:46:32.328734 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:32.328686    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9" podStartSLOduration=0.981934725 podStartE2EDuration="1.328671863s" totalImagesPullingTime="346.737138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:46:32.326324061 +0000 UTC m=+1958.156532120" watchObservedRunningTime="2026-09-09 16:46:32.328671863 +0000 UTC m=+1958.158879920"
Sep 09 16:46:32.543036 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:32.542960    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc"]
Sep 09 16:46:32.544023 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:46:32.543995    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9149dd7_8eac_4cd1_8087_6c166a8989ea.slice/crio-0e29e813e6f52c0c343c272c674137092f8735dff4742bc16c8e51878d1b9de8 WatchSource:0}: Error finding container 0e29e813e6f52c0c343c272c674137092f8735dff4742bc16c8e51878d1b9de8: Status 404 returned error can't find the container with id 0e29e813e6f52c0c343c272c674137092f8735dff4742bc16c8e51878d1b9de8
Sep 09 16:46:33.318978 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:33.318936    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc" event={"ID":"e9149dd7-8eac-4cd1-8087-6c166a8989ea","Type":"ContainerStarted","Data":"9a4e028ec0d2ab9742f5e16eaf7e61c7081fc662a84328654b5c24f12bf502f7"}
Sep 09 16:46:33.319487 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:33.319008    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc" event={"ID":"e9149dd7-8eac-4cd1-8087-6c166a8989ea","Type":"ContainerStarted","Data":"0e29e813e6f52c0c343c272c674137092f8735dff4742bc16c8e51878d1b9de8"}
Sep 09 16:46:33.319487 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:33.319102    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc"
Sep 09 16:46:33.332868 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:33.332794    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc" podStartSLOduration=2.332779457 podStartE2EDuration="2.332779457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:46:33.332369214 +0000 UTC m=+1959.162577273" watchObservedRunningTime="2026-09-09 16:46:33.332779457 +0000 UTC m=+1959.162987516"
Sep 09 16:46:37.288280 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:37.288243    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:46:38.324144 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:38.324113    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9"
Sep 09 16:46:44.327823 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:46:44.327793    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc"
Sep 09 16:47:10.424892 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:10.424853    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9"]
Sep 09 16:47:10.425812 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:10.425776    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9" podUID="c2bebda1-3586-4101-bd45-9f7f58b93e90" containerName="maas-api" containerID="cri-o://e75364c4c82258f4a7cc8bf8f03c3684c9ca3aae4d4a3d89bb100956f858974c" gracePeriod=30
Sep 09 16:47:10.475186 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:10.475152    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc"]
Sep 09 16:47:10.475528 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:10.475495    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc" podUID="e9149dd7-8eac-4cd1-8087-6c166a8989ea" containerName="payload-processing" containerID="cri-o://9a4e028ec0d2ab9742f5e16eaf7e61c7081fc662a84328654b5c24f12bf502f7" gracePeriod=30
Sep 09 16:47:10.739986 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:10.739956    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9"
Sep 09 16:47:10.768784 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:10.768761    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc"
Sep 09 16:47:10.774370 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:10.774348    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc5vn\" (UniqueName: \"kubernetes.io/projected/c2bebda1-3586-4101-bd45-9f7f58b93e90-kube-api-access-jc5vn\") pod \"c2bebda1-3586-4101-bd45-9f7f58b93e90\" (UID: \"c2bebda1-3586-4101-bd45-9f7f58b93e90\") "
Sep 09 16:47:10.774481 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:10.774380    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2bebda1-3586-4101-bd45-9f7f58b93e90-maas-api-tls\") pod \"c2bebda1-3586-4101-bd45-9f7f58b93e90\" (UID: \"c2bebda1-3586-4101-bd45-9f7f58b93e90\") "
Sep 09 16:47:10.774481 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:10.774425    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bebda1-3586-4101-bd45-9f7f58b93e90-openshift-service-ca\") pod \"c2bebda1-3586-4101-bd45-9f7f58b93e90\" (UID: \"c2bebda1-3586-4101-bd45-9f7f58b93e90\") "
Sep 09 16:47:10.774594 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:10.774511    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2bebda1-3586-4101-bd45-9f7f58b93e90-metrics-certs\") pod \"c2bebda1-3586-4101-bd45-9f7f58b93e90\" (UID: \"c2bebda1-3586-4101-bd45-9f7f58b93e90\") "
Sep 09 16:47:10.774758 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:10.774734    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2bebda1-3586-4101-bd45-9f7f58b93e90-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c2bebda1-3586-4101-bd45-9f7f58b93e90" (UID: "c2bebda1-3586-4101-bd45-9f7f58b93e90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:47:10.774895 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:10.774883    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bebda1-3586-4101-bd45-9f7f58b93e90-openshift-service-ca\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:47:10.776487 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:10.776459    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2bebda1-3586-4101-bd45-9f7f58b93e90-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c2bebda1-3586-4101-bd45-9f7f58b93e90" (UID: "c2bebda1-3586-4101-bd45-9f7f58b93e90"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:47:10.776839 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:10.776819    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2bebda1-3586-4101-bd45-9f7f58b93e90-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c2bebda1-3586-4101-bd45-9f7f58b93e90" (UID: "c2bebda1-3586-4101-bd45-9f7f58b93e90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:47:10.776910 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:10.776856    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2bebda1-3586-4101-bd45-9f7f58b93e90-kube-api-access-jc5vn" (OuterVolumeSpecName: "kube-api-access-jc5vn") pod "c2bebda1-3586-4101-bd45-9f7f58b93e90" (UID: "c2bebda1-3586-4101-bd45-9f7f58b93e90"). InnerVolumeSpecName "kube-api-access-jc5vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:47:10.876143 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:10.876112    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwc7f\" (UniqueName: \"kubernetes.io/projected/e9149dd7-8eac-4cd1-8087-6c166a8989ea-kube-api-access-nwc7f\") pod \"e9149dd7-8eac-4cd1-8087-6c166a8989ea\" (UID: \"e9149dd7-8eac-4cd1-8087-6c166a8989ea\") "
Sep 09 16:47:10.876323 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:10.876152    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9149dd7-8eac-4cd1-8087-6c166a8989ea-plugins-config-volume\") pod \"e9149dd7-8eac-4cd1-8087-6c166a8989ea\" (UID: \"e9149dd7-8eac-4cd1-8087-6c166a8989ea\") "
Sep 09 16:47:10.876379 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:10.876334    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc5vn\" (UniqueName: \"kubernetes.io/projected/c2bebda1-3586-4101-bd45-9f7f58b93e90-kube-api-access-jc5vn\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:47:10.876379 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:10.876355    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2bebda1-3586-4101-bd45-9f7f58b93e90-maas-api-tls\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:47:10.876379 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:10.876369    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2bebda1-3586-4101-bd45-9f7f58b93e90-metrics-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:47:10.876593 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:10.876566    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9149dd7-8eac-4cd1-8087-6c166a8989ea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e9149dd7-8eac-4cd1-8087-6c166a8989ea" (UID: "e9149dd7-8eac-4cd1-8087-6c166a8989ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:47:10.878099 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:10.878081    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9149dd7-8eac-4cd1-8087-6c166a8989ea-kube-api-access-nwc7f" (OuterVolumeSpecName: "kube-api-access-nwc7f") pod "e9149dd7-8eac-4cd1-8087-6c166a8989ea" (UID: "e9149dd7-8eac-4cd1-8087-6c166a8989ea"). InnerVolumeSpecName "kube-api-access-nwc7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:47:10.977597 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:10.977518    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwc7f\" (UniqueName: \"kubernetes.io/projected/e9149dd7-8eac-4cd1-8087-6c166a8989ea-kube-api-access-nwc7f\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:47:10.977597 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:10.977543    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9149dd7-8eac-4cd1-8087-6c166a8989ea-plugins-config-volume\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:47:11.543015 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:11.542971    2555 generic.go:356] "Generic (PLEG): container finished" podID="e9149dd7-8eac-4cd1-8087-6c166a8989ea" containerID="9a4e028ec0d2ab9742f5e16eaf7e61c7081fc662a84328654b5c24f12bf502f7" exitCode=0
Sep 09 16:47:11.543468 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:11.543017    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc"
Sep 09 16:47:11.543468 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:11.543030    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc" event={"ID":"e9149dd7-8eac-4cd1-8087-6c166a8989ea","Type":"ContainerDied","Data":"9a4e028ec0d2ab9742f5e16eaf7e61c7081fc662a84328654b5c24f12bf502f7"}
Sep 09 16:47:11.543468 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:11.543094    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc" event={"ID":"e9149dd7-8eac-4cd1-8087-6c166a8989ea","Type":"ContainerDied","Data":"0e29e813e6f52c0c343c272c674137092f8735dff4742bc16c8e51878d1b9de8"}
Sep 09 16:47:11.543468 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:11.543123    2555 scope.go:117] "RemoveContainer" containerID="9a4e028ec0d2ab9742f5e16eaf7e61c7081fc662a84328654b5c24f12bf502f7"
Sep 09 16:47:11.545167 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:11.545140    2555 generic.go:356] "Generic (PLEG): container finished" podID="c2bebda1-3586-4101-bd45-9f7f58b93e90" containerID="e75364c4c82258f4a7cc8bf8f03c3684c9ca3aae4d4a3d89bb100956f858974c" exitCode=0
Sep 09 16:47:11.545287 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:11.545182    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9" event={"ID":"c2bebda1-3586-4101-bd45-9f7f58b93e90","Type":"ContainerDied","Data":"e75364c4c82258f4a7cc8bf8f03c3684c9ca3aae4d4a3d89bb100956f858974c"}
Sep 09 16:47:11.545287 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:11.545206    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9"
Sep 09 16:47:11.545287 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:11.545225    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9" event={"ID":"c2bebda1-3586-4101-bd45-9f7f58b93e90","Type":"ContainerDied","Data":"3125350c613c8986799bff1d6639ed7ba5b52a14af101d773c8eaf0cc6699639"}
Sep 09 16:47:11.554519 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:11.554500    2555 scope.go:117] "RemoveContainer" containerID="9a4e028ec0d2ab9742f5e16eaf7e61c7081fc662a84328654b5c24f12bf502f7"
Sep 09 16:47:11.554815 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:47:11.554795    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4e028ec0d2ab9742f5e16eaf7e61c7081fc662a84328654b5c24f12bf502f7\": container with ID starting with 9a4e028ec0d2ab9742f5e16eaf7e61c7081fc662a84328654b5c24f12bf502f7 not found: ID does not exist" containerID="9a4e028ec0d2ab9742f5e16eaf7e61c7081fc662a84328654b5c24f12bf502f7"
Sep 09 16:47:11.554883 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:11.554825    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4e028ec0d2ab9742f5e16eaf7e61c7081fc662a84328654b5c24f12bf502f7"} err="failed to get container status \"9a4e028ec0d2ab9742f5e16eaf7e61c7081fc662a84328654b5c24f12bf502f7\": rpc error: code = NotFound desc = could not find container \"9a4e028ec0d2ab9742f5e16eaf7e61c7081fc662a84328654b5c24f12bf502f7\": container with ID starting with 9a4e028ec0d2ab9742f5e16eaf7e61c7081fc662a84328654b5c24f12bf502f7 not found: ID does not exist"
Sep 09 16:47:11.554883 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:11.554847    2555 scope.go:117] "RemoveContainer" containerID="e75364c4c82258f4a7cc8bf8f03c3684c9ca3aae4d4a3d89bb100956f858974c"
Sep 09 16:47:11.563093 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:11.563079    2555 scope.go:117] "RemoveContainer" containerID="e75364c4c82258f4a7cc8bf8f03c3684c9ca3aae4d4a3d89bb100956f858974c"
Sep 09 16:47:11.563354 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:47:11.563329    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e75364c4c82258f4a7cc8bf8f03c3684c9ca3aae4d4a3d89bb100956f858974c\": container with ID starting with e75364c4c82258f4a7cc8bf8f03c3684c9ca3aae4d4a3d89bb100956f858974c not found: ID does not exist" containerID="e75364c4c82258f4a7cc8bf8f03c3684c9ca3aae4d4a3d89bb100956f858974c"
Sep 09 16:47:11.563439 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:11.563357    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e75364c4c82258f4a7cc8bf8f03c3684c9ca3aae4d4a3d89bb100956f858974c"} err="failed to get container status \"e75364c4c82258f4a7cc8bf8f03c3684c9ca3aae4d4a3d89bb100956f858974c\": rpc error: code = NotFound desc = could not find container \"e75364c4c82258f4a7cc8bf8f03c3684c9ca3aae4d4a3d89bb100956f858974c\": container with ID starting with e75364c4c82258f4a7cc8bf8f03c3684c9ca3aae4d4a3d89bb100956f858974c not found: ID does not exist"
Sep 09 16:47:11.570519 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:11.570499    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc"]
Sep 09 16:47:11.573620 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:11.573593    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-89cc2b-5957c8445c-k7tzc"]
Sep 09 16:47:11.583337 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:11.583317    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9"]
Sep 09 16:47:11.585463 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:11.585444    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-89cc2b-5bd47cb5f6-x7fk9"]
Sep 09 16:47:12.608883 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:12.608850    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2bebda1-3586-4101-bd45-9f7f58b93e90" path="/var/lib/kubelet/pods/c2bebda1-3586-4101-bd45-9f7f58b93e90/volumes"
Sep 09 16:47:12.609274 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:12.609224    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9149dd7-8eac-4cd1-8087-6c166a8989ea" path="/var/lib/kubelet/pods/e9149dd7-8eac-4cd1-8087-6c166a8989ea/volumes"
Sep 09 16:47:15.097851 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:15.097810    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:47:33.792023 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:33.791985    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:47:36.130837 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.130800    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56"]
Sep 09 16:47:36.131313 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.131060    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56" podUID="6c6a8a6e-3e08-4b92-b544-d025ae443124" containerName="payload-processing" containerID="cri-o://7a2af65f6f6c164ec03b3a81868323059759d48a24a040c36433ce21bbe6d222" gracePeriod=30
Sep 09 16:47:36.137538 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.137509    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz"]
Sep 09 16:47:36.137725 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.137699    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz" podUID="a10d7d61-e121-4188-8891-f94995e314ed" containerName="payload-pre-processing" containerID="cri-o://6c795f0aaaa7f8624f8087a491e24000dc1f1d9bcdc33609ad06f496293754a1" gracePeriod=30
Sep 09 16:47:36.420619 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.420590    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56"
Sep 09 16:47:36.423381 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.423364    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz"
Sep 09 16:47:36.512485 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.512449    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4vqp\" (UniqueName: \"kubernetes.io/projected/6c6a8a6e-3e08-4b92-b544-d025ae443124-kube-api-access-k4vqp\") pod \"6c6a8a6e-3e08-4b92-b544-d025ae443124\" (UID: \"6c6a8a6e-3e08-4b92-b544-d025ae443124\") "
Sep 09 16:47:36.512651 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.512508    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10d7d61-e121-4188-8891-f94995e314ed-plugins-config-volume\") pod \"a10d7d61-e121-4188-8891-f94995e314ed\" (UID: \"a10d7d61-e121-4188-8891-f94995e314ed\") "
Sep 09 16:47:36.512651 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.512565    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dt6c\" (UniqueName: \"kubernetes.io/projected/a10d7d61-e121-4188-8891-f94995e314ed-kube-api-access-9dt6c\") pod \"a10d7d61-e121-4188-8891-f94995e314ed\" (UID: \"a10d7d61-e121-4188-8891-f94995e314ed\") "
Sep 09 16:47:36.512651 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.512601    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c6a8a6e-3e08-4b92-b544-d025ae443124-plugins-config-volume\") pod \"6c6a8a6e-3e08-4b92-b544-d025ae443124\" (UID: \"6c6a8a6e-3e08-4b92-b544-d025ae443124\") "
Sep 09 16:47:36.512952 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.512923    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10d7d61-e121-4188-8891-f94995e314ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a10d7d61-e121-4188-8891-f94995e314ed" (UID: "a10d7d61-e121-4188-8891-f94995e314ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:47:36.513035 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.512968    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6a8a6e-3e08-4b92-b544-d025ae443124-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6c6a8a6e-3e08-4b92-b544-d025ae443124" (UID: "6c6a8a6e-3e08-4b92-b544-d025ae443124"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:47:36.514546 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.514522    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6a8a6e-3e08-4b92-b544-d025ae443124-kube-api-access-k4vqp" (OuterVolumeSpecName: "kube-api-access-k4vqp") pod "6c6a8a6e-3e08-4b92-b544-d025ae443124" (UID: "6c6a8a6e-3e08-4b92-b544-d025ae443124"). InnerVolumeSpecName "kube-api-access-k4vqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:47:36.514618 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.514594    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10d7d61-e121-4188-8891-f94995e314ed-kube-api-access-9dt6c" (OuterVolumeSpecName: "kube-api-access-9dt6c") pod "a10d7d61-e121-4188-8891-f94995e314ed" (UID: "a10d7d61-e121-4188-8891-f94995e314ed"). InnerVolumeSpecName "kube-api-access-9dt6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:47:36.613185 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.613156    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4vqp\" (UniqueName: \"kubernetes.io/projected/6c6a8a6e-3e08-4b92-b544-d025ae443124-kube-api-access-k4vqp\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:47:36.613185 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.613181    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10d7d61-e121-4188-8891-f94995e314ed-plugins-config-volume\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:47:36.613185 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.613190    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dt6c\" (UniqueName: \"kubernetes.io/projected/a10d7d61-e121-4188-8891-f94995e314ed-kube-api-access-9dt6c\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:47:36.613445 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.613199    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c6a8a6e-3e08-4b92-b544-d025ae443124-plugins-config-volume\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:47:36.692984 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.692902    2555 generic.go:356] "Generic (PLEG): container finished" podID="6c6a8a6e-3e08-4b92-b544-d025ae443124" containerID="7a2af65f6f6c164ec03b3a81868323059759d48a24a040c36433ce21bbe6d222" exitCode=0
Sep 09 16:47:36.692984 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.692950    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56"
Sep 09 16:47:36.693191 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.692981    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56" event={"ID":"6c6a8a6e-3e08-4b92-b544-d025ae443124","Type":"ContainerDied","Data":"7a2af65f6f6c164ec03b3a81868323059759d48a24a040c36433ce21bbe6d222"}
Sep 09 16:47:36.693191 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.693035    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56" event={"ID":"6c6a8a6e-3e08-4b92-b544-d025ae443124","Type":"ContainerDied","Data":"07749f571b192e249b4d20c58f0278bf4c6b899c26acdfbbcb46228528f56906"}
Sep 09 16:47:36.693191 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.693054    2555 scope.go:117] "RemoveContainer" containerID="7a2af65f6f6c164ec03b3a81868323059759d48a24a040c36433ce21bbe6d222"
Sep 09 16:47:36.694619 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.694590    2555 generic.go:356] "Generic (PLEG): container finished" podID="a10d7d61-e121-4188-8891-f94995e314ed" containerID="6c795f0aaaa7f8624f8087a491e24000dc1f1d9bcdc33609ad06f496293754a1" exitCode=0
Sep 09 16:47:36.694734 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.694637    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz"
Sep 09 16:47:36.694798 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.694668    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz" event={"ID":"a10d7d61-e121-4188-8891-f94995e314ed","Type":"ContainerDied","Data":"6c795f0aaaa7f8624f8087a491e24000dc1f1d9bcdc33609ad06f496293754a1"}
Sep 09 16:47:36.694879 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.694860    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz" event={"ID":"a10d7d61-e121-4188-8891-f94995e314ed","Type":"ContainerDied","Data":"f1d6d7c37e8975b768aeff18829e8a36e3c208d6745b86e5d9a883e308efe1aa"}
Sep 09 16:47:36.702588 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.702574    2555 scope.go:117] "RemoveContainer" containerID="7a2af65f6f6c164ec03b3a81868323059759d48a24a040c36433ce21bbe6d222"
Sep 09 16:47:36.702812 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:47:36.702795    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a2af65f6f6c164ec03b3a81868323059759d48a24a040c36433ce21bbe6d222\": container with ID starting with 7a2af65f6f6c164ec03b3a81868323059759d48a24a040c36433ce21bbe6d222 not found: ID does not exist" containerID="7a2af65f6f6c164ec03b3a81868323059759d48a24a040c36433ce21bbe6d222"
Sep 09 16:47:36.702862 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.702819    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a2af65f6f6c164ec03b3a81868323059759d48a24a040c36433ce21bbe6d222"} err="failed to get container status \"7a2af65f6f6c164ec03b3a81868323059759d48a24a040c36433ce21bbe6d222\": rpc error: code = NotFound desc = could not find container \"7a2af65f6f6c164ec03b3a81868323059759d48a24a040c36433ce21bbe6d222\": container with ID starting with 7a2af65f6f6c164ec03b3a81868323059759d48a24a040c36433ce21bbe6d222 not found: ID does not exist"
Sep 09 16:47:36.702862 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.702834    2555 scope.go:117] "RemoveContainer" containerID="6c795f0aaaa7f8624f8087a491e24000dc1f1d9bcdc33609ad06f496293754a1"
Sep 09 16:47:36.711210 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.711187    2555 scope.go:117] "RemoveContainer" containerID="6c795f0aaaa7f8624f8087a491e24000dc1f1d9bcdc33609ad06f496293754a1"
Sep 09 16:47:36.711856 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:47:36.711827    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c795f0aaaa7f8624f8087a491e24000dc1f1d9bcdc33609ad06f496293754a1\": container with ID starting with 6c795f0aaaa7f8624f8087a491e24000dc1f1d9bcdc33609ad06f496293754a1 not found: ID does not exist" containerID="6c795f0aaaa7f8624f8087a491e24000dc1f1d9bcdc33609ad06f496293754a1"
Sep 09 16:47:36.711944 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.711863    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c795f0aaaa7f8624f8087a491e24000dc1f1d9bcdc33609ad06f496293754a1"} err="failed to get container status \"6c795f0aaaa7f8624f8087a491e24000dc1f1d9bcdc33609ad06f496293754a1\": rpc error: code = NotFound desc = could not find container \"6c795f0aaaa7f8624f8087a491e24000dc1f1d9bcdc33609ad06f496293754a1\": container with ID starting with 6c795f0aaaa7f8624f8087a491e24000dc1f1d9bcdc33609ad06f496293754a1 not found: ID does not exist"
Sep 09 16:47:36.712690 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.712670    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56"]
Sep 09 16:47:36.715201 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.715183    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-61f0f9-c5d4754fc-rkg56"]
Sep 09 16:47:36.723397 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.723380    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz"]
Sep 09 16:47:36.728501 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:36.728479    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-61f0f9-79d778489d-xwjwz"]
Sep 09 16:47:38.608892 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:38.608859    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c6a8a6e-3e08-4b92-b544-d025ae443124" path="/var/lib/kubelet/pods/6c6a8a6e-3e08-4b92-b544-d025ae443124/volumes"
Sep 09 16:47:38.609271 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:38.609210    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a10d7d61-e121-4188-8891-f94995e314ed" path="/var/lib/kubelet/pods/a10d7d61-e121-4188-8891-f94995e314ed/volumes"
Sep 09 16:47:46.495642 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:46.495604    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:47:55.140981 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:47:55.140950    2555 scope.go:117] "RemoveContainer" containerID="6330ef6c52728b18ab8594a18c3da329b9eee9214d8e7c6a973594e2ce175b7b"
Sep 09 16:48:35.338780 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:35.338743    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq"]
Sep 09 16:48:35.339686 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:35.339641    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq" podUID="56aee8a7-9d83-4ca2-a7e0-e4173caa1070" containerName="maas-api" containerID="cri-o://012ac724df3a971525cd49dad1754bedbd3906aa2e71f83688653ddcdfced8b1" gracePeriod=30
Sep 09 16:48:35.381521 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:35.381488    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27"]
Sep 09 16:48:35.381762 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:35.381729    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27" podUID="d38988e9-be38-4e83-b53e-379ecffd2c3b" containerName="payload-processing" containerID="cri-o://63c8d0562840ce65895d6a2d4749f2ff579f8d29874aa60efa204717fdb45624" gracePeriod=30
Sep 09 16:48:35.636919 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:35.636895    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq"
Sep 09 16:48:35.657361 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:35.657331    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56aee8a7-9d83-4ca2-a7e0-e4173caa1070-openshift-service-ca\") pod \"56aee8a7-9d83-4ca2-a7e0-e4173caa1070\" (UID: \"56aee8a7-9d83-4ca2-a7e0-e4173caa1070\") "
Sep 09 16:48:35.657475 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:35.657382    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56aee8a7-9d83-4ca2-a7e0-e4173caa1070-maas-api-tls\") pod \"56aee8a7-9d83-4ca2-a7e0-e4173caa1070\" (UID: \"56aee8a7-9d83-4ca2-a7e0-e4173caa1070\") "
Sep 09 16:48:35.657551 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:35.657475    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56aee8a7-9d83-4ca2-a7e0-e4173caa1070-metrics-certs\") pod \"56aee8a7-9d83-4ca2-a7e0-e4173caa1070\" (UID: \"56aee8a7-9d83-4ca2-a7e0-e4173caa1070\") "
Sep 09 16:48:35.657610 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:35.657562    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f555h\" (UniqueName: \"kubernetes.io/projected/56aee8a7-9d83-4ca2-a7e0-e4173caa1070-kube-api-access-f555h\") pod \"56aee8a7-9d83-4ca2-a7e0-e4173caa1070\" (UID: \"56aee8a7-9d83-4ca2-a7e0-e4173caa1070\") "
Sep 09 16:48:35.658567 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:35.658509    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56aee8a7-9d83-4ca2-a7e0-e4173caa1070-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "56aee8a7-9d83-4ca2-a7e0-e4173caa1070" (UID: "56aee8a7-9d83-4ca2-a7e0-e4173caa1070"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:48:35.660622 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:35.660541    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56aee8a7-9d83-4ca2-a7e0-e4173caa1070-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "56aee8a7-9d83-4ca2-a7e0-e4173caa1070" (UID: "56aee8a7-9d83-4ca2-a7e0-e4173caa1070"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:48:35.661091 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:35.661058    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56aee8a7-9d83-4ca2-a7e0-e4173caa1070-kube-api-access-f555h" (OuterVolumeSpecName: "kube-api-access-f555h") pod "56aee8a7-9d83-4ca2-a7e0-e4173caa1070" (UID: "56aee8a7-9d83-4ca2-a7e0-e4173caa1070"). InnerVolumeSpecName "kube-api-access-f555h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:48:35.662674 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:35.662647    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56aee8a7-9d83-4ca2-a7e0-e4173caa1070-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "56aee8a7-9d83-4ca2-a7e0-e4173caa1070" (UID: "56aee8a7-9d83-4ca2-a7e0-e4173caa1070"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:48:35.665955 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:35.665935    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27"
Sep 09 16:48:35.758943 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:35.758911    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsnjr\" (UniqueName: \"kubernetes.io/projected/d38988e9-be38-4e83-b53e-379ecffd2c3b-kube-api-access-hsnjr\") pod \"d38988e9-be38-4e83-b53e-379ecffd2c3b\" (UID: \"d38988e9-be38-4e83-b53e-379ecffd2c3b\") "
Sep 09 16:48:35.759136 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:35.758977    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38988e9-be38-4e83-b53e-379ecffd2c3b-plugins-config-volume\") pod \"d38988e9-be38-4e83-b53e-379ecffd2c3b\" (UID: \"d38988e9-be38-4e83-b53e-379ecffd2c3b\") "
Sep 09 16:48:35.759210 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:35.759201    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56aee8a7-9d83-4ca2-a7e0-e4173caa1070-metrics-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:48:35.759270 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:35.759214    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f555h\" (UniqueName: \"kubernetes.io/projected/56aee8a7-9d83-4ca2-a7e0-e4173caa1070-kube-api-access-f555h\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:48:35.759270 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:35.759223    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56aee8a7-9d83-4ca2-a7e0-e4173caa1070-openshift-service-ca\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:48:35.759270 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:35.759231    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56aee8a7-9d83-4ca2-a7e0-e4173caa1070-maas-api-tls\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:48:35.759494 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:35.759470    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d38988e9-be38-4e83-b53e-379ecffd2c3b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d38988e9-be38-4e83-b53e-379ecffd2c3b" (UID: "d38988e9-be38-4e83-b53e-379ecffd2c3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:48:35.761027 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:35.761004    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38988e9-be38-4e83-b53e-379ecffd2c3b-kube-api-access-hsnjr" (OuterVolumeSpecName: "kube-api-access-hsnjr") pod "d38988e9-be38-4e83-b53e-379ecffd2c3b" (UID: "d38988e9-be38-4e83-b53e-379ecffd2c3b"). InnerVolumeSpecName "kube-api-access-hsnjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:48:35.859735 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:35.859705    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsnjr\" (UniqueName: \"kubernetes.io/projected/d38988e9-be38-4e83-b53e-379ecffd2c3b-kube-api-access-hsnjr\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:48:35.859858 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:35.859731    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38988e9-be38-4e83-b53e-379ecffd2c3b-plugins-config-volume\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:48:36.035575 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:36.035542    2555 generic.go:356] "Generic (PLEG): container finished" podID="56aee8a7-9d83-4ca2-a7e0-e4173caa1070" containerID="012ac724df3a971525cd49dad1754bedbd3906aa2e71f83688653ddcdfced8b1" exitCode=0
Sep 09 16:48:36.035760 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:36.035589    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq"
Sep 09 16:48:36.035760 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:36.035623    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq" event={"ID":"56aee8a7-9d83-4ca2-a7e0-e4173caa1070","Type":"ContainerDied","Data":"012ac724df3a971525cd49dad1754bedbd3906aa2e71f83688653ddcdfced8b1"}
Sep 09 16:48:36.035760 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:36.035673    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq" event={"ID":"56aee8a7-9d83-4ca2-a7e0-e4173caa1070","Type":"ContainerDied","Data":"2d26b66367b272be2bdc3e78ac30e41f3436e3655d89d1ea8b4a50562967f3f0"}
Sep 09 16:48:36.035760 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:36.035699    2555 scope.go:117] "RemoveContainer" containerID="012ac724df3a971525cd49dad1754bedbd3906aa2e71f83688653ddcdfced8b1"
Sep 09 16:48:36.037079 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:36.037056    2555 generic.go:356] "Generic (PLEG): container finished" podID="d38988e9-be38-4e83-b53e-379ecffd2c3b" containerID="63c8d0562840ce65895d6a2d4749f2ff579f8d29874aa60efa204717fdb45624" exitCode=0
Sep 09 16:48:36.037194 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:36.037104    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27"
Sep 09 16:48:36.037194 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:36.037104    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27" event={"ID":"d38988e9-be38-4e83-b53e-379ecffd2c3b","Type":"ContainerDied","Data":"63c8d0562840ce65895d6a2d4749f2ff579f8d29874aa60efa204717fdb45624"}
Sep 09 16:48:36.037329 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:36.037222    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27" event={"ID":"d38988e9-be38-4e83-b53e-379ecffd2c3b","Type":"ContainerDied","Data":"40d34f0cda88137bcac8c820b191dd3ca0efead9ff6b247e5e534e344cc360c2"}
Sep 09 16:48:36.046819 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:36.046799    2555 scope.go:117] "RemoveContainer" containerID="012ac724df3a971525cd49dad1754bedbd3906aa2e71f83688653ddcdfced8b1"
Sep 09 16:48:36.047115 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:48:36.047097    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"012ac724df3a971525cd49dad1754bedbd3906aa2e71f83688653ddcdfced8b1\": container with ID starting with 012ac724df3a971525cd49dad1754bedbd3906aa2e71f83688653ddcdfced8b1 not found: ID does not exist" containerID="012ac724df3a971525cd49dad1754bedbd3906aa2e71f83688653ddcdfced8b1"
Sep 09 16:48:36.047208 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:36.047122    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"012ac724df3a971525cd49dad1754bedbd3906aa2e71f83688653ddcdfced8b1"} err="failed to get container status \"012ac724df3a971525cd49dad1754bedbd3906aa2e71f83688653ddcdfced8b1\": rpc error: code = NotFound desc = could not find container \"012ac724df3a971525cd49dad1754bedbd3906aa2e71f83688653ddcdfced8b1\": container with ID starting with 012ac724df3a971525cd49dad1754bedbd3906aa2e71f83688653ddcdfced8b1 not found: ID does not exist"
Sep 09 16:48:36.047208 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:36.047143    2555 scope.go:117] "RemoveContainer" containerID="63c8d0562840ce65895d6a2d4749f2ff579f8d29874aa60efa204717fdb45624"
Sep 09 16:48:36.056120 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:36.056103    2555 scope.go:117] "RemoveContainer" containerID="63c8d0562840ce65895d6a2d4749f2ff579f8d29874aa60efa204717fdb45624"
Sep 09 16:48:36.056426 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:48:36.056407    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63c8d0562840ce65895d6a2d4749f2ff579f8d29874aa60efa204717fdb45624\": container with ID starting with 63c8d0562840ce65895d6a2d4749f2ff579f8d29874aa60efa204717fdb45624 not found: ID does not exist" containerID="63c8d0562840ce65895d6a2d4749f2ff579f8d29874aa60efa204717fdb45624"
Sep 09 16:48:36.056480 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:36.056431    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63c8d0562840ce65895d6a2d4749f2ff579f8d29874aa60efa204717fdb45624"} err="failed to get container status \"63c8d0562840ce65895d6a2d4749f2ff579f8d29874aa60efa204717fdb45624\": rpc error: code = NotFound desc = could not find container \"63c8d0562840ce65895d6a2d4749f2ff579f8d29874aa60efa204717fdb45624\": container with ID starting with 63c8d0562840ce65895d6a2d4749f2ff579f8d29874aa60efa204717fdb45624 not found: ID does not exist"
Sep 09 16:48:36.060452 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:36.060428    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27"]
Sep 09 16:48:36.062815 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:36.062795    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cd0c91-748b475b8c-gns27"]
Sep 09 16:48:36.070575 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:36.070554    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq"]
Sep 09 16:48:36.074830 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:36.074812    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd0c91-998f7f5b5-9rwnq"]
Sep 09 16:48:36.608119 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:36.608035    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56aee8a7-9d83-4ca2-a7e0-e4173caa1070" path="/var/lib/kubelet/pods/56aee8a7-9d83-4ca2-a7e0-e4173caa1070/volumes"
Sep 09 16:48:36.608527 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:36.608437    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d38988e9-be38-4e83-b53e-379ecffd2c3b" path="/var/lib/kubelet/pods/d38988e9-be38-4e83-b53e-379ecffd2c3b/volumes"
Sep 09 16:48:38.691001 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:38.690968    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:48:52.890942 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:52.890907    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk"]
Sep 09 16:48:52.891436 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:52.891185    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk" podUID="455643fc-fc8d-486d-b035-0c2d9b482612" containerName="payload-processing" containerID="cri-o://de00597574f27c829dc84aee60b25e55017b6c7d83c1b6be08e2754ca8322c2f" gracePeriod=30
Sep 09 16:48:53.142053 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:53.141991    2555 generic.go:356] "Generic (PLEG): container finished" podID="455643fc-fc8d-486d-b035-0c2d9b482612" containerID="de00597574f27c829dc84aee60b25e55017b6c7d83c1b6be08e2754ca8322c2f" exitCode=0
Sep 09 16:48:53.142053 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:53.142043    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk" event={"ID":"455643fc-fc8d-486d-b035-0c2d9b482612","Type":"ContainerDied","Data":"de00597574f27c829dc84aee60b25e55017b6c7d83c1b6be08e2754ca8322c2f"}
Sep 09 16:48:53.142216 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:53.142104    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk" event={"ID":"455643fc-fc8d-486d-b035-0c2d9b482612","Type":"ContainerDied","Data":"291ec6eea29630634ac11e99b4847efba6d58d1b99733245a32bc9a334296a8f"}
Sep 09 16:48:53.142216 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:53.142126    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="291ec6eea29630634ac11e99b4847efba6d58d1b99733245a32bc9a334296a8f"
Sep 09 16:48:53.153134 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:53.153114    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk"
Sep 09 16:48:53.210921 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:53.210895    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm79c\" (UniqueName: \"kubernetes.io/projected/455643fc-fc8d-486d-b035-0c2d9b482612-kube-api-access-zm79c\") pod \"455643fc-fc8d-486d-b035-0c2d9b482612\" (UID: \"455643fc-fc8d-486d-b035-0c2d9b482612\") "
Sep 09 16:48:53.211058 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:53.210978    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455643fc-fc8d-486d-b035-0c2d9b482612-plugins-config-volume\") pod \"455643fc-fc8d-486d-b035-0c2d9b482612\" (UID: \"455643fc-fc8d-486d-b035-0c2d9b482612\") "
Sep 09 16:48:53.211275 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:53.211258    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/455643fc-fc8d-486d-b035-0c2d9b482612-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "455643fc-fc8d-486d-b035-0c2d9b482612" (UID: "455643fc-fc8d-486d-b035-0c2d9b482612"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:48:53.212904 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:53.212879    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/455643fc-fc8d-486d-b035-0c2d9b482612-kube-api-access-zm79c" (OuterVolumeSpecName: "kube-api-access-zm79c") pod "455643fc-fc8d-486d-b035-0c2d9b482612" (UID: "455643fc-fc8d-486d-b035-0c2d9b482612"). InnerVolumeSpecName "kube-api-access-zm79c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:48:53.312266 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:53.312233    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455643fc-fc8d-486d-b035-0c2d9b482612-plugins-config-volume\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:48:53.312266 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:53.312260    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm79c\" (UniqueName: \"kubernetes.io/projected/455643fc-fc8d-486d-b035-0c2d9b482612-kube-api-access-zm79c\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:48:54.148437 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:54.148403    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk"
Sep 09 16:48:54.170428 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:54.170399    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk"]
Sep 09 16:48:54.173979 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:54.173959    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-331bf8-859b85b5cc-wwzxk"]
Sep 09 16:48:54.607871 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:54.607841    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="455643fc-fc8d-486d-b035-0c2d9b482612" path="/var/lib/kubelet/pods/455643fc-fc8d-486d-b035-0c2d9b482612/volumes"
Sep 09 16:48:54.753208 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:54.753178    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgcjl_561d9dd7-04c3-42dc-bcf6-2c92fd61706b/console-operator/1.log"
Sep 09 16:48:54.763065 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:54.763045    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgcjl_561d9dd7-04c3-42dc-bcf6-2c92fd61706b/console-operator/1.log"
Sep 09 16:48:55.179494 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:55.179467    2555 scope.go:117] "RemoveContainer" containerID="de00597574f27c829dc84aee60b25e55017b6c7d83c1b6be08e2754ca8322c2f"
Sep 09 16:48:55.816459 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:55.811255    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:48:59.340868 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:59.340834    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"]
Sep 09 16:48:59.341358 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:48:59.341066    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76" podUID="a3e0ba21-b30d-40e3-8950-46e7e68958ec" containerName="istio-proxy" containerID="cri-o://89ac5ce27a743d5ba7f988818bd04328b2da6862a5d3308898325d8be9d46fae" gracePeriod=30
Sep 09 16:49:04.590212 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.590184    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:49:04.714915 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.714813    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istio-podinfo\") pod \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") "
Sep 09 16:49:04.714915 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.714884    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-workload-certs\") pod \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") "
Sep 09 16:49:04.714915 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.714907    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istio-data\") pod \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") "
Sep 09 16:49:04.715206 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.714940    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istio-envoy\") pod \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") "
Sep 09 16:49:04.715206 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.714973    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-credential-socket\") pod \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") "
Sep 09 16:49:04.715206 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.715016    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istio-token\") pod \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") "
Sep 09 16:49:04.715206 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.715049    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istiod-ca-cert\") pod \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") "
Sep 09 16:49:04.715206 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.715082    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdprs\" (UniqueName: \"kubernetes.io/projected/a3e0ba21-b30d-40e3-8950-46e7e68958ec-kube-api-access-jdprs\") pod \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") "
Sep 09 16:49:04.715206 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.715140    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-workload-socket\") pod \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\" (UID: \"a3e0ba21-b30d-40e3-8950-46e7e68958ec\") "
Sep 09 16:49:04.715534 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.715235    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istio-data" (OuterVolumeSpecName: "istio-data") pod "a3e0ba21-b30d-40e3-8950-46e7e68958ec" (UID: "a3e0ba21-b30d-40e3-8950-46e7e68958ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:49:04.715534 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.715250    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a3e0ba21-b30d-40e3-8950-46e7e68958ec" (UID: "a3e0ba21-b30d-40e3-8950-46e7e68958ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:49:04.715534 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.715372    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a3e0ba21-b30d-40e3-8950-46e7e68958ec" (UID: "a3e0ba21-b30d-40e3-8950-46e7e68958ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:49:04.715534 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.715490    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a3e0ba21-b30d-40e3-8950-46e7e68958ec" (UID: "a3e0ba21-b30d-40e3-8950-46e7e68958ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:49:04.715701 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.715571    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-workload-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:49:04.715701 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.715580    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a3e0ba21-b30d-40e3-8950-46e7e68958ec" (UID: "a3e0ba21-b30d-40e3-8950-46e7e68958ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:49:04.715701 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.715590    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istio-data\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:49:04.715701 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.715604    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-credential-socket\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:49:04.715701 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.715617    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-workload-socket\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:49:04.717120 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.717083    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a3e0ba21-b30d-40e3-8950-46e7e68958ec" (UID: "a3e0ba21-b30d-40e3-8950-46e7e68958ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:49:04.717239 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.717215    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e0ba21-b30d-40e3-8950-46e7e68958ec-kube-api-access-jdprs" (OuterVolumeSpecName: "kube-api-access-jdprs") pod "a3e0ba21-b30d-40e3-8950-46e7e68958ec" (UID: "a3e0ba21-b30d-40e3-8950-46e7e68958ec"). InnerVolumeSpecName "kube-api-access-jdprs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:49:04.717475 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.717454    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a3e0ba21-b30d-40e3-8950-46e7e68958ec" (UID: "a3e0ba21-b30d-40e3-8950-46e7e68958ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 16:49:04.717548 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.717528    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istio-token" (OuterVolumeSpecName: "istio-token") pod "a3e0ba21-b30d-40e3-8950-46e7e68958ec" (UID: "a3e0ba21-b30d-40e3-8950-46e7e68958ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:49:04.816477 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.816425    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istio-token\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:49:04.816477 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.816469    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istiod-ca-cert\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:49:04.816477 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.816481    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdprs\" (UniqueName: \"kubernetes.io/projected/a3e0ba21-b30d-40e3-8950-46e7e68958ec-kube-api-access-jdprs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:49:04.816477 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.816491    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istio-podinfo\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:49:04.816477 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:04.816500    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3e0ba21-b30d-40e3-8950-46e7e68958ec-istio-envoy\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:49:05.210923 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:05.210889    2555 generic.go:356] "Generic (PLEG): container finished" podID="a3e0ba21-b30d-40e3-8950-46e7e68958ec" containerID="89ac5ce27a743d5ba7f988818bd04328b2da6862a5d3308898325d8be9d46fae" exitCode=0
Sep 09 16:49:05.211129 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:05.210936    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"
Sep 09 16:49:05.211129 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:05.210982    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76" event={"ID":"a3e0ba21-b30d-40e3-8950-46e7e68958ec","Type":"ContainerDied","Data":"89ac5ce27a743d5ba7f988818bd04328b2da6862a5d3308898325d8be9d46fae"}
Sep 09 16:49:05.211129 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:05.211042    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76" event={"ID":"a3e0ba21-b30d-40e3-8950-46e7e68958ec","Type":"ContainerDied","Data":"0c8723a7c6aa3285f719060e50578eef34d480658ee2c05873626d06c3ea3074"}
Sep 09 16:49:05.211129 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:05.211071    2555 scope.go:117] "RemoveContainer" containerID="89ac5ce27a743d5ba7f988818bd04328b2da6862a5d3308898325d8be9d46fae"
Sep 09 16:49:05.221067 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:05.221050    2555 scope.go:117] "RemoveContainer" containerID="75a1f82de876d72d188f9dff5a21645b296ee0deb4669fc3ee245e68810104a7"
Sep 09 16:49:05.228669 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:05.228652    2555 scope.go:117] "RemoveContainer" containerID="89ac5ce27a743d5ba7f988818bd04328b2da6862a5d3308898325d8be9d46fae"
Sep 09 16:49:05.228900 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:49:05.228885    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ac5ce27a743d5ba7f988818bd04328b2da6862a5d3308898325d8be9d46fae\": container with ID starting with 89ac5ce27a743d5ba7f988818bd04328b2da6862a5d3308898325d8be9d46fae not found: ID does not exist" containerID="89ac5ce27a743d5ba7f988818bd04328b2da6862a5d3308898325d8be9d46fae"
Sep 09 16:49:05.228940 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:05.228907    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ac5ce27a743d5ba7f988818bd04328b2da6862a5d3308898325d8be9d46fae"} err="failed to get container status \"89ac5ce27a743d5ba7f988818bd04328b2da6862a5d3308898325d8be9d46fae\": rpc error: code = NotFound desc = could not find container \"89ac5ce27a743d5ba7f988818bd04328b2da6862a5d3308898325d8be9d46fae\": container with ID starting with 89ac5ce27a743d5ba7f988818bd04328b2da6862a5d3308898325d8be9d46fae not found: ID does not exist"
Sep 09 16:49:05.228940 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:05.228922    2555 scope.go:117] "RemoveContainer" containerID="75a1f82de876d72d188f9dff5a21645b296ee0deb4669fc3ee245e68810104a7"
Sep 09 16:49:05.229156 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:49:05.229139    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75a1f82de876d72d188f9dff5a21645b296ee0deb4669fc3ee245e68810104a7\": container with ID starting with 75a1f82de876d72d188f9dff5a21645b296ee0deb4669fc3ee245e68810104a7 not found: ID does not exist" containerID="75a1f82de876d72d188f9dff5a21645b296ee0deb4669fc3ee245e68810104a7"
Sep 09 16:49:05.229197 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:05.229162    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a1f82de876d72d188f9dff5a21645b296ee0deb4669fc3ee245e68810104a7"} err="failed to get container status \"75a1f82de876d72d188f9dff5a21645b296ee0deb4669fc3ee245e68810104a7\": rpc error: code = NotFound desc = could not find container \"75a1f82de876d72d188f9dff5a21645b296ee0deb4669fc3ee245e68810104a7\": container with ID starting with 75a1f82de876d72d188f9dff5a21645b296ee0deb4669fc3ee245e68810104a7 not found: ID does not exist"
Sep 09 16:49:05.233230 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:05.233210    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"]
Sep 09 16:49:05.237041 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:05.237022    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-331bf8-openshift-default-7699746b54-cmz76"]
Sep 09 16:49:06.607974 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:06.607936    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3e0ba21-b30d-40e3-8950-46e7e68958ec" path="/var/lib/kubelet/pods/a3e0ba21-b30d-40e3-8950-46e7e68958ec/volumes"
Sep 09 16:49:10.104661 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:10.104630    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-50fcbc-29816205-cvfzp"]
Sep 09 16:49:10.106661 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:10.106636    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-50fcbc-29816205-cvfzp"]
Sep 09 16:49:10.129778 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:10.129749    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x"]
Sep 09 16:49:10.130034 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:10.129993    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x" podUID="0241646d-a637-4ddf-9c1e-e9e42314ac97" containerName="payload-processing" containerID="cri-o://c1d88d4fceb84bd04dc0aa86ba6e161dd04bd6564974f1c9743108bd8e49694b" gracePeriod=30
Sep 09 16:49:10.138672 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:10.138646    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff"]
Sep 09 16:49:10.138890 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:10.138860    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff" podUID="f50bdeb2-c287-4b13-af6f-2a8b29d91469" containerName="payload-pre-processing" containerID="cri-o://8880099fa4367bbaf5113fffbcdb39cd35f45daf812c57f817ff47ba90faa79c" gracePeriod=30
Sep 09 16:49:10.416443 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:10.416422    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x"
Sep 09 16:49:10.418982 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:10.418965    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff"
Sep 09 16:49:10.568392 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:10.568352    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djgsk\" (UniqueName: \"kubernetes.io/projected/f50bdeb2-c287-4b13-af6f-2a8b29d91469-kube-api-access-djgsk\") pod \"f50bdeb2-c287-4b13-af6f-2a8b29d91469\" (UID: \"f50bdeb2-c287-4b13-af6f-2a8b29d91469\") "
Sep 09 16:49:10.568573 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:10.568405    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0241646d-a637-4ddf-9c1e-e9e42314ac97-plugins-config-volume\") pod \"0241646d-a637-4ddf-9c1e-e9e42314ac97\" (UID: \"0241646d-a637-4ddf-9c1e-e9e42314ac97\") "
Sep 09 16:49:10.568573 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:10.568467    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f50bdeb2-c287-4b13-af6f-2a8b29d91469-plugins-config-volume\") pod \"f50bdeb2-c287-4b13-af6f-2a8b29d91469\" (UID: \"f50bdeb2-c287-4b13-af6f-2a8b29d91469\") "
Sep 09 16:49:10.568573 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:10.568495    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c579f\" (UniqueName: \"kubernetes.io/projected/0241646d-a637-4ddf-9c1e-e9e42314ac97-kube-api-access-c579f\") pod \"0241646d-a637-4ddf-9c1e-e9e42314ac97\" (UID: \"0241646d-a637-4ddf-9c1e-e9e42314ac97\") "
Sep 09 16:49:10.568831 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:10.568800    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f50bdeb2-c287-4b13-af6f-2a8b29d91469-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f50bdeb2-c287-4b13-af6f-2a8b29d91469" (UID: "f50bdeb2-c287-4b13-af6f-2a8b29d91469"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:49:10.568831 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:10.568815    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0241646d-a637-4ddf-9c1e-e9e42314ac97-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0241646d-a637-4ddf-9c1e-e9e42314ac97" (UID: "0241646d-a637-4ddf-9c1e-e9e42314ac97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:49:10.570532 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:10.570510    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0241646d-a637-4ddf-9c1e-e9e42314ac97-kube-api-access-c579f" (OuterVolumeSpecName: "kube-api-access-c579f") pod "0241646d-a637-4ddf-9c1e-e9e42314ac97" (UID: "0241646d-a637-4ddf-9c1e-e9e42314ac97"). InnerVolumeSpecName "kube-api-access-c579f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:49:10.570621 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:10.570594    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f50bdeb2-c287-4b13-af6f-2a8b29d91469-kube-api-access-djgsk" (OuterVolumeSpecName: "kube-api-access-djgsk") pod "f50bdeb2-c287-4b13-af6f-2a8b29d91469" (UID: "f50bdeb2-c287-4b13-af6f-2a8b29d91469"). InnerVolumeSpecName "kube-api-access-djgsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:49:10.607471 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:10.607442    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad0c00ad-1883-44e7-8de5-dd4e396b5b05" path="/var/lib/kubelet/pods/ad0c00ad-1883-44e7-8de5-dd4e396b5b05/volumes"
Sep 09 16:49:10.669735 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:10.669672    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f50bdeb2-c287-4b13-af6f-2a8b29d91469-plugins-config-volume\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:49:10.669735 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:10.669694    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c579f\" (UniqueName: \"kubernetes.io/projected/0241646d-a637-4ddf-9c1e-e9e42314ac97-kube-api-access-c579f\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:49:10.669735 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:10.669703    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djgsk\" (UniqueName: \"kubernetes.io/projected/f50bdeb2-c287-4b13-af6f-2a8b29d91469-kube-api-access-djgsk\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:49:10.669735 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:10.669711    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0241646d-a637-4ddf-9c1e-e9e42314ac97-plugins-config-volume\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:49:11.244370 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:11.244331    2555 generic.go:356] "Generic (PLEG): container finished" podID="0241646d-a637-4ddf-9c1e-e9e42314ac97" containerID="c1d88d4fceb84bd04dc0aa86ba6e161dd04bd6564974f1c9743108bd8e49694b" exitCode=0
Sep 09 16:49:11.244815 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:11.244386    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x"
Sep 09 16:49:11.244815 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:11.244414    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x" event={"ID":"0241646d-a637-4ddf-9c1e-e9e42314ac97","Type":"ContainerDied","Data":"c1d88d4fceb84bd04dc0aa86ba6e161dd04bd6564974f1c9743108bd8e49694b"}
Sep 09 16:49:11.244815 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:11.244474    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x" event={"ID":"0241646d-a637-4ddf-9c1e-e9e42314ac97","Type":"ContainerDied","Data":"f9b709adca886507160c457312785ffbfe191bfb8e1c133947d91c8b486871f7"}
Sep 09 16:49:11.244815 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:11.244497    2555 scope.go:117] "RemoveContainer" containerID="c1d88d4fceb84bd04dc0aa86ba6e161dd04bd6564974f1c9743108bd8e49694b"
Sep 09 16:49:11.245985 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:11.245882    2555 generic.go:356] "Generic (PLEG): container finished" podID="f50bdeb2-c287-4b13-af6f-2a8b29d91469" containerID="8880099fa4367bbaf5113fffbcdb39cd35f45daf812c57f817ff47ba90faa79c" exitCode=0
Sep 09 16:49:11.245985 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:11.245936    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff" event={"ID":"f50bdeb2-c287-4b13-af6f-2a8b29d91469","Type":"ContainerDied","Data":"8880099fa4367bbaf5113fffbcdb39cd35f45daf812c57f817ff47ba90faa79c"}
Sep 09 16:49:11.245985 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:11.245974    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff" event={"ID":"f50bdeb2-c287-4b13-af6f-2a8b29d91469","Type":"ContainerDied","Data":"2dbc5b1cf0499d51451912a39636e5e37e3079bb47a94d6de4844994a4a040ab"}
Sep 09 16:49:11.245985 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:11.245984    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff"
Sep 09 16:49:11.256588 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:11.255389    2555 scope.go:117] "RemoveContainer" containerID="c1d88d4fceb84bd04dc0aa86ba6e161dd04bd6564974f1c9743108bd8e49694b"
Sep 09 16:49:11.256801 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:49:11.256667    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d88d4fceb84bd04dc0aa86ba6e161dd04bd6564974f1c9743108bd8e49694b\": container with ID starting with c1d88d4fceb84bd04dc0aa86ba6e161dd04bd6564974f1c9743108bd8e49694b not found: ID does not exist" containerID="c1d88d4fceb84bd04dc0aa86ba6e161dd04bd6564974f1c9743108bd8e49694b"
Sep 09 16:49:11.256801 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:11.256700    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d88d4fceb84bd04dc0aa86ba6e161dd04bd6564974f1c9743108bd8e49694b"} err="failed to get container status \"c1d88d4fceb84bd04dc0aa86ba6e161dd04bd6564974f1c9743108bd8e49694b\": rpc error: code = NotFound desc = could not find container \"c1d88d4fceb84bd04dc0aa86ba6e161dd04bd6564974f1c9743108bd8e49694b\": container with ID starting with c1d88d4fceb84bd04dc0aa86ba6e161dd04bd6564974f1c9743108bd8e49694b not found: ID does not exist"
Sep 09 16:49:11.256801 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:11.256722    2555 scope.go:117] "RemoveContainer" containerID="8880099fa4367bbaf5113fffbcdb39cd35f45daf812c57f817ff47ba90faa79c"
Sep 09 16:49:11.264882 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:11.264856    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x"]
Sep 09 16:49:11.265893 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:11.265878    2555 scope.go:117] "RemoveContainer" containerID="8880099fa4367bbaf5113fffbcdb39cd35f45daf812c57f817ff47ba90faa79c"
Sep 09 16:49:11.266193 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:49:11.266177    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8880099fa4367bbaf5113fffbcdb39cd35f45daf812c57f817ff47ba90faa79c\": container with ID starting with 8880099fa4367bbaf5113fffbcdb39cd35f45daf812c57f817ff47ba90faa79c not found: ID does not exist" containerID="8880099fa4367bbaf5113fffbcdb39cd35f45daf812c57f817ff47ba90faa79c"
Sep 09 16:49:11.266281 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:11.266199    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8880099fa4367bbaf5113fffbcdb39cd35f45daf812c57f817ff47ba90faa79c"} err="failed to get container status \"8880099fa4367bbaf5113fffbcdb39cd35f45daf812c57f817ff47ba90faa79c\": rpc error: code = NotFound desc = could not find container \"8880099fa4367bbaf5113fffbcdb39cd35f45daf812c57f817ff47ba90faa79c\": container with ID starting with 8880099fa4367bbaf5113fffbcdb39cd35f45daf812c57f817ff47ba90faa79c not found: ID does not exist"
Sep 09 16:49:11.267663 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:11.267641    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-50fcbc-66686df675-q7t6x"]
Sep 09 16:49:11.277127 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:11.277108    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff"]
Sep 09 16:49:11.281033 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:11.281014    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-50fcbc-7d748fd858-vh6ff"]
Sep 09 16:49:12.608143 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:12.608108    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0241646d-a637-4ddf-9c1e-e9e42314ac97" path="/var/lib/kubelet/pods/0241646d-a637-4ddf-9c1e-e9e42314ac97/volumes"
Sep 09 16:49:12.608520 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:12.608475    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f50bdeb2-c287-4b13-af6f-2a8b29d91469" path="/var/lib/kubelet/pods/f50bdeb2-c287-4b13-af6f-2a8b29d91469/volumes"
Sep 09 16:49:12.700053 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:12.700018    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:49:20.494991 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:49:20.494953    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:50:20.719226 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719191    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c78477868-rhw4w"]
Sep 09 16:50:20.719651 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719527    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d38988e9-be38-4e83-b53e-379ecffd2c3b" containerName="payload-processing"
Sep 09 16:50:20.719651 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719565    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d38988e9-be38-4e83-b53e-379ecffd2c3b" containerName="payload-processing"
Sep 09 16:50:20.719651 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719577    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f50bdeb2-c287-4b13-af6f-2a8b29d91469" containerName="payload-pre-processing"
Sep 09 16:50:20.719651 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719583    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f50bdeb2-c287-4b13-af6f-2a8b29d91469" containerName="payload-pre-processing"
Sep 09 16:50:20.719651 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719592    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a10d7d61-e121-4188-8891-f94995e314ed" containerName="payload-pre-processing"
Sep 09 16:50:20.719651 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719597    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a10d7d61-e121-4188-8891-f94995e314ed" containerName="payload-pre-processing"
Sep 09 16:50:20.719651 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719636    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56aee8a7-9d83-4ca2-a7e0-e4173caa1070" containerName="maas-api"
Sep 09 16:50:20.719651 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719645    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="56aee8a7-9d83-4ca2-a7e0-e4173caa1070" containerName="maas-api"
Sep 09 16:50:20.719651 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719653    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9149dd7-8eac-4cd1-8087-6c166a8989ea" containerName="payload-processing"
Sep 09 16:50:20.720090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719663    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9149dd7-8eac-4cd1-8087-6c166a8989ea" containerName="payload-processing"
Sep 09 16:50:20.720090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719672    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="455643fc-fc8d-486d-b035-0c2d9b482612" containerName="payload-processing"
Sep 09 16:50:20.720090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719684    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="455643fc-fc8d-486d-b035-0c2d9b482612" containerName="payload-processing"
Sep 09 16:50:20.720090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719690    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0241646d-a637-4ddf-9c1e-e9e42314ac97" containerName="payload-processing"
Sep 09 16:50:20.720090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719695    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0241646d-a637-4ddf-9c1e-e9e42314ac97" containerName="payload-processing"
Sep 09 16:50:20.720090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719706    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e0ba21-b30d-40e3-8950-46e7e68958ec" containerName="istio-proxy"
Sep 09 16:50:20.720090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719714    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e0ba21-b30d-40e3-8950-46e7e68958ec" containerName="istio-proxy"
Sep 09 16:50:20.720090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719722    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6a8a6e-3e08-4b92-b544-d025ae443124" containerName="payload-processing"
Sep 09 16:50:20.720090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719729    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6a8a6e-3e08-4b92-b544-d025ae443124" containerName="payload-processing"
Sep 09 16:50:20.720090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719738    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e0ba21-b30d-40e3-8950-46e7e68958ec" containerName="istio-proxy"
Sep 09 16:50:20.720090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719746    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e0ba21-b30d-40e3-8950-46e7e68958ec" containerName="istio-proxy"
Sep 09 16:50:20.720090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719753    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2bebda1-3586-4101-bd45-9f7f58b93e90" containerName="maas-api"
Sep 09 16:50:20.720090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719761    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2bebda1-3586-4101-bd45-9f7f58b93e90" containerName="maas-api"
Sep 09 16:50:20.720090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719855    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f50bdeb2-c287-4b13-af6f-2a8b29d91469" containerName="payload-pre-processing"
Sep 09 16:50:20.720090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719870    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="56aee8a7-9d83-4ca2-a7e0-e4173caa1070" containerName="maas-api"
Sep 09 16:50:20.720090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719879    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9149dd7-8eac-4cd1-8087-6c166a8989ea" containerName="payload-processing"
Sep 09 16:50:20.720090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719891    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0241646d-a637-4ddf-9c1e-e9e42314ac97" containerName="payload-processing"
Sep 09 16:50:20.720090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719900    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d38988e9-be38-4e83-b53e-379ecffd2c3b" containerName="payload-processing"
Sep 09 16:50:20.720090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719915    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c6a8a6e-3e08-4b92-b544-d025ae443124" containerName="payload-processing"
Sep 09 16:50:20.720090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719925    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="455643fc-fc8d-486d-b035-0c2d9b482612" containerName="payload-processing"
Sep 09 16:50:20.720090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719935    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e0ba21-b30d-40e3-8950-46e7e68958ec" containerName="istio-proxy"
Sep 09 16:50:20.720090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719943    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e0ba21-b30d-40e3-8950-46e7e68958ec" containerName="istio-proxy"
Sep 09 16:50:20.720090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719949    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2bebda1-3586-4101-bd45-9f7f58b93e90" containerName="maas-api"
Sep 09 16:50:20.720090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.719957    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a10d7d61-e121-4188-8891-f94995e314ed" containerName="payload-pre-processing"
Sep 09 16:50:20.723088 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.723073    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c78477868-rhw4w"
Sep 09 16:50:20.725788 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.725762    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 16:50:20.727014 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.726994    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nm5jk\""
Sep 09 16:50:20.727096 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.727056    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 16:50:20.727149 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.727056    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 16:50:20.732658 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.732636    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c78477868-rhw4w"]
Sep 09 16:50:20.771650 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.771618    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pp9x\" (UniqueName: \"kubernetes.io/projected/bffec614-0ddd-4a4e-955a-f3578de82c99-kube-api-access-9pp9x\") pod \"maas-controller-5c78477868-rhw4w\" (UID: \"bffec614-0ddd-4a4e-955a-f3578de82c99\") " pod="opendatahub/maas-controller-5c78477868-rhw4w"
Sep 09 16:50:20.771774 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.771665    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bffec614-0ddd-4a4e-955a-f3578de82c99-metrics-certs\") pod \"maas-controller-5c78477868-rhw4w\" (UID: \"bffec614-0ddd-4a4e-955a-f3578de82c99\") " pod="opendatahub/maas-controller-5c78477868-rhw4w"
Sep 09 16:50:20.771774 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.771702    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bffec614-0ddd-4a4e-955a-f3578de82c99-webhook-cert\") pod \"maas-controller-5c78477868-rhw4w\" (UID: \"bffec614-0ddd-4a4e-955a-f3578de82c99\") " pod="opendatahub/maas-controller-5c78477868-rhw4w"
Sep 09 16:50:20.872673 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.872644    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bffec614-0ddd-4a4e-955a-f3578de82c99-metrics-certs\") pod \"maas-controller-5c78477868-rhw4w\" (UID: \"bffec614-0ddd-4a4e-955a-f3578de82c99\") " pod="opendatahub/maas-controller-5c78477868-rhw4w"
Sep 09 16:50:20.872809 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.872688    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bffec614-0ddd-4a4e-955a-f3578de82c99-webhook-cert\") pod \"maas-controller-5c78477868-rhw4w\" (UID: \"bffec614-0ddd-4a4e-955a-f3578de82c99\") " pod="opendatahub/maas-controller-5c78477868-rhw4w"
Sep 09 16:50:20.872809 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.872734    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pp9x\" (UniqueName: \"kubernetes.io/projected/bffec614-0ddd-4a4e-955a-f3578de82c99-kube-api-access-9pp9x\") pod \"maas-controller-5c78477868-rhw4w\" (UID: \"bffec614-0ddd-4a4e-955a-f3578de82c99\") " pod="opendatahub/maas-controller-5c78477868-rhw4w"
Sep 09 16:50:20.875171 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.875142    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bffec614-0ddd-4a4e-955a-f3578de82c99-metrics-certs\") pod \"maas-controller-5c78477868-rhw4w\" (UID: \"bffec614-0ddd-4a4e-955a-f3578de82c99\") " pod="opendatahub/maas-controller-5c78477868-rhw4w"
Sep 09 16:50:20.875268 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.875142    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bffec614-0ddd-4a4e-955a-f3578de82c99-webhook-cert\") pod \"maas-controller-5c78477868-rhw4w\" (UID: \"bffec614-0ddd-4a4e-955a-f3578de82c99\") " pod="opendatahub/maas-controller-5c78477868-rhw4w"
Sep 09 16:50:20.880147 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:20.880119    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pp9x\" (UniqueName: \"kubernetes.io/projected/bffec614-0ddd-4a4e-955a-f3578de82c99-kube-api-access-9pp9x\") pod \"maas-controller-5c78477868-rhw4w\" (UID: \"bffec614-0ddd-4a4e-955a-f3578de82c99\") " pod="opendatahub/maas-controller-5c78477868-rhw4w"
Sep 09 16:50:21.033337 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:21.033282    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c78477868-rhw4w"
Sep 09 16:50:21.300795 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:21.300767    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c78477868-rhw4w"]
Sep 09 16:50:21.302638 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:50:21.302609    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbffec614_0ddd_4a4e_955a_f3578de82c99.slice/crio-623cc7e9c67aab0cc8aefbb196192504fd4def8a0bd63a4f98ba0d3cab75757f WatchSource:0}: Error finding container 623cc7e9c67aab0cc8aefbb196192504fd4def8a0bd63a4f98ba0d3cab75757f: Status 404 returned error can't find the container with id 623cc7e9c67aab0cc8aefbb196192504fd4def8a0bd63a4f98ba0d3cab75757f
Sep 09 16:50:21.304009 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:21.303990    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 16:50:21.617382 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:21.617282    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c78477868-rhw4w" event={"ID":"bffec614-0ddd-4a4e-955a-f3578de82c99","Type":"ContainerStarted","Data":"623cc7e9c67aab0cc8aefbb196192504fd4def8a0bd63a4f98ba0d3cab75757f"}
Sep 09 16:50:22.623771 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:22.623688    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c78477868-rhw4w" event={"ID":"bffec614-0ddd-4a4e-955a-f3578de82c99","Type":"ContainerStarted","Data":"6a45ab66392df8bc479e1ceff491b316225504d3040d457033f06e073aa18134"}
Sep 09 16:50:22.624143 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:22.623810    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c78477868-rhw4w"
Sep 09 16:50:22.642561 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:22.642507    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5c78477868-rhw4w" podStartSLOduration=1.603167045 podStartE2EDuration="2.642490008s" totalImagesPullingTime="1.039322963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:50:22.638971951 +0000 UTC m=+2188.469180008" watchObservedRunningTime="2026-09-09 16:50:22.642490008 +0000 UTC m=+2188.472698068"
Sep 09 16:50:33.630427 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:33.630396    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c78477868-rhw4w"
Sep 09 16:50:43.036890 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:43.036850    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c78477868-rhw4w"]
Sep 09 16:50:43.037391 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:43.037114    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5c78477868-rhw4w" podUID="bffec614-0ddd-4a4e-955a-f3578de82c99" containerName="manager" containerID="cri-o://6a45ab66392df8bc479e1ceff491b316225504d3040d457033f06e073aa18134" gracePeriod=30
Sep 09 16:50:43.287860 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:43.287802    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c78477868-rhw4w"
Sep 09 16:50:43.365200 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:43.365173    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pp9x\" (UniqueName: \"kubernetes.io/projected/bffec614-0ddd-4a4e-955a-f3578de82c99-kube-api-access-9pp9x\") pod \"bffec614-0ddd-4a4e-955a-f3578de82c99\" (UID: \"bffec614-0ddd-4a4e-955a-f3578de82c99\") "
Sep 09 16:50:43.365365 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:43.365213    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bffec614-0ddd-4a4e-955a-f3578de82c99-metrics-certs\") pod \"bffec614-0ddd-4a4e-955a-f3578de82c99\" (UID: \"bffec614-0ddd-4a4e-955a-f3578de82c99\") "
Sep 09 16:50:43.365365 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:43.365287    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bffec614-0ddd-4a4e-955a-f3578de82c99-webhook-cert\") pod \"bffec614-0ddd-4a4e-955a-f3578de82c99\" (UID: \"bffec614-0ddd-4a4e-955a-f3578de82c99\") "
Sep 09 16:50:43.367223 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:43.367191    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bffec614-0ddd-4a4e-955a-f3578de82c99-kube-api-access-9pp9x" (OuterVolumeSpecName: "kube-api-access-9pp9x") pod "bffec614-0ddd-4a4e-955a-f3578de82c99" (UID: "bffec614-0ddd-4a4e-955a-f3578de82c99"). InnerVolumeSpecName "kube-api-access-9pp9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:50:43.367223 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:43.367213    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bffec614-0ddd-4a4e-955a-f3578de82c99-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "bffec614-0ddd-4a4e-955a-f3578de82c99" (UID: "bffec614-0ddd-4a4e-955a-f3578de82c99"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:50:43.367396 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:43.367195    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bffec614-0ddd-4a4e-955a-f3578de82c99-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bffec614-0ddd-4a4e-955a-f3578de82c99" (UID: "bffec614-0ddd-4a4e-955a-f3578de82c99"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:50:43.465993 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:43.465965    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pp9x\" (UniqueName: \"kubernetes.io/projected/bffec614-0ddd-4a4e-955a-f3578de82c99-kube-api-access-9pp9x\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:50:43.465993 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:43.465989    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bffec614-0ddd-4a4e-955a-f3578de82c99-metrics-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:50:43.465993 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:43.465998    2555 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bffec614-0ddd-4a4e-955a-f3578de82c99-webhook-cert\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:50:43.731898 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:43.731806    2555 generic.go:356] "Generic (PLEG): container finished" podID="bffec614-0ddd-4a4e-955a-f3578de82c99" containerID="6a45ab66392df8bc479e1ceff491b316225504d3040d457033f06e073aa18134" exitCode=0
Sep 09 16:50:43.731898 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:43.731858    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c78477868-rhw4w"
Sep 09 16:50:43.731898 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:43.731886    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c78477868-rhw4w" event={"ID":"bffec614-0ddd-4a4e-955a-f3578de82c99","Type":"ContainerDied","Data":"6a45ab66392df8bc479e1ceff491b316225504d3040d457033f06e073aa18134"}
Sep 09 16:50:43.732129 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:43.731932    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c78477868-rhw4w" event={"ID":"bffec614-0ddd-4a4e-955a-f3578de82c99","Type":"ContainerDied","Data":"623cc7e9c67aab0cc8aefbb196192504fd4def8a0bd63a4f98ba0d3cab75757f"}
Sep 09 16:50:43.732129 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:43.731956    2555 scope.go:117] "RemoveContainer" containerID="6a45ab66392df8bc479e1ceff491b316225504d3040d457033f06e073aa18134"
Sep 09 16:50:43.741855 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:43.741836    2555 scope.go:117] "RemoveContainer" containerID="6a45ab66392df8bc479e1ceff491b316225504d3040d457033f06e073aa18134"
Sep 09 16:50:43.742119 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:50:43.742100    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a45ab66392df8bc479e1ceff491b316225504d3040d457033f06e073aa18134\": container with ID starting with 6a45ab66392df8bc479e1ceff491b316225504d3040d457033f06e073aa18134 not found: ID does not exist" containerID="6a45ab66392df8bc479e1ceff491b316225504d3040d457033f06e073aa18134"
Sep 09 16:50:43.742177 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:43.742128    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a45ab66392df8bc479e1ceff491b316225504d3040d457033f06e073aa18134"} err="failed to get container status \"6a45ab66392df8bc479e1ceff491b316225504d3040d457033f06e073aa18134\": rpc error: code = NotFound desc = could not find container \"6a45ab66392df8bc479e1ceff491b316225504d3040d457033f06e073aa18134\": container with ID starting with 6a45ab66392df8bc479e1ceff491b316225504d3040d457033f06e073aa18134 not found: ID does not exist"
Sep 09 16:50:43.752697 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:43.752674    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c78477868-rhw4w"]
Sep 09 16:50:43.759063 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:43.759039    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5c78477868-rhw4w"]
Sep 09 16:50:43.792611 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:43.792579    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:50:44.569311 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:44.569260    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c78477868-q462t"]
Sep 09 16:50:44.569832 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:44.569812    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bffec614-0ddd-4a4e-955a-f3578de82c99" containerName="manager"
Sep 09 16:50:44.569888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:44.569839    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bffec614-0ddd-4a4e-955a-f3578de82c99" containerName="manager"
Sep 09 16:50:44.569937 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:44.569922    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bffec614-0ddd-4a4e-955a-f3578de82c99" containerName="manager"
Sep 09 16:50:44.574489 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:44.574463    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c78477868-q462t"
Sep 09 16:50:44.578414 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:44.578382    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 16:50:44.578538 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:44.578423    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nm5jk\""
Sep 09 16:50:44.578538 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:44.578456    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 16:50:44.578538 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:44.578382    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 16:50:44.582513 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:44.582494    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c78477868-q462t"]
Sep 09 16:50:44.610455 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:44.610418    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bffec614-0ddd-4a4e-955a-f3578de82c99" path="/var/lib/kubelet/pods/bffec614-0ddd-4a4e-955a-f3578de82c99/volumes"
Sep 09 16:50:44.675600 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:44.675565    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ef1c0123-b6a7-40d3-a947-eb7403570bbd-webhook-cert\") pod \"maas-controller-5c78477868-q462t\" (UID: \"ef1c0123-b6a7-40d3-a947-eb7403570bbd\") " pod="opendatahub/maas-controller-5c78477868-q462t"
Sep 09 16:50:44.675769 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:44.675678    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef1c0123-b6a7-40d3-a947-eb7403570bbd-metrics-certs\") pod \"maas-controller-5c78477868-q462t\" (UID: \"ef1c0123-b6a7-40d3-a947-eb7403570bbd\") " pod="opendatahub/maas-controller-5c78477868-q462t"
Sep 09 16:50:44.675769 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:44.675708    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxpmd\" (UniqueName: \"kubernetes.io/projected/ef1c0123-b6a7-40d3-a947-eb7403570bbd-kube-api-access-hxpmd\") pod \"maas-controller-5c78477868-q462t\" (UID: \"ef1c0123-b6a7-40d3-a947-eb7403570bbd\") " pod="opendatahub/maas-controller-5c78477868-q462t"
Sep 09 16:50:44.776431 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:44.776403    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ef1c0123-b6a7-40d3-a947-eb7403570bbd-webhook-cert\") pod \"maas-controller-5c78477868-q462t\" (UID: \"ef1c0123-b6a7-40d3-a947-eb7403570bbd\") " pod="opendatahub/maas-controller-5c78477868-q462t"
Sep 09 16:50:44.776609 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:44.776536    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef1c0123-b6a7-40d3-a947-eb7403570bbd-metrics-certs\") pod \"maas-controller-5c78477868-q462t\" (UID: \"ef1c0123-b6a7-40d3-a947-eb7403570bbd\") " pod="opendatahub/maas-controller-5c78477868-q462t"
Sep 09 16:50:44.776609 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:44.776567    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxpmd\" (UniqueName: \"kubernetes.io/projected/ef1c0123-b6a7-40d3-a947-eb7403570bbd-kube-api-access-hxpmd\") pod \"maas-controller-5c78477868-q462t\" (UID: \"ef1c0123-b6a7-40d3-a947-eb7403570bbd\") " pod="opendatahub/maas-controller-5c78477868-q462t"
Sep 09 16:50:44.778823 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:44.778802    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef1c0123-b6a7-40d3-a947-eb7403570bbd-metrics-certs\") pod \"maas-controller-5c78477868-q462t\" (UID: \"ef1c0123-b6a7-40d3-a947-eb7403570bbd\") " pod="opendatahub/maas-controller-5c78477868-q462t"
Sep 09 16:50:44.778961 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:44.778840    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ef1c0123-b6a7-40d3-a947-eb7403570bbd-webhook-cert\") pod \"maas-controller-5c78477868-q462t\" (UID: \"ef1c0123-b6a7-40d3-a947-eb7403570bbd\") " pod="opendatahub/maas-controller-5c78477868-q462t"
Sep 09 16:50:44.784442 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:44.784421    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxpmd\" (UniqueName: \"kubernetes.io/projected/ef1c0123-b6a7-40d3-a947-eb7403570bbd-kube-api-access-hxpmd\") pod \"maas-controller-5c78477868-q462t\" (UID: \"ef1c0123-b6a7-40d3-a947-eb7403570bbd\") " pod="opendatahub/maas-controller-5c78477868-q462t"
Sep 09 16:50:44.888821 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:44.888756    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c78477868-q462t"
Sep 09 16:50:45.156480 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:45.156453    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c78477868-q462t"]
Sep 09 16:50:45.157774 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:50:45.157752    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1c0123_b6a7_40d3_a947_eb7403570bbd.slice/crio-7ecb47ee98e0e46586bf9db4877c4dba57d30ced3d6dbe96796784729597059d WatchSource:0}: Error finding container 7ecb47ee98e0e46586bf9db4877c4dba57d30ced3d6dbe96796784729597059d: Status 404 returned error can't find the container with id 7ecb47ee98e0e46586bf9db4877c4dba57d30ced3d6dbe96796784729597059d
Sep 09 16:50:45.745882 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:45.745841    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c78477868-q462t" event={"ID":"ef1c0123-b6a7-40d3-a947-eb7403570bbd","Type":"ContainerStarted","Data":"75afe2ce0e64774d8274f9e26a4b04ea274df40872cb2824fe3c2d27417fd17c"}
Sep 09 16:50:45.746241 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:45.745904    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c78477868-q462t" event={"ID":"ef1c0123-b6a7-40d3-a947-eb7403570bbd","Type":"ContainerStarted","Data":"7ecb47ee98e0e46586bf9db4877c4dba57d30ced3d6dbe96796784729597059d"}
Sep 09 16:50:45.746241 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:45.746044    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c78477868-q462t"
Sep 09 16:50:45.762480 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:45.762432    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5c78477868-q462t" podStartSLOduration=1.307328954 podStartE2EDuration="1.762420389s" totalImagesPullingTime="455.091435ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:50:45.760153494 +0000 UTC m=+2211.590361555" watchObservedRunningTime="2026-09-09 16:50:45.762420389 +0000 UTC m=+2211.592628444"
Sep 09 16:50:56.755233 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:50:56.755206    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c78477868-q462t"
Sep 09 16:51:08.789318 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:51:08.789271    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:51:16.398183 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:51:16.398141    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:51:23.186018 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:51:23.185974    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:51:40.597133 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:51:40.597096    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:51:46.395952 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:51:46.395916    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:51:48.213204 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:51:48.213173    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w5vbz"]
Sep 09 16:51:48.213615 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:51:48.213361    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w5vbz" podUID="b0ecae37-92af-4dfc-98d1-3741d1366000" containerName="manager" containerID="cri-o://8ef50e5d6b7f905a56799e4c56579f89ebaa6fc3905001f2618aca48560d5107" gracePeriod=10
Sep 09 16:51:49.083397 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:51:49.083361    2555 generic.go:356] "Generic (PLEG): container finished" podID="b0ecae37-92af-4dfc-98d1-3741d1366000" containerID="8ef50e5d6b7f905a56799e4c56579f89ebaa6fc3905001f2618aca48560d5107" exitCode=0
Sep 09 16:51:49.083561 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:51:49.083439    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w5vbz" event={"ID":"b0ecae37-92af-4dfc-98d1-3741d1366000","Type":"ContainerDied","Data":"8ef50e5d6b7f905a56799e4c56579f89ebaa6fc3905001f2618aca48560d5107"}
Sep 09 16:51:49.554459 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:51:49.554436    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w5vbz"
Sep 09 16:51:49.741167 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:51:49.741134    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jr472\" (UniqueName: \"kubernetes.io/projected/b0ecae37-92af-4dfc-98d1-3741d1366000-kube-api-access-jr472\") pod \"b0ecae37-92af-4dfc-98d1-3741d1366000\" (UID: \"b0ecae37-92af-4dfc-98d1-3741d1366000\") "
Sep 09 16:51:49.741385 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:51:49.741185    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0ecae37-92af-4dfc-98d1-3741d1366000-extensions-socket-volume\") pod \"b0ecae37-92af-4dfc-98d1-3741d1366000\" (UID: \"b0ecae37-92af-4dfc-98d1-3741d1366000\") "
Sep 09 16:51:49.741605 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:51:49.741580    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ecae37-92af-4dfc-98d1-3741d1366000-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b0ecae37-92af-4dfc-98d1-3741d1366000" (UID: "b0ecae37-92af-4dfc-98d1-3741d1366000"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:51:49.743254 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:51:49.743231    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ecae37-92af-4dfc-98d1-3741d1366000-kube-api-access-jr472" (OuterVolumeSpecName: "kube-api-access-jr472") pod "b0ecae37-92af-4dfc-98d1-3741d1366000" (UID: "b0ecae37-92af-4dfc-98d1-3741d1366000"). InnerVolumeSpecName "kube-api-access-jr472". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:51:49.842812 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:51:49.842776    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr472\" (UniqueName: \"kubernetes.io/projected/b0ecae37-92af-4dfc-98d1-3741d1366000-kube-api-access-jr472\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:51:49.842812 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:51:49.842806    2555 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0ecae37-92af-4dfc-98d1-3741d1366000-extensions-socket-volume\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:51:50.090515 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:51:50.090431    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w5vbz"
Sep 09 16:51:50.090660 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:51:50.090433    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w5vbz" event={"ID":"b0ecae37-92af-4dfc-98d1-3741d1366000","Type":"ContainerDied","Data":"4613fff4ec8f606d7f33bfac8cbc39a1b9b97d97daf84c82f173215f3228ff84"}
Sep 09 16:51:50.090660 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:51:50.090584    2555 scope.go:117] "RemoveContainer" containerID="8ef50e5d6b7f905a56799e4c56579f89ebaa6fc3905001f2618aca48560d5107"
Sep 09 16:51:50.111733 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:51:50.111708    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w5vbz"]
Sep 09 16:51:50.117542 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:51:50.117506    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w5vbz"]
Sep 09 16:51:50.607281 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:51:50.607246    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0ecae37-92af-4dfc-98d1-3741d1366000" path="/var/lib/kubelet/pods/b0ecae37-92af-4dfc-98d1-3741d1366000/volumes"
Sep 09 16:51:55.245179 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:51:55.245146    2555 scope.go:117] "RemoveContainer" containerID="38a9ff5e3627e23169b291f8ba389afec79f057daa0cd6c2478ac1041c280b0b"
Sep 09 16:52:48.966119 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:52:48.966040    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wkjvw"]
Sep 09 16:52:48.966622 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:52:48.966428    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0ecae37-92af-4dfc-98d1-3741d1366000" containerName="manager"
Sep 09 16:52:48.966622 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:52:48.966440    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0ecae37-92af-4dfc-98d1-3741d1366000" containerName="manager"
Sep 09 16:52:48.966622 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:52:48.966506    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0ecae37-92af-4dfc-98d1-3741d1366000" containerName="manager"
Sep 09 16:52:48.969454 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:52:48.969436    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wkjvw"
Sep 09 16:52:48.972094 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:52:48.972076    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bghjj\""
Sep 09 16:52:48.979586 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:52:48.979564    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wkjvw"]
Sep 09 16:52:49.033519 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:52:49.033489    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2hlj\" (UniqueName: \"kubernetes.io/projected/9f12cad5-5ad6-47e3-9b75-246392ca465c-kube-api-access-m2hlj\") pod \"kuadrant-operator-controller-manager-5769f99d9f-wkjvw\" (UID: \"9f12cad5-5ad6-47e3-9b75-246392ca465c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wkjvw"
Sep 09 16:52:49.033696 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:52:49.033535    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f12cad5-5ad6-47e3-9b75-246392ca465c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-wkjvw\" (UID: \"9f12cad5-5ad6-47e3-9b75-246392ca465c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wkjvw"
Sep 09 16:52:49.134453 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:52:49.134416    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2hlj\" (UniqueName: \"kubernetes.io/projected/9f12cad5-5ad6-47e3-9b75-246392ca465c-kube-api-access-m2hlj\") pod \"kuadrant-operator-controller-manager-5769f99d9f-wkjvw\" (UID: \"9f12cad5-5ad6-47e3-9b75-246392ca465c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wkjvw"
Sep 09 16:52:49.134629 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:52:49.134475    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f12cad5-5ad6-47e3-9b75-246392ca465c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-wkjvw\" (UID: \"9f12cad5-5ad6-47e3-9b75-246392ca465c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wkjvw"
Sep 09 16:52:49.134780 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:52:49.134765    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f12cad5-5ad6-47e3-9b75-246392ca465c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-wkjvw\" (UID: \"9f12cad5-5ad6-47e3-9b75-246392ca465c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wkjvw"
Sep 09 16:52:49.142738 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:52:49.142712    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2hlj\" (UniqueName: \"kubernetes.io/projected/9f12cad5-5ad6-47e3-9b75-246392ca465c-kube-api-access-m2hlj\") pod \"kuadrant-operator-controller-manager-5769f99d9f-wkjvw\" (UID: \"9f12cad5-5ad6-47e3-9b75-246392ca465c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wkjvw"
Sep 09 16:52:49.280120 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:52:49.280094    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wkjvw"
Sep 09 16:52:49.554700 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:52:49.554673    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wkjvw"]
Sep 09 16:52:49.556037 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:52:49.556012    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f12cad5_5ad6_47e3_9b75_246392ca465c.slice/crio-59082b77f77993f9ffe2d1645d768922d7cc16f7913de624f965a47e5e9f1dba WatchSource:0}: Error finding container 59082b77f77993f9ffe2d1645d768922d7cc16f7913de624f965a47e5e9f1dba: Status 404 returned error can't find the container with id 59082b77f77993f9ffe2d1645d768922d7cc16f7913de624f965a47e5e9f1dba
Sep 09 16:52:50.405857 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:52:50.405822    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wkjvw" event={"ID":"9f12cad5-5ad6-47e3-9b75-246392ca465c","Type":"ContainerStarted","Data":"a2ae6e6e915e1e39f629f2f0dc34f4e5d162a3a8884a9ea1b9be021cb39b5025"}
Sep 09 16:52:50.406260 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:52:50.405874    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wkjvw" event={"ID":"9f12cad5-5ad6-47e3-9b75-246392ca465c","Type":"ContainerStarted","Data":"59082b77f77993f9ffe2d1645d768922d7cc16f7913de624f965a47e5e9f1dba"}
Sep 09 16:52:50.406260 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:52:50.405919    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wkjvw"
Sep 09 16:52:50.434545 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:52:50.434488    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wkjvw" podStartSLOduration=2.434471599 podStartE2EDuration="2.434471599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:52:50.434022407 +0000 UTC m=+2336.264230468" watchObservedRunningTime="2026-09-09 16:52:50.434471599 +0000 UTC m=+2336.264679657"
Sep 09 16:53:01.413475 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:01.413441    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wkjvw"
Sep 09 16:53:14.194863 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:14.194822    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:53:21.096354 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:21.096309    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:53:37.301872 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:37.301833    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"]
Sep 09 16:53:37.305800 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:37.305779    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:53:37.309797 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:37.309761    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-585e4a6c-kserve-self-signed-certs\""
Sep 09 16:53:37.312969 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:37.312942    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"]
Sep 09 16:53:37.365100 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:37.365070    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c25eb85b-87e6-4786-9301-e79893b29412-tls-certs\") pod \"e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt\" (UID: \"c25eb85b-87e6-4786-9301-e79893b29412\") " pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:53:37.365236 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:37.365111    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c25eb85b-87e6-4786-9301-e79893b29412-model-cache\") pod \"e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt\" (UID: \"c25eb85b-87e6-4786-9301-e79893b29412\") " pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:53:37.365236 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:37.365136    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c25eb85b-87e6-4786-9301-e79893b29412-tmp-dir\") pod \"e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt\" (UID: \"c25eb85b-87e6-4786-9301-e79893b29412\") " pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:53:37.365236 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:37.365212    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c25eb85b-87e6-4786-9301-e79893b29412-home\") pod \"e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt\" (UID: \"c25eb85b-87e6-4786-9301-e79893b29412\") " pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:53:37.365388 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:37.365262    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn5r8\" (UniqueName: \"kubernetes.io/projected/c25eb85b-87e6-4786-9301-e79893b29412-kube-api-access-qn5r8\") pod \"e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt\" (UID: \"c25eb85b-87e6-4786-9301-e79893b29412\") " pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:53:37.365388 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:37.365355    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c25eb85b-87e6-4786-9301-e79893b29412-dshm\") pod \"e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt\" (UID: \"c25eb85b-87e6-4786-9301-e79893b29412\") " pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:53:37.466112 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:37.466078    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c25eb85b-87e6-4786-9301-e79893b29412-tls-certs\") pod \"e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt\" (UID: \"c25eb85b-87e6-4786-9301-e79893b29412\") " pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:53:37.466312 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:37.466134    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c25eb85b-87e6-4786-9301-e79893b29412-model-cache\") pod \"e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt\" (UID: \"c25eb85b-87e6-4786-9301-e79893b29412\") " pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:53:37.466312 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:37.466164    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c25eb85b-87e6-4786-9301-e79893b29412-tmp-dir\") pod \"e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt\" (UID: \"c25eb85b-87e6-4786-9301-e79893b29412\") " pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:53:37.466312 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:37.466210    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c25eb85b-87e6-4786-9301-e79893b29412-home\") pod \"e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt\" (UID: \"c25eb85b-87e6-4786-9301-e79893b29412\") " pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:53:37.466499 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:37.466288    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn5r8\" (UniqueName: \"kubernetes.io/projected/c25eb85b-87e6-4786-9301-e79893b29412-kube-api-access-qn5r8\") pod \"e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt\" (UID: \"c25eb85b-87e6-4786-9301-e79893b29412\") " pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:53:37.466579 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:37.466549    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c25eb85b-87e6-4786-9301-e79893b29412-dshm\") pod \"e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt\" (UID: \"c25eb85b-87e6-4786-9301-e79893b29412\") " pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:53:37.466737 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:37.466712    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c25eb85b-87e6-4786-9301-e79893b29412-model-cache\") pod \"e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt\" (UID: \"c25eb85b-87e6-4786-9301-e79893b29412\") " pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:53:37.466855 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:37.466834    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c25eb85b-87e6-4786-9301-e79893b29412-tmp-dir\") pod \"e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt\" (UID: \"c25eb85b-87e6-4786-9301-e79893b29412\") " pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:53:37.466919 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:37.466857    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c25eb85b-87e6-4786-9301-e79893b29412-home\") pod \"e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt\" (UID: \"c25eb85b-87e6-4786-9301-e79893b29412\") " pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:53:37.468850 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:37.468830    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c25eb85b-87e6-4786-9301-e79893b29412-dshm\") pod \"e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt\" (UID: \"c25eb85b-87e6-4786-9301-e79893b29412\") " pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:53:37.468927 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:37.468898    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c25eb85b-87e6-4786-9301-e79893b29412-tls-certs\") pod \"e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt\" (UID: \"c25eb85b-87e6-4786-9301-e79893b29412\") " pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:53:37.475669 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:37.475649    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn5r8\" (UniqueName: \"kubernetes.io/projected/c25eb85b-87e6-4786-9301-e79893b29412-kube-api-access-qn5r8\") pod \"e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt\" (UID: \"c25eb85b-87e6-4786-9301-e79893b29412\") " pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:53:37.618068 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:37.617997    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:53:37.888839 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:37.888811    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"]
Sep 09 16:53:37.890890 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:53:37.890864    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc25eb85b_87e6_4786_9301_e79893b29412.slice/crio-5344a78c2bf4ce95f8d02adef42baa4e8d6e3a2364c9899aba447032d4fcc428 WatchSource:0}: Error finding container 5344a78c2bf4ce95f8d02adef42baa4e8d6e3a2364c9899aba447032d4fcc428: Status 404 returned error can't find the container with id 5344a78c2bf4ce95f8d02adef42baa4e8d6e3a2364c9899aba447032d4fcc428
Sep 09 16:53:38.670895 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:38.670858    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt" event={"ID":"c25eb85b-87e6-4786-9301-e79893b29412","Type":"ContainerStarted","Data":"0307b187157d834896804b6de69a1786dc606c62daf5106fde19efeaf6a4dc7f"}
Sep 09 16:53:38.671337 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:38.670904    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt" event={"ID":"c25eb85b-87e6-4786-9301-e79893b29412","Type":"ContainerStarted","Data":"5344a78c2bf4ce95f8d02adef42baa4e8d6e3a2364c9899aba447032d4fcc428"}
Sep 09 16:53:38.689481 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:38.689428    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt" podStartSLOduration=1.689414124 podStartE2EDuration="1.689414124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:53:38.687237447 +0000 UTC m=+2384.517445507" watchObservedRunningTime="2026-09-09 16:53:38.689414124 +0000 UTC m=+2384.519622181"
Sep 09 16:53:39.901236 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:39.901202    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:53:47.618634 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:47.618595    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:53:47.619123 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:47.618651    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:53:47.630791 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:47.630769    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:53:47.729832 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:47.729798    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:53:52.999694 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:52.999653    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"]
Sep 09 16:53:53.003306 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.003274    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:53:53.005887 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.005865    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-585e4a6c-kserve-self-signed-certs\""
Sep 09 16:53:53.013614 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.013592    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"]
Sep 09 16:53:53.114382 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.114345    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459bccce-2689-42a6-9b00-4488f53bf3d9-model-cache\") pod \"e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx\" (UID: \"459bccce-2689-42a6-9b00-4488f53bf3d9\") " pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:53:53.114549 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.114408    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/459bccce-2689-42a6-9b00-4488f53bf3d9-dshm\") pod \"e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx\" (UID: \"459bccce-2689-42a6-9b00-4488f53bf3d9\") " pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:53:53.114549 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.114471    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459bccce-2689-42a6-9b00-4488f53bf3d9-tmp-dir\") pod \"e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx\" (UID: \"459bccce-2689-42a6-9b00-4488f53bf3d9\") " pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:53:53.114622 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.114545    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/459bccce-2689-42a6-9b00-4488f53bf3d9-home\") pod \"e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx\" (UID: \"459bccce-2689-42a6-9b00-4488f53bf3d9\") " pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:53:53.114622 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.114607    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbdkt\" (UniqueName: \"kubernetes.io/projected/459bccce-2689-42a6-9b00-4488f53bf3d9-kube-api-access-hbdkt\") pod \"e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx\" (UID: \"459bccce-2689-42a6-9b00-4488f53bf3d9\") " pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:53:53.114690 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.114627    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/459bccce-2689-42a6-9b00-4488f53bf3d9-tls-certs\") pod \"e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx\" (UID: \"459bccce-2689-42a6-9b00-4488f53bf3d9\") " pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:53:53.215578 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.215530    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/459bccce-2689-42a6-9b00-4488f53bf3d9-home\") pod \"e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx\" (UID: \"459bccce-2689-42a6-9b00-4488f53bf3d9\") " pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:53:53.215771 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.215607    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbdkt\" (UniqueName: \"kubernetes.io/projected/459bccce-2689-42a6-9b00-4488f53bf3d9-kube-api-access-hbdkt\") pod \"e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx\" (UID: \"459bccce-2689-42a6-9b00-4488f53bf3d9\") " pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:53:53.215771 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.215627    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/459bccce-2689-42a6-9b00-4488f53bf3d9-tls-certs\") pod \"e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx\" (UID: \"459bccce-2689-42a6-9b00-4488f53bf3d9\") " pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:53:53.215771 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.215664    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459bccce-2689-42a6-9b00-4488f53bf3d9-model-cache\") pod \"e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx\" (UID: \"459bccce-2689-42a6-9b00-4488f53bf3d9\") " pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:53:53.215771 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.215695    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/459bccce-2689-42a6-9b00-4488f53bf3d9-dshm\") pod \"e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx\" (UID: \"459bccce-2689-42a6-9b00-4488f53bf3d9\") " pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:53:53.215771 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.215717    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459bccce-2689-42a6-9b00-4488f53bf3d9-tmp-dir\") pod \"e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx\" (UID: \"459bccce-2689-42a6-9b00-4488f53bf3d9\") " pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:53:53.216049 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.215946    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/459bccce-2689-42a6-9b00-4488f53bf3d9-home\") pod \"e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx\" (UID: \"459bccce-2689-42a6-9b00-4488f53bf3d9\") " pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:53:53.216049 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.216039    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459bccce-2689-42a6-9b00-4488f53bf3d9-model-cache\") pod \"e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx\" (UID: \"459bccce-2689-42a6-9b00-4488f53bf3d9\") " pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:53:53.216143 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.216101    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459bccce-2689-42a6-9b00-4488f53bf3d9-tmp-dir\") pod \"e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx\" (UID: \"459bccce-2689-42a6-9b00-4488f53bf3d9\") " pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:53:53.217933 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.217912    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/459bccce-2689-42a6-9b00-4488f53bf3d9-dshm\") pod \"e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx\" (UID: \"459bccce-2689-42a6-9b00-4488f53bf3d9\") " pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:53:53.218102 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.218081    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/459bccce-2689-42a6-9b00-4488f53bf3d9-tls-certs\") pod \"e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx\" (UID: \"459bccce-2689-42a6-9b00-4488f53bf3d9\") " pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:53:53.222939 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.222910    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbdkt\" (UniqueName: \"kubernetes.io/projected/459bccce-2689-42a6-9b00-4488f53bf3d9-kube-api-access-hbdkt\") pod \"e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx\" (UID: \"459bccce-2689-42a6-9b00-4488f53bf3d9\") " pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:53:53.312918 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.312838    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:53:53.587637 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.587614    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"]
Sep 09 16:53:53.588709 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:53:53.588686    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod459bccce_2689_42a6_9b00_4488f53bf3d9.slice/crio-f270b87dab9fa2f62196166672186c2129ff705d8c98ce19023c457d54bd5266 WatchSource:0}: Error finding container f270b87dab9fa2f62196166672186c2129ff705d8c98ce19023c457d54bd5266: Status 404 returned error can't find the container with id f270b87dab9fa2f62196166672186c2129ff705d8c98ce19023c457d54bd5266
Sep 09 16:53:53.750532 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.750493    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx" event={"ID":"459bccce-2689-42a6-9b00-4488f53bf3d9","Type":"ContainerStarted","Data":"b319b73f494eb4b8f49f5d7204af3d2dfc5f71730d24b47942384b4223eb5cf5"}
Sep 09 16:53:53.750700 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.750544    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx" event={"ID":"459bccce-2689-42a6-9b00-4488f53bf3d9","Type":"ContainerStarted","Data":"f270b87dab9fa2f62196166672186c2129ff705d8c98ce19023c457d54bd5266"}
Sep 09 16:53:53.766626 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:53.766582    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx" podStartSLOduration=1.766568905 podStartE2EDuration="1.766568905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:53:53.76448592 +0000 UTC m=+2399.594693980" watchObservedRunningTime="2026-09-09 16:53:53.766568905 +0000 UTC m=+2399.596776961"
Sep 09 16:53:54.790172 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:54.790147    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgcjl_561d9dd7-04c3-42dc-bcf6-2c92fd61706b/console-operator/1.log"
Sep 09 16:53:54.808173 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:54.808153    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgcjl_561d9dd7-04c3-42dc-bcf6-2c92fd61706b/console-operator/1.log"
Sep 09 16:53:56.196092 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:53:56.196056    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:54:03.313963 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:03.313926    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:54:03.314381 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:03.314211    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:54:03.326403 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:03.326376    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:54:03.819207 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:03.819172    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:54:06.628122 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:06.628079    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"]
Sep 09 16:54:07.589083 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:07.589045    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"]
Sep 09 16:54:09.499667 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:09.499633    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:54:17.888051 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:17.888014    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:54:21.651330 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:21.651273    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt" podUID="c25eb85b-87e6-4786-9301-e79893b29412" containerName="main" containerID="cri-o://0307b187157d834896804b6de69a1786dc606c62daf5106fde19efeaf6a4dc7f" gracePeriod=45
Sep 09 16:54:21.903699 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:21.903646    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:54:21.916665 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:21.916635    2555 generic.go:356] "Generic (PLEG): container finished" podID="c25eb85b-87e6-4786-9301-e79893b29412" containerID="0307b187157d834896804b6de69a1786dc606c62daf5106fde19efeaf6a4dc7f" exitCode=0
Sep 09 16:54:21.916768 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:21.916672    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"
Sep 09 16:54:21.916768 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:21.916719    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt" event={"ID":"c25eb85b-87e6-4786-9301-e79893b29412","Type":"ContainerDied","Data":"0307b187157d834896804b6de69a1786dc606c62daf5106fde19efeaf6a4dc7f"}
Sep 09 16:54:21.916768 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:21.916757    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt" event={"ID":"c25eb85b-87e6-4786-9301-e79893b29412","Type":"ContainerDied","Data":"5344a78c2bf4ce95f8d02adef42baa4e8d6e3a2364c9899aba447032d4fcc428"}
Sep 09 16:54:21.916867 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:21.916778    2555 scope.go:117] "RemoveContainer" containerID="0307b187157d834896804b6de69a1786dc606c62daf5106fde19efeaf6a4dc7f"
Sep 09 16:54:21.925952 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:21.925932    2555 scope.go:117] "RemoveContainer" containerID="0307b187157d834896804b6de69a1786dc606c62daf5106fde19efeaf6a4dc7f"
Sep 09 16:54:21.926259 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:54:21.926234    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0307b187157d834896804b6de69a1786dc606c62daf5106fde19efeaf6a4dc7f\": container with ID starting with 0307b187157d834896804b6de69a1786dc606c62daf5106fde19efeaf6a4dc7f not found: ID does not exist" containerID="0307b187157d834896804b6de69a1786dc606c62daf5106fde19efeaf6a4dc7f"
Sep 09 16:54:21.926349 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:21.926273    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0307b187157d834896804b6de69a1786dc606c62daf5106fde19efeaf6a4dc7f"} err="failed to get container status \"0307b187157d834896804b6de69a1786dc606c62daf5106fde19efeaf6a4dc7f\": rpc error: code = NotFound desc = could not find container \"0307b187157d834896804b6de69a1786dc606c62daf5106fde19efeaf6a4dc7f\": container with ID starting with 0307b187157d834896804b6de69a1786dc606c62daf5106fde19efeaf6a4dc7f not found: ID does not exist"
Sep 09 16:54:21.979473 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:21.979451    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c25eb85b-87e6-4786-9301-e79893b29412-dshm\") pod \"c25eb85b-87e6-4786-9301-e79893b29412\" (UID: \"c25eb85b-87e6-4786-9301-e79893b29412\") "
Sep 09 16:54:21.979598 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:21.979504    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c25eb85b-87e6-4786-9301-e79893b29412-home\") pod \"c25eb85b-87e6-4786-9301-e79893b29412\" (UID: \"c25eb85b-87e6-4786-9301-e79893b29412\") "
Sep 09 16:54:21.979598 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:21.979530    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn5r8\" (UniqueName: \"kubernetes.io/projected/c25eb85b-87e6-4786-9301-e79893b29412-kube-api-access-qn5r8\") pod \"c25eb85b-87e6-4786-9301-e79893b29412\" (UID: \"c25eb85b-87e6-4786-9301-e79893b29412\") "
Sep 09 16:54:21.979598 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:21.979586    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c25eb85b-87e6-4786-9301-e79893b29412-tls-certs\") pod \"c25eb85b-87e6-4786-9301-e79893b29412\" (UID: \"c25eb85b-87e6-4786-9301-e79893b29412\") "
Sep 09 16:54:21.979764 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:21.979614    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c25eb85b-87e6-4786-9301-e79893b29412-tmp-dir\") pod \"c25eb85b-87e6-4786-9301-e79893b29412\" (UID: \"c25eb85b-87e6-4786-9301-e79893b29412\") "
Sep 09 16:54:21.979764 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:21.979694    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c25eb85b-87e6-4786-9301-e79893b29412-model-cache\") pod \"c25eb85b-87e6-4786-9301-e79893b29412\" (UID: \"c25eb85b-87e6-4786-9301-e79893b29412\") "
Sep 09 16:54:21.979764 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:21.979753    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25eb85b-87e6-4786-9301-e79893b29412-home" (OuterVolumeSpecName: "home") pod "c25eb85b-87e6-4786-9301-e79893b29412" (UID: "c25eb85b-87e6-4786-9301-e79893b29412"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:54:21.979966 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:21.979918    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25eb85b-87e6-4786-9301-e79893b29412-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c25eb85b-87e6-4786-9301-e79893b29412" (UID: "c25eb85b-87e6-4786-9301-e79893b29412"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:54:21.980070 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:21.979967    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c25eb85b-87e6-4786-9301-e79893b29412-home\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:54:21.980070 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:21.979988    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25eb85b-87e6-4786-9301-e79893b29412-model-cache" (OuterVolumeSpecName: "model-cache") pod "c25eb85b-87e6-4786-9301-e79893b29412" (UID: "c25eb85b-87e6-4786-9301-e79893b29412"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:54:21.981686 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:21.981667    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25eb85b-87e6-4786-9301-e79893b29412-kube-api-access-qn5r8" (OuterVolumeSpecName: "kube-api-access-qn5r8") pod "c25eb85b-87e6-4786-9301-e79893b29412" (UID: "c25eb85b-87e6-4786-9301-e79893b29412"). InnerVolumeSpecName "kube-api-access-qn5r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:54:21.981937 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:21.981918    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c25eb85b-87e6-4786-9301-e79893b29412-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c25eb85b-87e6-4786-9301-e79893b29412" (UID: "c25eb85b-87e6-4786-9301-e79893b29412"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:54:21.981937 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:21.981925    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25eb85b-87e6-4786-9301-e79893b29412-dshm" (OuterVolumeSpecName: "dshm") pod "c25eb85b-87e6-4786-9301-e79893b29412" (UID: "c25eb85b-87e6-4786-9301-e79893b29412"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:54:22.081357 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.081327    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn5r8\" (UniqueName: \"kubernetes.io/projected/c25eb85b-87e6-4786-9301-e79893b29412-kube-api-access-qn5r8\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:54:22.081357 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.081353    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c25eb85b-87e6-4786-9301-e79893b29412-tls-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:54:22.081357 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.081365    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c25eb85b-87e6-4786-9301-e79893b29412-tmp-dir\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:54:22.081592 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.081372    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c25eb85b-87e6-4786-9301-e79893b29412-model-cache\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:54:22.081592 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.081380    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c25eb85b-87e6-4786-9301-e79893b29412-dshm\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:54:22.237387 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.237356    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"]
Sep 09 16:54:22.241245 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.241219    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-585e4a6c-kserve-85c8c5fd6c-btfgt"]
Sep 09 16:54:22.608498 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.608457    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx" podUID="459bccce-2689-42a6-9b00-4488f53bf3d9" containerName="main" containerID="cri-o://b319b73f494eb4b8f49f5d7204af3d2dfc5f71730d24b47942384b4223eb5cf5" gracePeriod=45
Sep 09 16:54:22.608838 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.608814    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c25eb85b-87e6-4786-9301-e79893b29412" path="/var/lib/kubelet/pods/c25eb85b-87e6-4786-9301-e79893b29412/volumes"
Sep 09 16:54:22.847068 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.847047    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:54:22.889854 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.889794    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459bccce-2689-42a6-9b00-4488f53bf3d9-tmp-dir\") pod \"459bccce-2689-42a6-9b00-4488f53bf3d9\" (UID: \"459bccce-2689-42a6-9b00-4488f53bf3d9\") "
Sep 09 16:54:22.889964 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.889865    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/459bccce-2689-42a6-9b00-4488f53bf3d9-dshm\") pod \"459bccce-2689-42a6-9b00-4488f53bf3d9\" (UID: \"459bccce-2689-42a6-9b00-4488f53bf3d9\") "
Sep 09 16:54:22.889964 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.889892    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/459bccce-2689-42a6-9b00-4488f53bf3d9-tls-certs\") pod \"459bccce-2689-42a6-9b00-4488f53bf3d9\" (UID: \"459bccce-2689-42a6-9b00-4488f53bf3d9\") "
Sep 09 16:54:22.889964 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.889911    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbdkt\" (UniqueName: \"kubernetes.io/projected/459bccce-2689-42a6-9b00-4488f53bf3d9-kube-api-access-hbdkt\") pod \"459bccce-2689-42a6-9b00-4488f53bf3d9\" (UID: \"459bccce-2689-42a6-9b00-4488f53bf3d9\") "
Sep 09 16:54:22.889964 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.889929    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459bccce-2689-42a6-9b00-4488f53bf3d9-model-cache\") pod \"459bccce-2689-42a6-9b00-4488f53bf3d9\" (UID: \"459bccce-2689-42a6-9b00-4488f53bf3d9\") "
Sep 09 16:54:22.890169 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.890030    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/459bccce-2689-42a6-9b00-4488f53bf3d9-home\") pod \"459bccce-2689-42a6-9b00-4488f53bf3d9\" (UID: \"459bccce-2689-42a6-9b00-4488f53bf3d9\") "
Sep 09 16:54:22.890169 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.890042    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459bccce-2689-42a6-9b00-4488f53bf3d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "459bccce-2689-42a6-9b00-4488f53bf3d9" (UID: "459bccce-2689-42a6-9b00-4488f53bf3d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:54:22.890275 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.890187    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459bccce-2689-42a6-9b00-4488f53bf3d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "459bccce-2689-42a6-9b00-4488f53bf3d9" (UID: "459bccce-2689-42a6-9b00-4488f53bf3d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:54:22.890363 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.890329    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459bccce-2689-42a6-9b00-4488f53bf3d9-home" (OuterVolumeSpecName: "home") pod "459bccce-2689-42a6-9b00-4488f53bf3d9" (UID: "459bccce-2689-42a6-9b00-4488f53bf3d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:54:22.890363 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.890343    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/459bccce-2689-42a6-9b00-4488f53bf3d9-model-cache\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:54:22.890363 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.890361    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/459bccce-2689-42a6-9b00-4488f53bf3d9-tmp-dir\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:54:22.891985 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.891957    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459bccce-2689-42a6-9b00-4488f53bf3d9-dshm" (OuterVolumeSpecName: "dshm") pod "459bccce-2689-42a6-9b00-4488f53bf3d9" (UID: "459bccce-2689-42a6-9b00-4488f53bf3d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:54:22.892103 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.892016    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/459bccce-2689-42a6-9b00-4488f53bf3d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "459bccce-2689-42a6-9b00-4488f53bf3d9" (UID: "459bccce-2689-42a6-9b00-4488f53bf3d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:54:22.892503 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.892465    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459bccce-2689-42a6-9b00-4488f53bf3d9-kube-api-access-hbdkt" (OuterVolumeSpecName: "kube-api-access-hbdkt") pod "459bccce-2689-42a6-9b00-4488f53bf3d9" (UID: "459bccce-2689-42a6-9b00-4488f53bf3d9"). InnerVolumeSpecName "kube-api-access-hbdkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:54:22.924703 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.924677    2555 generic.go:356] "Generic (PLEG): container finished" podID="459bccce-2689-42a6-9b00-4488f53bf3d9" containerID="b319b73f494eb4b8f49f5d7204af3d2dfc5f71730d24b47942384b4223eb5cf5" exitCode=0
Sep 09 16:54:22.924816 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.924720    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"
Sep 09 16:54:22.924816 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.924763    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx" event={"ID":"459bccce-2689-42a6-9b00-4488f53bf3d9","Type":"ContainerDied","Data":"b319b73f494eb4b8f49f5d7204af3d2dfc5f71730d24b47942384b4223eb5cf5"}
Sep 09 16:54:22.924816 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.924803    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx" event={"ID":"459bccce-2689-42a6-9b00-4488f53bf3d9","Type":"ContainerDied","Data":"f270b87dab9fa2f62196166672186c2129ff705d8c98ce19023c457d54bd5266"}
Sep 09 16:54:22.924921 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.924820    2555 scope.go:117] "RemoveContainer" containerID="b319b73f494eb4b8f49f5d7204af3d2dfc5f71730d24b47942384b4223eb5cf5"
Sep 09 16:54:22.934230 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.934212    2555 scope.go:117] "RemoveContainer" containerID="b319b73f494eb4b8f49f5d7204af3d2dfc5f71730d24b47942384b4223eb5cf5"
Sep 09 16:54:22.934504 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:54:22.934485    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b319b73f494eb4b8f49f5d7204af3d2dfc5f71730d24b47942384b4223eb5cf5\": container with ID starting with b319b73f494eb4b8f49f5d7204af3d2dfc5f71730d24b47942384b4223eb5cf5 not found: ID does not exist" containerID="b319b73f494eb4b8f49f5d7204af3d2dfc5f71730d24b47942384b4223eb5cf5"
Sep 09 16:54:22.934552 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.934512    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b319b73f494eb4b8f49f5d7204af3d2dfc5f71730d24b47942384b4223eb5cf5"} err="failed to get container status \"b319b73f494eb4b8f49f5d7204af3d2dfc5f71730d24b47942384b4223eb5cf5\": rpc error: code = NotFound desc = could not find container \"b319b73f494eb4b8f49f5d7204af3d2dfc5f71730d24b47942384b4223eb5cf5\": container with ID starting with b319b73f494eb4b8f49f5d7204af3d2dfc5f71730d24b47942384b4223eb5cf5 not found: ID does not exist"
Sep 09 16:54:22.946088 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.946067    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"]
Sep 09 16:54:22.948067 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.948049    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-585e4a6c-kserve-5bdf487595-2xwqx"]
Sep 09 16:54:22.991173 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.991148    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/459bccce-2689-42a6-9b00-4488f53bf3d9-home\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:54:22.991280 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.991179    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/459bccce-2689-42a6-9b00-4488f53bf3d9-dshm\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:54:22.991280 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.991192    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/459bccce-2689-42a6-9b00-4488f53bf3d9-tls-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:54:22.991280 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:22.991203    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbdkt\" (UniqueName: \"kubernetes.io/projected/459bccce-2689-42a6-9b00-4488f53bf3d9-kube-api-access-hbdkt\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:54:24.607754 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:24.607719    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="459bccce-2689-42a6-9b00-4488f53bf3d9" path="/var/lib/kubelet/pods/459bccce-2689-42a6-9b00-4488f53bf3d9/volumes"
Sep 09 16:54:25.795550 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:25.795509    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:54:26.016901 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:26.016867    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d6c5d7bdb-cknd4"]
Sep 09 16:54:26.017431 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:26.017408    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c25eb85b-87e6-4786-9301-e79893b29412" containerName="main"
Sep 09 16:54:26.017568 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:26.017432    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c25eb85b-87e6-4786-9301-e79893b29412" containerName="main"
Sep 09 16:54:26.017568 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:26.017495    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="459bccce-2689-42a6-9b00-4488f53bf3d9" containerName="main"
Sep 09 16:54:26.017568 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:26.017505    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="459bccce-2689-42a6-9b00-4488f53bf3d9" containerName="main"
Sep 09 16:54:26.017717 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:26.017603    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="459bccce-2689-42a6-9b00-4488f53bf3d9" containerName="main"
Sep 09 16:54:26.017717 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:26.017614    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c25eb85b-87e6-4786-9301-e79893b29412" containerName="main"
Sep 09 16:54:26.022007 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:26.021983    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-cknd4"
Sep 09 16:54:26.032512 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:26.032489    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d6c5d7bdb-cknd4"]
Sep 09 16:54:26.118576 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:26.118494    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2bwn\" (UniqueName: \"kubernetes.io/projected/9bef953c-3ba5-499c-83f5-ce7909e23a18-kube-api-access-r2bwn\") pod \"maas-api-6d6c5d7bdb-cknd4\" (UID: \"9bef953c-3ba5-499c-83f5-ce7909e23a18\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-cknd4"
Sep 09 16:54:26.118576 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:26.118535    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bef953c-3ba5-499c-83f5-ce7909e23a18-maas-api-tls\") pod \"maas-api-6d6c5d7bdb-cknd4\" (UID: \"9bef953c-3ba5-499c-83f5-ce7909e23a18\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-cknd4"
Sep 09 16:54:26.118576 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:26.118566    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bef953c-3ba5-499c-83f5-ce7909e23a18-metrics-certs\") pod \"maas-api-6d6c5d7bdb-cknd4\" (UID: \"9bef953c-3ba5-499c-83f5-ce7909e23a18\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-cknd4"
Sep 09 16:54:26.118833 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:26.118663    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bef953c-3ba5-499c-83f5-ce7909e23a18-openshift-service-ca\") pod \"maas-api-6d6c5d7bdb-cknd4\" (UID: \"9bef953c-3ba5-499c-83f5-ce7909e23a18\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-cknd4"
Sep 09 16:54:26.219806 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:26.219768    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2bwn\" (UniqueName: \"kubernetes.io/projected/9bef953c-3ba5-499c-83f5-ce7909e23a18-kube-api-access-r2bwn\") pod \"maas-api-6d6c5d7bdb-cknd4\" (UID: \"9bef953c-3ba5-499c-83f5-ce7909e23a18\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-cknd4"
Sep 09 16:54:26.219972 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:26.219821    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bef953c-3ba5-499c-83f5-ce7909e23a18-maas-api-tls\") pod \"maas-api-6d6c5d7bdb-cknd4\" (UID: \"9bef953c-3ba5-499c-83f5-ce7909e23a18\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-cknd4"
Sep 09 16:54:26.219972 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:26.219860    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bef953c-3ba5-499c-83f5-ce7909e23a18-metrics-certs\") pod \"maas-api-6d6c5d7bdb-cknd4\" (UID: \"9bef953c-3ba5-499c-83f5-ce7909e23a18\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-cknd4"
Sep 09 16:54:26.219972 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:26.219897    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bef953c-3ba5-499c-83f5-ce7909e23a18-openshift-service-ca\") pod \"maas-api-6d6c5d7bdb-cknd4\" (UID: \"9bef953c-3ba5-499c-83f5-ce7909e23a18\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-cknd4"
Sep 09 16:54:26.220604 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:26.220582    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bef953c-3ba5-499c-83f5-ce7909e23a18-openshift-service-ca\") pod \"maas-api-6d6c5d7bdb-cknd4\" (UID: \"9bef953c-3ba5-499c-83f5-ce7909e23a18\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-cknd4"
Sep 09 16:54:26.222252 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:26.222232    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bef953c-3ba5-499c-83f5-ce7909e23a18-metrics-certs\") pod \"maas-api-6d6c5d7bdb-cknd4\" (UID: \"9bef953c-3ba5-499c-83f5-ce7909e23a18\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-cknd4"
Sep 09 16:54:26.222411 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:26.222394    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bef953c-3ba5-499c-83f5-ce7909e23a18-maas-api-tls\") pod \"maas-api-6d6c5d7bdb-cknd4\" (UID: \"9bef953c-3ba5-499c-83f5-ce7909e23a18\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-cknd4"
Sep 09 16:54:26.227365 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:26.227339    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2bwn\" (UniqueName: \"kubernetes.io/projected/9bef953c-3ba5-499c-83f5-ce7909e23a18-kube-api-access-r2bwn\") pod \"maas-api-6d6c5d7bdb-cknd4\" (UID: \"9bef953c-3ba5-499c-83f5-ce7909e23a18\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-cknd4"
Sep 09 16:54:26.333365 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:26.333331    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-cknd4"
Sep 09 16:54:26.613110 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:26.613081    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d6c5d7bdb-cknd4"]
Sep 09 16:54:26.613627 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:54:26.613596    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bef953c_3ba5_499c_83f5_ce7909e23a18.slice/crio-b738f294f177a2e17f1e1b7608fb2d3e7ae5f557b7ee30fc4debd55ef8077ffd WatchSource:0}: Error finding container b738f294f177a2e17f1e1b7608fb2d3e7ae5f557b7ee30fc4debd55ef8077ffd: Status 404 returned error can't find the container with id b738f294f177a2e17f1e1b7608fb2d3e7ae5f557b7ee30fc4debd55ef8077ffd
Sep 09 16:54:26.950400 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:26.950374    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-cknd4" event={"ID":"9bef953c-3ba5-499c-83f5-ce7909e23a18","Type":"ContainerStarted","Data":"b738f294f177a2e17f1e1b7608fb2d3e7ae5f557b7ee30fc4debd55ef8077ffd"}
Sep 09 16:54:27.958331 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:27.958273    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-cknd4" event={"ID":"9bef953c-3ba5-499c-83f5-ce7909e23a18","Type":"ContainerStarted","Data":"e4f5ae8d9401a190e115b005ce895e285fcca0aa4cd1ef0cfb3a8e225f9eeaa3"}
Sep 09 16:54:27.958811 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:27.958353    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-cknd4"
Sep 09 16:54:27.974890 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:27.974842    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-cknd4" podStartSLOduration=2.645418525 podStartE2EDuration="2.97482809s" totalImagesPullingTime="329.409565ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:54:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:54:27.974604711 +0000 UTC m=+2433.804812770" watchObservedRunningTime="2026-09-09 16:54:27.97482809 +0000 UTC m=+2433.805036147"
Sep 09 16:54:33.289904 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:33.289864    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:54:33.968819 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:33.968790    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-cknd4"
Sep 09 16:54:34.003580 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:34.003550    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"]
Sep 09 16:54:34.003753 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:34.003728    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2" podUID="531455b0-04f0-42bc-948f-ba17688f2d93" containerName="maas-api" containerID="cri-o://5eb6d59f52113930bd123fe50341094dae36fefe5f718c8b60f8c7c932a20f13" gracePeriod=30
Sep 09 16:54:35.240156 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:35.240132    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"
Sep 09 16:54:35.301094 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:35.301066    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531455b0-04f0-42bc-948f-ba17688f2d93-maas-api-tls\") pod \"531455b0-04f0-42bc-948f-ba17688f2d93\" (UID: \"531455b0-04f0-42bc-948f-ba17688f2d93\") "
Sep 09 16:54:35.301313 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:35.301105    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531455b0-04f0-42bc-948f-ba17688f2d93-metrics-certs\") pod \"531455b0-04f0-42bc-948f-ba17688f2d93\" (UID: \"531455b0-04f0-42bc-948f-ba17688f2d93\") "
Sep 09 16:54:35.301313 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:35.301137    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxkqc\" (UniqueName: \"kubernetes.io/projected/531455b0-04f0-42bc-948f-ba17688f2d93-kube-api-access-hxkqc\") pod \"531455b0-04f0-42bc-948f-ba17688f2d93\" (UID: \"531455b0-04f0-42bc-948f-ba17688f2d93\") "
Sep 09 16:54:35.301313 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:35.301244    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/531455b0-04f0-42bc-948f-ba17688f2d93-openshift-service-ca\") pod \"531455b0-04f0-42bc-948f-ba17688f2d93\" (UID: \"531455b0-04f0-42bc-948f-ba17688f2d93\") "
Sep 09 16:54:35.301598 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:35.301571    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/531455b0-04f0-42bc-948f-ba17688f2d93-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "531455b0-04f0-42bc-948f-ba17688f2d93" (UID: "531455b0-04f0-42bc-948f-ba17688f2d93"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:54:35.303266 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:35.303233    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/531455b0-04f0-42bc-948f-ba17688f2d93-kube-api-access-hxkqc" (OuterVolumeSpecName: "kube-api-access-hxkqc") pod "531455b0-04f0-42bc-948f-ba17688f2d93" (UID: "531455b0-04f0-42bc-948f-ba17688f2d93"). InnerVolumeSpecName "kube-api-access-hxkqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:54:35.303266 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:35.303246    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/531455b0-04f0-42bc-948f-ba17688f2d93-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "531455b0-04f0-42bc-948f-ba17688f2d93" (UID: "531455b0-04f0-42bc-948f-ba17688f2d93"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:54:35.303412 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:35.303265    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/531455b0-04f0-42bc-948f-ba17688f2d93-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "531455b0-04f0-42bc-948f-ba17688f2d93" (UID: "531455b0-04f0-42bc-948f-ba17688f2d93"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:54:35.402847 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:35.402780    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531455b0-04f0-42bc-948f-ba17688f2d93-maas-api-tls\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:54:35.402847 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:35.402804    2555 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531455b0-04f0-42bc-948f-ba17688f2d93-metrics-certs\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:54:35.402847 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:35.402812    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxkqc\" (UniqueName: \"kubernetes.io/projected/531455b0-04f0-42bc-948f-ba17688f2d93-kube-api-access-hxkqc\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:54:35.402847 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:35.402821    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/531455b0-04f0-42bc-948f-ba17688f2d93-openshift-service-ca\") on node \"ip-10-0-141-202.ec2.internal\" DevicePath \"\""
Sep 09 16:54:36.009566 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:36.009529    2555 generic.go:356] "Generic (PLEG): container finished" podID="531455b0-04f0-42bc-948f-ba17688f2d93" containerID="5eb6d59f52113930bd123fe50341094dae36fefe5f718c8b60f8c7c932a20f13" exitCode=0
Sep 09 16:54:36.009767 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:36.009573    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"
Sep 09 16:54:36.009767 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:36.009621    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2" event={"ID":"531455b0-04f0-42bc-948f-ba17688f2d93","Type":"ContainerDied","Data":"5eb6d59f52113930bd123fe50341094dae36fefe5f718c8b60f8c7c932a20f13"}
Sep 09 16:54:36.009767 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:36.009679    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2" event={"ID":"531455b0-04f0-42bc-948f-ba17688f2d93","Type":"ContainerDied","Data":"2cb8a9ba6b87cfbb3563fe42a4eab75c5967005221cfcacc0294eb7041bd1108"}
Sep 09 16:54:36.009767 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:36.009738    2555 scope.go:117] "RemoveContainer" containerID="5eb6d59f52113930bd123fe50341094dae36fefe5f718c8b60f8c7c932a20f13"
Sep 09 16:54:36.019353 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:36.019338    2555 scope.go:117] "RemoveContainer" containerID="5eb6d59f52113930bd123fe50341094dae36fefe5f718c8b60f8c7c932a20f13"
Sep 09 16:54:36.019614 ip-10-0-141-202 kubenswrapper[2555]: E0909 16:54:36.019598    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eb6d59f52113930bd123fe50341094dae36fefe5f718c8b60f8c7c932a20f13\": container with ID starting with 5eb6d59f52113930bd123fe50341094dae36fefe5f718c8b60f8c7c932a20f13 not found: ID does not exist" containerID="5eb6d59f52113930bd123fe50341094dae36fefe5f718c8b60f8c7c932a20f13"
Sep 09 16:54:36.019664 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:36.019620    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb6d59f52113930bd123fe50341094dae36fefe5f718c8b60f8c7c932a20f13"} err="failed to get container status \"5eb6d59f52113930bd123fe50341094dae36fefe5f718c8b60f8c7c932a20f13\": rpc error: code = NotFound desc = could not find container \"5eb6d59f52113930bd123fe50341094dae36fefe5f718c8b60f8c7c932a20f13\": container with ID starting with 5eb6d59f52113930bd123fe50341094dae36fefe5f718c8b60f8c7c932a20f13 not found: ID does not exist"
Sep 09 16:54:36.030943 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:36.030921    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"]
Sep 09 16:54:36.034723 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:36.034703    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f975847c4-8rlg2"]
Sep 09 16:54:36.607879 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:36.607847    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="531455b0-04f0-42bc-948f-ba17688f2d93" path="/var/lib/kubelet/pods/531455b0-04f0-42bc-948f-ba17688f2d93/volumes"
Sep 09 16:54:40.091231 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:40.091200    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:54:45.586728 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:45.586693    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:54:52.390025 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:54:52.389990    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5lx6m"]
Sep 09 16:55:01.292846 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:01.292814    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-567dd_9b92f3d8-9f59-4f7f-8e59-e82b115c2426/manager/0.log"
Sep 09 16:55:01.414649 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:01.414616    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-596jg_613f5bc6-e044-40f3-a80f-1d65e6f59d6c/manager/0.log"
Sep 09 16:55:01.534263 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:01.534225    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c78477868-q462t_ef1c0123-b6a7-40d3-a947-eb7403570bbd/manager/0.log"
Sep 09 16:55:01.878543 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:01.878516    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b9b77fc68-4qwds_39e57a13-2360-4e1f-b466-f0c6749db833/copy-manifests/0.log"
Sep 09 16:55:01.889355 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:01.889329    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b9b77fc68-4qwds_39e57a13-2360-4e1f-b466-f0c6749db833/manager/0.log"
Sep 09 16:55:02.464755 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:02.464725    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6d6c5d7bdb-cknd4_9bef953c-3ba5-499c-83f5-ce7909e23a18/maas-api/0.log"
Sep 09 16:55:03.851014 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:03.850984    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-89jtl_5882ad0b-e8d0-42f5-ab1e-67e64234cc2b/manager/0.log"
Sep 09 16:55:03.964630 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:03.964605    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-n267w_2defd344-8312-4ef2-a1f7-4a18cf2866a4/manager/0.log"
Sep 09 16:55:04.216592 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:04.216514    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-wkjvw_9f12cad5-5ad6-47e3-9b75-246392ca465c/manager/0.log"
Sep 09 16:55:04.334979 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:04.334947    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-5lx6m_40a70a72-34ff-4e7e-a553-63449eb5b379/limitador/0.log"
Sep 09 16:55:05.008149 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:05.008120    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-769d969b5b-crxch_d13928a1-4968-4e90-89a9-fee8bb59477e/kube-auth-proxy/0.log"
Sep 09 16:55:05.224864 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:05.224830    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-cnkj9_b512776f-22c4-40b8-89fd-c3cd475e6d09/istio-proxy/1.log"
Sep 09 16:55:05.899683 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:05.899651    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-fr2pp_28a263ea-58bc-4f76-961e-4231faea205a/main/0.log"
Sep 09 16:55:06.571787 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:06.571760    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-4n4z7_4f590638-b2f6-434b-81d0-ec3a9a6d590d/main/0.log"
Sep 09 16:55:11.871641 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:11.871611    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8s7bt/must-gather-frc68"]
Sep 09 16:55:11.872262 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:11.872228    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="531455b0-04f0-42bc-948f-ba17688f2d93" containerName="maas-api"
Sep 09 16:55:11.872262 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:11.872261    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="531455b0-04f0-42bc-948f-ba17688f2d93" containerName="maas-api"
Sep 09 16:55:11.872480 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:11.872436    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="531455b0-04f0-42bc-948f-ba17688f2d93" containerName="maas-api"
Sep 09 16:55:11.876005 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:11.875984    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8s7bt/must-gather-frc68"
Sep 09 16:55:11.878892 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:11.878869    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8s7bt\"/\"default-dockercfg-jl267\""
Sep 09 16:55:11.879014 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:11.878869    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8s7bt\"/\"openshift-service-ca.crt\""
Sep 09 16:55:11.879014 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:11.878915    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8s7bt\"/\"kube-root-ca.crt\""
Sep 09 16:55:11.883400 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:11.883377    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8s7bt/must-gather-frc68"]
Sep 09 16:55:11.914104 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:11.914077    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k99np\" (UniqueName: \"kubernetes.io/projected/eeffb3e3-b579-409b-8a05-aaf8ed92fea8-kube-api-access-k99np\") pod \"must-gather-frc68\" (UID: \"eeffb3e3-b579-409b-8a05-aaf8ed92fea8\") " pod="openshift-must-gather-8s7bt/must-gather-frc68"
Sep 09 16:55:11.914212 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:11.914136    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/eeffb3e3-b579-409b-8a05-aaf8ed92fea8-must-gather-output\") pod \"must-gather-frc68\" (UID: \"eeffb3e3-b579-409b-8a05-aaf8ed92fea8\") " pod="openshift-must-gather-8s7bt/must-gather-frc68"
Sep 09 16:55:12.014784 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:12.014752    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/eeffb3e3-b579-409b-8a05-aaf8ed92fea8-must-gather-output\") pod \"must-gather-frc68\" (UID: \"eeffb3e3-b579-409b-8a05-aaf8ed92fea8\") " pod="openshift-must-gather-8s7bt/must-gather-frc68"
Sep 09 16:55:12.014950 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:12.014845    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k99np\" (UniqueName: \"kubernetes.io/projected/eeffb3e3-b579-409b-8a05-aaf8ed92fea8-kube-api-access-k99np\") pod \"must-gather-frc68\" (UID: \"eeffb3e3-b579-409b-8a05-aaf8ed92fea8\") " pod="openshift-must-gather-8s7bt/must-gather-frc68"
Sep 09 16:55:12.015145 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:12.015123    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/eeffb3e3-b579-409b-8a05-aaf8ed92fea8-must-gather-output\") pod \"must-gather-frc68\" (UID: \"eeffb3e3-b579-409b-8a05-aaf8ed92fea8\") " pod="openshift-must-gather-8s7bt/must-gather-frc68"
Sep 09 16:55:12.028898 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:12.028873    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k99np\" (UniqueName: \"kubernetes.io/projected/eeffb3e3-b579-409b-8a05-aaf8ed92fea8-kube-api-access-k99np\") pod \"must-gather-frc68\" (UID: \"eeffb3e3-b579-409b-8a05-aaf8ed92fea8\") " pod="openshift-must-gather-8s7bt/must-gather-frc68"
Sep 09 16:55:12.186678 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:12.186598    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8s7bt/must-gather-frc68"
Sep 09 16:55:12.455641 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:12.455618    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8s7bt/must-gather-frc68"]
Sep 09 16:55:12.456826 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:55:12.456800    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeeffb3e3_b579_409b_8a05_aaf8ed92fea8.slice/crio-c66a0d81b9c3cbcf663c0b8183226a988d9efab2e1328cc5d1ac9b670da499e5 WatchSource:0}: Error finding container c66a0d81b9c3cbcf663c0b8183226a988d9efab2e1328cc5d1ac9b670da499e5: Status 404 returned error can't find the container with id c66a0d81b9c3cbcf663c0b8183226a988d9efab2e1328cc5d1ac9b670da499e5
Sep 09 16:55:13.212922 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:13.212882    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8s7bt/must-gather-frc68" event={"ID":"eeffb3e3-b579-409b-8a05-aaf8ed92fea8","Type":"ContainerStarted","Data":"c66a0d81b9c3cbcf663c0b8183226a988d9efab2e1328cc5d1ac9b670da499e5"}
Sep 09 16:55:14.221843 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:14.221810    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8s7bt/must-gather-frc68" event={"ID":"eeffb3e3-b579-409b-8a05-aaf8ed92fea8","Type":"ContainerStarted","Data":"ab43afdd35764571c6ffe5f07da2022b13cce321a2c48c19309fbe6d8cdfffea"}
Sep 09 16:55:14.222242 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:14.221855    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8s7bt/must-gather-frc68" event={"ID":"eeffb3e3-b579-409b-8a05-aaf8ed92fea8","Type":"ContainerStarted","Data":"8d919df590367bf8a8ed1a64c539806134190ed7187910a87b5d173bfa852000"}
Sep 09 16:55:14.235705 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:14.235660    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8s7bt/must-gather-frc68" podStartSLOduration=2.20793073 podStartE2EDuration="3.235644821s" totalImagesPullingTime="1.027714091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:55:14.235263642 +0000 UTC m=+2480.065471699" watchObservedRunningTime="2026-09-09 16:55:14.235644821 +0000 UTC m=+2480.065852879"
Sep 09 16:55:14.954706 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:14.954678    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hjz7d_98c72774-4b40-43b2-b4dd-7590320acc60/global-pull-secret-syncer/0.log"
Sep 09 16:55:15.061099 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:15.061064    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9shcg_68024951-17f4-4df9-b48a-7674411fd492/konnectivity-agent/0.log"
Sep 09 16:55:15.154761 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:15.154735    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-202.ec2.internal_a40d2b498bc21892b32d687ae581fc71/haproxy/0.log"
Sep 09 16:55:19.803761 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:19.803725    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-89jtl_5882ad0b-e8d0-42f5-ab1e-67e64234cc2b/manager/0.log"
Sep 09 16:55:19.836608 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:19.836574    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-n267w_2defd344-8312-4ef2-a1f7-4a18cf2866a4/manager/0.log"
Sep 09 16:55:19.930306 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:19.930257    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-wkjvw_9f12cad5-5ad6-47e3-9b75-246392ca465c/manager/0.log"
Sep 09 16:55:19.983728 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:19.983678    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-5lx6m_40a70a72-34ff-4e7e-a553-63449eb5b379/limitador/0.log"
Sep 09 16:55:21.894315 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:21.893325    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5zhcs_c1fc3122-4351-4a27-9fca-ba12043e0f2a/node-exporter/0.log"
Sep 09 16:55:21.919287 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:21.919254    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5zhcs_c1fc3122-4351-4a27-9fca-ba12043e0f2a/kube-rbac-proxy/0.log"
Sep 09 16:55:21.940794 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:21.940698    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5zhcs_c1fc3122-4351-4a27-9fca-ba12043e0f2a/init-textfile/0.log"
Sep 09 16:55:24.338684 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:24.338655    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgcjl_561d9dd7-04c3-42dc-bcf6-2c92fd61706b/console-operator/1.log"
Sep 09 16:55:24.345231 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:24.345201    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zgcjl_561d9dd7-04c3-42dc-bcf6-2c92fd61706b/console-operator/2.log"
Sep 09 16:55:24.827807 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:24.827772    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf"]
Sep 09 16:55:24.836119 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:24.836095    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf"
Sep 09 16:55:24.840559 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:24.840509    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf"]
Sep 09 16:55:24.864928 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:24.864901    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b6fdc5d75-lrmst_784b5b32-6777-4bf9-83af-73c3c5f9f69a/console/0.log"
Sep 09 16:55:24.946565 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:24.946527    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06747d1b-eb3b-4f96-81a2-98282a05712a-sys\") pod \"perf-node-gather-daemonset-hjbcf\" (UID: \"06747d1b-eb3b-4f96-81a2-98282a05712a\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf"
Sep 09 16:55:24.946825 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:24.946807    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06747d1b-eb3b-4f96-81a2-98282a05712a-lib-modules\") pod \"perf-node-gather-daemonset-hjbcf\" (UID: \"06747d1b-eb3b-4f96-81a2-98282a05712a\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf"
Sep 09 16:55:24.946945 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:24.946930    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdjbq\" (UniqueName: \"kubernetes.io/projected/06747d1b-eb3b-4f96-81a2-98282a05712a-kube-api-access-hdjbq\") pod \"perf-node-gather-daemonset-hjbcf\" (UID: \"06747d1b-eb3b-4f96-81a2-98282a05712a\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf"
Sep 09 16:55:24.947079 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:24.947067    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/06747d1b-eb3b-4f96-81a2-98282a05712a-proc\") pod \"perf-node-gather-daemonset-hjbcf\" (UID: \"06747d1b-eb3b-4f96-81a2-98282a05712a\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf"
Sep 09 16:55:24.947185 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:24.947174    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/06747d1b-eb3b-4f96-81a2-98282a05712a-podres\") pod \"perf-node-gather-daemonset-hjbcf\" (UID: \"06747d1b-eb3b-4f96-81a2-98282a05712a\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf"
Sep 09 16:55:25.048170 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:25.048128    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06747d1b-eb3b-4f96-81a2-98282a05712a-sys\") pod \"perf-node-gather-daemonset-hjbcf\" (UID: \"06747d1b-eb3b-4f96-81a2-98282a05712a\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf"
Sep 09 16:55:25.048592 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:25.048566    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06747d1b-eb3b-4f96-81a2-98282a05712a-lib-modules\") pod \"perf-node-gather-daemonset-hjbcf\" (UID: \"06747d1b-eb3b-4f96-81a2-98282a05712a\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf"
Sep 09 16:55:25.048877 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:25.048859    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdjbq\" (UniqueName: \"kubernetes.io/projected/06747d1b-eb3b-4f96-81a2-98282a05712a-kube-api-access-hdjbq\") pod \"perf-node-gather-daemonset-hjbcf\" (UID: \"06747d1b-eb3b-4f96-81a2-98282a05712a\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf"
Sep 09 16:55:25.049393 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:25.049373    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/06747d1b-eb3b-4f96-81a2-98282a05712a-proc\") pod \"perf-node-gather-daemonset-hjbcf\" (UID: \"06747d1b-eb3b-4f96-81a2-98282a05712a\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf"
Sep 09 16:55:25.049624 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:25.049606    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/06747d1b-eb3b-4f96-81a2-98282a05712a-podres\") pod \"perf-node-gather-daemonset-hjbcf\" (UID: \"06747d1b-eb3b-4f96-81a2-98282a05712a\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf"
Sep 09 16:55:25.049887 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:25.048813    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06747d1b-eb3b-4f96-81a2-98282a05712a-lib-modules\") pod \"perf-node-gather-daemonset-hjbcf\" (UID: \"06747d1b-eb3b-4f96-81a2-98282a05712a\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf"
Sep 09 16:55:25.050013 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:25.049559    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/06747d1b-eb3b-4f96-81a2-98282a05712a-proc\") pod \"perf-node-gather-daemonset-hjbcf\" (UID: \"06747d1b-eb3b-4f96-81a2-98282a05712a\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf"
Sep 09 16:55:25.050119 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:25.048485    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06747d1b-eb3b-4f96-81a2-98282a05712a-sys\") pod \"perf-node-gather-daemonset-hjbcf\" (UID: \"06747d1b-eb3b-4f96-81a2-98282a05712a\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf"
Sep 09 16:55:25.050213 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:25.049805    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/06747d1b-eb3b-4f96-81a2-98282a05712a-podres\") pod \"perf-node-gather-daemonset-hjbcf\" (UID: \"06747d1b-eb3b-4f96-81a2-98282a05712a\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf"
Sep 09 16:55:25.056931 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:25.056910    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdjbq\" (UniqueName: \"kubernetes.io/projected/06747d1b-eb3b-4f96-81a2-98282a05712a-kube-api-access-hdjbq\") pod \"perf-node-gather-daemonset-hjbcf\" (UID: \"06747d1b-eb3b-4f96-81a2-98282a05712a\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf"
Sep 09 16:55:25.149840 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:25.149758    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf"
Sep 09 16:55:25.457580 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:25.457552    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf"]
Sep 09 16:55:25.466103 ip-10-0-141-202 kubenswrapper[2555]: W0909 16:55:25.466071    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod06747d1b_eb3b_4f96_81a2_98282a05712a.slice/crio-6e0b017fadc819c95b4e16bc5fa0ccd19450e0fbbdd6f00e35661fd65f4db4b9 WatchSource:0}: Error finding container 6e0b017fadc819c95b4e16bc5fa0ccd19450e0fbbdd6f00e35661fd65f4db4b9: Status 404 returned error can't find the container with id 6e0b017fadc819c95b4e16bc5fa0ccd19450e0fbbdd6f00e35661fd65f4db4b9
Sep 09 16:55:26.325894 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:26.325850    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf" event={"ID":"06747d1b-eb3b-4f96-81a2-98282a05712a","Type":"ContainerStarted","Data":"d996b2223fb0e7aef16c5f45c4de5881e5b806b428763839ce61a64194615e33"}
Sep 09 16:55:26.326090 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:26.325926    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf" event={"ID":"06747d1b-eb3b-4f96-81a2-98282a05712a","Type":"ContainerStarted","Data":"6e0b017fadc819c95b4e16bc5fa0ccd19450e0fbbdd6f00e35661fd65f4db4b9"}
Sep 09 16:55:26.326768 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:26.326739    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf"
Sep 09 16:55:26.668638 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:26.668555    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xs54n_b21b58de-81cf-4ff7-9352-40854e5e0246/dns/0.log"
Sep 09 16:55:26.686274 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:26.686248    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xs54n_b21b58de-81cf-4ff7-9352-40854e5e0246/kube-rbac-proxy/0.log"
Sep 09 16:55:26.765696 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:26.765669    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-92lzv_fc6eae27-5d1d-4957-99b5-e87c1d177376/dns-node-resolver/0.log"
Sep 09 16:55:27.282528 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:27.282499    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-cc689c778-vg2sn_fc4dce56-bf35-434f-9b43-c3ea87ef83e3/registry/0.log"
Sep 09 16:55:27.322574 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:27.322546    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hkc7k_3f37385e-b449-4281-b738-7edce088387c/node-ca/0.log"
Sep 09 16:55:28.527518 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:28.527487    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-769d969b5b-crxch_d13928a1-4968-4e90-89a9-fee8bb59477e/kube-auth-proxy/0.log"
Sep 09 16:55:28.573760 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:28.573727    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-cnkj9_b512776f-22c4-40b8-89fd-c3cd475e6d09/istio-proxy/0.log"
Sep 09 16:55:28.585799 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:28.585775    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-cnkj9_b512776f-22c4-40b8-89fd-c3cd475e6d09/istio-proxy/1.log"
Sep 09 16:55:29.514247 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:29.514206    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-snd6v_b7c6ef20-237d-46e2-a83a-c20ac88ccfdd/serve-healthcheck-canary/0.log"
Sep 09 16:55:29.929433 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:29.929349    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-srgtn_ca9bcf27-b906-4735-bab2-b17bffa33bce/insights-operator/0.log"
Sep 09 16:55:29.929842 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:29.929642    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-srgtn_ca9bcf27-b906-4735-bab2-b17bffa33bce/insights-operator/1.log"
Sep 09 16:55:29.947888 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:29.947865    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5wzqv_c931a390-ae5a-451b-9b76-258aa5ff5bba/kube-rbac-proxy/0.log"
Sep 09 16:55:29.966398 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:29.966372    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5wzqv_c931a390-ae5a-451b-9b76-258aa5ff5bba/exporter/0.log"
Sep 09 16:55:29.983184 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:29.983163    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5wzqv_c931a390-ae5a-451b-9b76-258aa5ff5bba/extractor/0.log"
Sep 09 16:55:32.048054 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:32.048017    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-567dd_9b92f3d8-9f59-4f7f-8e59-e82b115c2426/manager/0.log"
Sep 09 16:55:32.266558 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:32.266527    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-596jg_613f5bc6-e044-40f3-a80f-1d65e6f59d6c/manager/0.log"
Sep 09 16:55:32.317082 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:32.316997    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c78477868-q462t_ef1c0123-b6a7-40d3-a947-eb7403570bbd/manager/0.log"
Sep 09 16:55:32.516621 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:32.516581    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b9b77fc68-4qwds_39e57a13-2360-4e1f-b466-f0c6749db833/manager/0.log"
Sep 09 16:55:32.532542 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:32.532518    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b9b77fc68-4qwds_39e57a13-2360-4e1f-b466-f0c6749db833/copy-manifests/0.log"
Sep 09 16:55:33.351835 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:33.351804    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf"
Sep 09 16:55:33.367244 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:33.367184    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-hjbcf" podStartSLOduration=9.367163879 podStartE2EDuration="9.367163879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:55:26.343372454 +0000 UTC m=+2492.173580513" watchObservedRunningTime="2026-09-09 16:55:33.367163879 +0000 UTC m=+2499.197371944"
Sep 09 16:55:33.695581 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:33.695510    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-x62wl_d28b0a55-6832-41d7-a55e-a92b3d921803/openshift-lws-operator/0.log"
Sep 09 16:55:34.802728 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:34.802650    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-769d969b5b-crxch_d13928a1-4968-4e90-89a9-fee8bb59477e/kube-auth-proxy/0.log"
Sep 09 16:55:34.841983 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:34.841920    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-cnkj9_b512776f-22c4-40b8-89fd-c3cd475e6d09/istio-proxy/0.log"
Sep 09 16:55:34.855129 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:34.855092    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-cnkj9_b512776f-22c4-40b8-89fd-c3cd475e6d09/istio-proxy/1.log"
Sep 09 16:55:36.939207 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:36.939175    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-769d969b5b-crxch_d13928a1-4968-4e90-89a9-fee8bb59477e/kube-auth-proxy/0.log"
Sep 09 16:55:36.980568 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:36.980540    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-cnkj9_b512776f-22c4-40b8-89fd-c3cd475e6d09/istio-proxy/0.log"
Sep 09 16:55:36.991002 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:36.990967    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-cnkj9_b512776f-22c4-40b8-89fd-c3cd475e6d09/istio-proxy/1.log"
Sep 09 16:55:39.911395 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:39.911363    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2d5w7_bf8b3ddd-efd5-4277-a2dc-c5e93467f290/kube-multus/0.log"
Sep 09 16:55:39.930205 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:39.930180    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6hgj9_bd350005-0808-410e-bef5-9ecfd227869d/kube-multus-additional-cni-plugins/0.log"
Sep 09 16:55:39.948332 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:39.948275    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6hgj9_bd350005-0808-410e-bef5-9ecfd227869d/egress-router-binary-copy/0.log"
Sep 09 16:55:39.966956 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:39.966933    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6hgj9_bd350005-0808-410e-bef5-9ecfd227869d/cni-plugins/0.log"
Sep 09 16:55:39.983258 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:39.983226    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6hgj9_bd350005-0808-410e-bef5-9ecfd227869d/bond-cni-plugin/0.log"
Sep 09 16:55:40.001205 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:40.001187    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6hgj9_bd350005-0808-410e-bef5-9ecfd227869d/routeoverride-cni/0.log"
Sep 09 16:55:40.018492 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:40.018477    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6hgj9_bd350005-0808-410e-bef5-9ecfd227869d/whereabouts-cni-bincopy/0.log"
Sep 09 16:55:40.035711 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:40.035696    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6hgj9_bd350005-0808-410e-bef5-9ecfd227869d/whereabouts-cni/0.log"
Sep 09 16:55:40.404894 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:40.404867    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gcb4p_7334d4d3-9b79-4b0d-ab28-e0aec88c18cb/network-metrics-daemon/0.log"
Sep 09 16:55:40.419038 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:40.419016    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gcb4p_7334d4d3-9b79-4b0d-ab28-e0aec88c18cb/kube-rbac-proxy/0.log"
Sep 09 16:55:41.630242 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:41.630212    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9w_e4fce212-3111-43ec-8fee-a58bd7ffdff9/ovn-controller/0.log"
Sep 09 16:55:41.643663 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:41.643635    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9w_e4fce212-3111-43ec-8fee-a58bd7ffdff9/ovn-acl-logging/0.log"
Sep 09 16:55:41.667154 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:41.667135    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9w_e4fce212-3111-43ec-8fee-a58bd7ffdff9/kube-rbac-proxy-node/0.log"
Sep 09 16:55:41.685475 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:41.685454    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9w_e4fce212-3111-43ec-8fee-a58bd7ffdff9/kube-rbac-proxy-ovn-metrics/0.log"
Sep 09 16:55:41.699482 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:41.699466    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9w_e4fce212-3111-43ec-8fee-a58bd7ffdff9/northd/0.log"
Sep 09 16:55:41.716822 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:41.716804    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9w_e4fce212-3111-43ec-8fee-a58bd7ffdff9/nbdb/0.log"
Sep 09 16:55:41.734687 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:41.734667    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9w_e4fce212-3111-43ec-8fee-a58bd7ffdff9/sbdb/0.log"
Sep 09 16:55:41.796478 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:41.796454    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phs9w_e4fce212-3111-43ec-8fee-a58bd7ffdff9/ovnkube-controller/0.log"
Sep 09 16:55:42.809635 ip-10-0-141-202 kubenswrapper[2555]: I0909 16:55:42.809605    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-slbqw_f1bd8aa2-35fd-458b-8a84-69ccf7a6276f/network-check-target-container/0.log"