Sep 11 21:05:32.140980 ip-10-0-142-224 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 21:05:32.593628 ip-10-0-142-224 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 21:05:32.593628 ip-10-0-142-224 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 21:05:32.593628 ip-10-0-142-224 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 21:05:32.593628 ip-10-0-142-224 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 21:05:32.593628 ip-10-0-142-224 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 21:05:32.594776 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.594672    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 21:05:32.598912 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598896    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:05:32.598951 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598913    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:05:32.598951 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598918    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:05:32.598951 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598922    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:05:32.598951 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598925    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:05:32.598951 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598928    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:05:32.598951 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598931    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:05:32.598951 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598933    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:05:32.598951 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598936    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:05:32.598951 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598939    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:05:32.598951 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598942    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:05:32.598951 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598944    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:05:32.598951 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598948    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:05:32.598951 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598950    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:05:32.598951 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598953    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:05:32.598951 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598956    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:05:32.599332 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598959    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:05:32.599332 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598962    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:05:32.599332 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598965    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:05:32.599332 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598967    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:05:32.599332 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598970    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:05:32.599332 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598974    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:05:32.599332 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598977    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:05:32.599332 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598980    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:05:32.599332 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598983    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:05:32.599332 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598986    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:05:32.599332 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598989    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:05:32.599332 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598992    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:05:32.599332 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598994    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:05:32.599332 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.598998    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:05:32.599332 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599000    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:05:32.599332 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599003    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:05:32.599332 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599005    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:05:32.599332 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599008    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:05:32.599332 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599010    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:05:32.599332 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599013    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:05:32.599835 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599015    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:05:32.599835 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599017    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:05:32.599835 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599020    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:05:32.599835 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599023    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:05:32.599835 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599026    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:05:32.599835 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599029    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:05:32.599835 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599031    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:05:32.599835 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599034    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:05:32.599835 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599036    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:05:32.599835 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599039    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:05:32.599835 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599041    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:05:32.599835 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599044    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:05:32.599835 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599046    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:05:32.599835 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599049    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:05:32.599835 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599054    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:05:32.599835 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599058    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:05:32.599835 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599060    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:05:32.599835 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599063    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:05:32.599835 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599066    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:05:32.599835 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599068    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:05:32.600305 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599071    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:05:32.600305 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599074    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:05:32.600305 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599076    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:05:32.600305 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599079    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:05:32.600305 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599081    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:05:32.600305 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599084    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:05:32.600305 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599086    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:05:32.600305 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599088    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:05:32.600305 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599091    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:05:32.600305 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599094    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:05:32.600305 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599096    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:05:32.600305 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599099    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:05:32.600305 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599101    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:05:32.600305 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599104    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:05:32.600305 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599106    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:05:32.600305 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599110    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:05:32.600305 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599113    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:05:32.600305 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599116    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:05:32.600305 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599119    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:05:32.600305 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599121    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:05:32.600770 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599124    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:05:32.600770 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599127    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:05:32.600770 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599130    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:05:32.600770 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599133    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:05:32.600770 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599135    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:05:32.600770 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599138    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:05:32.600770 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599141    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:05:32.600770 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599144    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:05:32.600770 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599146    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:05:32.600770 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599149    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:05:32.600770 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599151    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:05:32.600770 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599154    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:05:32.600770 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599156    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:05:32.600770 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599159    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:05:32.600770 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599161    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:05:32.600770 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599163    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:05:32.600770 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599166    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:05:32.600770 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599169    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:05:32.600770 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599171    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:05:32.601226 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599173    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:05:32.601226 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599176    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:05:32.601226 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599179    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:05:32.601226 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599181    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:05:32.601226 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599876    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:05:32.601226 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599883    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:05:32.601226 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599888    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:05:32.601226 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599892    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:05:32.601226 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599895    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:05:32.601226 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599899    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:05:32.601226 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599903    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:05:32.601226 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599906    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:05:32.601226 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599909    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:05:32.601226 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599913    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:05:32.601226 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599916    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:05:32.601226 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599919    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:05:32.601226 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599923    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:05:32.601226 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599925    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:05:32.601226 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599928    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:05:32.601675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599931    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:05:32.601675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599934    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:05:32.601675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599936    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:05:32.601675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599939    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:05:32.601675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599941    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:05:32.601675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599944    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:05:32.601675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599947    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:05:32.601675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599949    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:05:32.601675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599952    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:05:32.601675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599954    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:05:32.601675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599964    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:05:32.601675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599967    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:05:32.601675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599969    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:05:32.601675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599972    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:05:32.601675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599975    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:05:32.601675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599978    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:05:32.601675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599980    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:05:32.601675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599983    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:05:32.601675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599985    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:05:32.602230 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599988    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:05:32.602230 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599990    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:05:32.602230 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599993    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:05:32.602230 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599995    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:05:32.602230 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.599998    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:05:32.602230 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600001    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:05:32.602230 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600004    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:05:32.602230 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600007    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:05:32.602230 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600009    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:05:32.602230 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600012    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:05:32.602230 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600014    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:05:32.602230 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600016    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:05:32.602230 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600019    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:05:32.602230 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600022    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:05:32.602230 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600025    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:05:32.602230 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600027    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:05:32.602230 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600030    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:05:32.602230 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600032    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:05:32.602230 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600035    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:05:32.602230 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600037    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:05:32.602734 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600040    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:05:32.602734 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600042    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:05:32.602734 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600045    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:05:32.602734 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600047    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:05:32.602734 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600050    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:05:32.602734 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600052    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:05:32.602734 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600055    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:05:32.602734 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600057    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:05:32.602734 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600059    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:05:32.602734 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600062    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:05:32.602734 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600064    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:05:32.602734 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600067    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:05:32.602734 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600069    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:05:32.602734 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600071    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:05:32.602734 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600074    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:05:32.602734 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600076    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:05:32.602734 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600078    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:05:32.602734 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600081    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:05:32.602734 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600084    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:05:32.602734 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600086    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:05:32.603221 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600089    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:05:32.603221 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600091    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:05:32.603221 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600094    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:05:32.603221 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600096    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:05:32.603221 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600100    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:05:32.603221 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600102    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:05:32.603221 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600105    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:05:32.603221 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600108    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:05:32.603221 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600110    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:05:32.603221 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600112    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:05:32.603221 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600115    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:05:32.603221 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600117    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:05:32.603221 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600121    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:05:32.603221 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600124    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:05:32.603221 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600127    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:05:32.603221 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600130    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:05:32.603221 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600133    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:05:32.603221 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600136    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:05:32.603221 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600139    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:05:32.603690 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600141    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:05:32.603690 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600144    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:05:32.603690 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600146    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:05:32.603690 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600149    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:05:32.603690 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600151    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:05:32.603690 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600154    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:05:32.603690 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600499    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:05:32.603690 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600503    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:05:32.603690 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600506    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:05:32.603690 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600509    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:05:32.603690 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600512    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:05:32.603690 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600514    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:05:32.603690 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600517    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:05:32.603690 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600519    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:05:32.603690 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600521    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:05:32.603690 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600524    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:05:32.603690 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600528    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:05:32.603690 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600532    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:05:32.603690 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600535    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:05:32.604139 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600538    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:05:32.604139 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600541    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:05:32.604139 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600544    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:05:32.604139 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600546    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:05:32.604139 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600549    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:05:32.604139 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600552    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:05:32.604139 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600555    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:05:32.604139 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600558    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:05:32.604139 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600560    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:05:32.604139 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600563    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:05:32.604139 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600565    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:05:32.604139 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600567    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:05:32.604139 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600570    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:05:32.604139 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600572    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:05:32.604139 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600575    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:05:32.604139 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600577    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:05:32.604139 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600579    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:05:32.604139 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600582    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:05:32.604139 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600584    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:05:32.604139 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600586    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:05:32.604633 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600589    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:05:32.604633 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600591    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:05:32.604633 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600594    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:05:32.604633 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600597    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:05:32.604633 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600600    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:05:32.604633 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600602    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:05:32.604633 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600605    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:05:32.604633 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600607    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:05:32.604633 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600609    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:05:32.604633 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600612    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:05:32.604633 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600615    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:05:32.604633 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600617    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:05:32.604633 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600620    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:05:32.604633 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600623    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:05:32.604633 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600625    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:05:32.604633 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600627    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:05:32.604633 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600630    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:05:32.604633 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600632    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:05:32.604633 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600635    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:05:32.604633 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600637    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:05:32.605137 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600640    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:05:32.605137 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600643    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:05:32.605137 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600645    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:05:32.605137 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600647    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:05:32.605137 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600650    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:05:32.605137 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600652    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:05:32.605137 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600654    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:05:32.605137 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600657    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:05:32.605137 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600659    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:05:32.605137 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600662    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:05:32.605137 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600664    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:05:32.605137 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600666    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:05:32.605137 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600669    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:05:32.605137 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600671    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:05:32.605137 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600674    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:05:32.605137 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600676    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:05:32.605137 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600694    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:05:32.605137 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600697    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:05:32.605137 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600699    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:05:32.605137 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600702    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:05:32.605601 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600704    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:05:32.605601 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600707    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:05:32.605601 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600710    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:05:32.605601 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600714    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:05:32.605601 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600717    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:05:32.605601 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600719    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:05:32.605601 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600722    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:05:32.605601 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600724    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:05:32.605601 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600727    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:05:32.605601 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600730    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:05:32.605601 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600732    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:05:32.605601 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600735    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:05:32.605601 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600737    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:05:32.605601 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600740    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:05:32.605601 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600743    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:05:32.605601 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600746    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:05:32.605601 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600748    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:05:32.605601 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600751    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:05:32.605601 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600753    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:05:32.606124 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600757    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:05:32.606124 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600760    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:05:32.606124 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600763    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:05:32.606124 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600766    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:05:32.606124 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600769    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:05:32.606124 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600772    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:05:32.606124 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.600774    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:05:32.606124 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600849    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 21:05:32.606124 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600879    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 21:05:32.606124 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600886    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 21:05:32.606124 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600890    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 21:05:32.606124 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600895    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 21:05:32.606124 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600898    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 21:05:32.606124 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600902    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 21:05:32.606124 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600906    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 21:05:32.606124 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600912    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 21:05:32.606124 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600915    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 21:05:32.606124 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600919    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 21:05:32.606124 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600922    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 21:05:32.606124 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600925    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 21:05:32.606124 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600928    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 11 21:05:32.606124 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600931    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600934    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600937    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600940    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600944    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600947    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600950    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600953    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600956    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600960    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600963    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600966    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600970    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600972    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600975    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600979    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600982    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600985    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600989    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600992    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600995    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.600998    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601001    2578 flags.go:64] FLAG: --enable-server="true"
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601004    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601009    2578 flags.go:64] FLAG: --event-burst="100"
Sep 11 21:05:32.606647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601012    2578 flags.go:64] FLAG: --event-qps="50"
Sep 11 21:05:32.607248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601016    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 21:05:32.607248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601019    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 21:05:32.607248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601022    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 11 21:05:32.607248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601026    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 21:05:32.607248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601029    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 21:05:32.607248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601032    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 21:05:32.607248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601035    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 11 21:05:32.607248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601038    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 21:05:32.607248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601041    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 21:05:32.607248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601044    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 21:05:32.607248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601047    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 21:05:32.607248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601050    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 21:05:32.607248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601053    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 21:05:32.607248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601056    2578 flags.go:64] FLAG: --feature-gates=""
Sep 11 21:05:32.607248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601059    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 21:05:32.607248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601062    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 21:05:32.607248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601065    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 21:05:32.607248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601069    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 21:05:32.607248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601072    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 21:05:32.607248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601075    2578 flags.go:64] FLAG: --help="false"
Sep 11 21:05:32.607248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601077    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-142-224.ec2.internal"
Sep 11 21:05:32.607248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601080    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 21:05:32.607248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601083    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 21:05:32.607248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601086    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 21:05:32.607861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601089    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 21:05:32.607861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601092    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 21:05:32.607861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601095    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 21:05:32.607861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601098    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 21:05:32.607861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601101    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 21:05:32.607861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601103    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 21:05:32.607861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601107    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 21:05:32.607861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601110    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 21:05:32.607861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601114    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 11 21:05:32.607861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601118    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 21:05:32.607861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601121    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 21:05:32.607861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601125    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 21:05:32.607861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601127    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 21:05:32.607861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601130    2578 flags.go:64] FLAG: --lock-file=""
Sep 11 21:05:32.607861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601133    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 21:05:32.607861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601136    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 21:05:32.607861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601138    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 21:05:32.607861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601143    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 21:05:32.607861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601146    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 21:05:32.607861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601149    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 21:05:32.607861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601152    2578 flags.go:64] FLAG: --logging-format="text"
Sep 11 21:05:32.607861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601155    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 21:05:32.607861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601158    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 21:05:32.607861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601161    2578 flags.go:64] FLAG: --manifest-url=""
Sep 11 21:05:32.608475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601164    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 21:05:32.608475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601168    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 21:05:32.608475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601171    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 21:05:32.608475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601175    2578 flags.go:64] FLAG: --max-pods="110"
Sep 11 21:05:32.608475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601178    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 21:05:32.608475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601181    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 21:05:32.608475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601184    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 21:05:32.608475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601187    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 21:05:32.608475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601190    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 21:05:32.608475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601193    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 21:05:32.608475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601196    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 21:05:32.608475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601203    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 21:05:32.608475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601206    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 21:05:32.608475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601208    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 21:05:32.608475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601211    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 11 21:05:32.608475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601215    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 21:05:32.608475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601220    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 21:05:32.608475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601225    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 21:05:32.608475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601228    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 21:05:32.608475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601231    2578 flags.go:64] FLAG: --port="10250"
Sep 11 21:05:32.608475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601235    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 21:05:32.608475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601237    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06061d284e65cc8d7"
Sep 11 21:05:32.608475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601241    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 11 21:05:32.608475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601244    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601246    2578 flags.go:64] FLAG: --register-node="true"
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601249    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601255    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601258    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601261    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601264    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601267    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601270    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601273    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601276    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601279    2578 flags.go:64] FLAG: --runonce="false"
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601281    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601284    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601287    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601290    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601292    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601295    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601299    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601301    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601304    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601307    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601310    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601312    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601315    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 21:05:32.609094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601318    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 11 21:05:32.609729 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601323    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 21:05:32.609729 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601328    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 21:05:32.609729 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601331    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 21:05:32.609729 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601334    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 21:05:32.609729 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601338    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 11 21:05:32.609729 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601340    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 21:05:32.609729 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601343    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 21:05:32.609729 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601346    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 21:05:32.609729 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601349    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 21:05:32.609729 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601352    2578 flags.go:64] FLAG: --v="2"
Sep 11 21:05:32.609729 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601355    2578 flags.go:64] FLAG: --version="false"
Sep 11 21:05:32.609729 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601360    2578 flags.go:64] FLAG: --vmodule=""
Sep 11 21:05:32.609729 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601363    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 21:05:32.609729 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601366    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 21:05:32.609729 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601478    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:05:32.609729 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601481    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:05:32.609729 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601485    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:05:32.609729 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601488    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:05:32.609729 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601492    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:05:32.609729 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601496    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:05:32.609729 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601499    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:05:32.609729 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601502    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:05:32.610245 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601504    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:05:32.610245 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601507    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:05:32.610245 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601509    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:05:32.610245 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601511    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:05:32.610245 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601514    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:05:32.610245 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601516    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:05:32.610245 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601519    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:05:32.610245 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601521    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:05:32.610245 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601524    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:05:32.610245 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601526    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:05:32.610245 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601531    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:05:32.610245 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601533    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:05:32.610245 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601536    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:05:32.610245 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601538    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:05:32.610245 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601541    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:05:32.610245 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601543    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:05:32.610245 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601546    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:05:32.610245 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601548    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:05:32.610245 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601551    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:05:32.610245 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601553    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:05:32.610738 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601556    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:05:32.610738 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601560    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:05:32.610738 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601563    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:05:32.610738 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601566    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:05:32.610738 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601569    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:05:32.610738 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601573    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:05:32.610738 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601576    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:05:32.610738 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601580    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:05:32.610738 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601583    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:05:32.610738 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601585    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:05:32.610738 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601588    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:05:32.610738 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601590    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:05:32.610738 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601593    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:05:32.610738 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601595    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:05:32.610738 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601598    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:05:32.610738 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601600    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:05:32.610738 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601603    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:05:32.610738 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601605    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:05:32.610738 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601608    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:05:32.611194 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601610    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:05:32.611194 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601612    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:05:32.611194 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601615    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:05:32.611194 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601619    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:05:32.611194 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601622    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:05:32.611194 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601624    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:05:32.611194 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601627    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:05:32.611194 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601629    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:05:32.611194 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601632    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:05:32.611194 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601634    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:05:32.611194 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601637    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:05:32.611194 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601639    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:05:32.611194 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601642    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:05:32.611194 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601644    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:05:32.611194 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601647    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:05:32.611194 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601649    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:05:32.611194 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601652    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:05:32.611194 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601654    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:05:32.611194 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601657    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:05:32.611194 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601659    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:05:32.611675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601662    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:05:32.611675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601664    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:05:32.611675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601667    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:05:32.611675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601669    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:05:32.611675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601672    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:05:32.611675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601674    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:05:32.611675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601677    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:05:32.611675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601692    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:05:32.611675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601695    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:05:32.611675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601698    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:05:32.611675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601701    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:05:32.611675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601703    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:05:32.611675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601706    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:05:32.611675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601708    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:05:32.611675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601711    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:05:32.611675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601715    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:05:32.611675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601717    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:05:32.611675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601720    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:05:32.611675 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601723    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:05:32.612231 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601726    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:05:32.612231 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601728    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:05:32.612231 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601731    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:05:32.612231 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601733    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:05:32.612231 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601736    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:05:32.612231 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601738    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:05:32.612231 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601740    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:05:32.612231 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601743    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:05:32.612231 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601746    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:05:32.612231 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601748    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:05:32.612231 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601751    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:05:32.612231 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601753    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:05:32.612231 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.601756    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:05:32.612231 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.601764    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 21:05:32.612231 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.610003    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 21:05:32.612231 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.610018    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 21:05:32.612612 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.611069    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 21:05:32.612612 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.611076    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 21:05:32.612612 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611142    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:05:32.612612 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611147    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:05:32.612612 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611150    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:05:32.612612 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611153    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:05:32.612612 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611156    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:05:32.612612 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611158    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:05:32.612612 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611161    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:05:32.612612 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611164    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:05:32.612612 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611166    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:05:32.612612 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611169    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:05:32.612612 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611171    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:05:32.612612 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611174    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:05:32.612612 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611176    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:05:32.612612 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611179    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:05:32.612612 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611182    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:05:32.612612 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611185    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:05:32.612612 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611187    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:05:32.612612 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611190    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:05:32.613284 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611193    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:05:32.613284 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611197    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:05:32.613284 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611201    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:05:32.613284 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611206    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:05:32.613284 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611210    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:05:32.613284 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611213    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:05:32.613284 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611215    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:05:32.613284 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611218    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:05:32.613284 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611221    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:05:32.613284 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611224    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:05:32.613284 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611226    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:05:32.613284 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611235    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:05:32.613284 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611238    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:05:32.613284 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611241    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:05:32.613284 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611243    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:05:32.613284 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611245    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:05:32.613284 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611249    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:05:32.613284 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611253    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:05:32.613284 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611255    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:05:32.613761 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611258    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:05:32.613761 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611261    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:05:32.613761 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611263    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:05:32.613761 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611266    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:05:32.613761 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611268    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:05:32.613761 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611271    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:05:32.613761 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611278    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:05:32.613761 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611281    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:05:32.613761 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611284    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:05:32.613761 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611286    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:05:32.613761 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611288    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:05:32.613761 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611291    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:05:32.613761 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611293    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:05:32.613761 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611296    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:05:32.613761 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611298    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:05:32.613761 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611301    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:05:32.613761 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611304    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:05:32.613761 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611306    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:05:32.613761 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611309    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:05:32.613761 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611311    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:05:32.614234 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611314    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:05:32.614234 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611316    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:05:32.614234 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611319    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:05:32.614234 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611321    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:05:32.614234 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611324    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:05:32.614234 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611332    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:05:32.614234 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611335    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:05:32.614234 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611338    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:05:32.614234 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611340    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:05:32.614234 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611343    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:05:32.614234 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611346    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:05:32.614234 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611348    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:05:32.614234 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611351    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:05:32.614234 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611353    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:05:32.614234 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611355    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:05:32.614234 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611358    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:05:32.614234 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611360    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:05:32.614234 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611363    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:05:32.614234 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611365    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:05:32.614705 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611368    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:05:32.614705 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611371    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:05:32.614705 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611373    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:05:32.614705 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611375    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:05:32.614705 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611378    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:05:32.614705 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611381    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:05:32.614705 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611383    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:05:32.614705 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611386    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:05:32.614705 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611389    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:05:32.614705 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611391    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:05:32.614705 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611394    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:05:32.614705 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611396    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:05:32.614705 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611399    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:05:32.614705 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611401    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:05:32.614705 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611404    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:05:32.614705 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611406    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:05:32.614705 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611409    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:05:32.614705 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611412    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:05:32.614705 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611424    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:05:32.614705 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611433    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:05:32.615180 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611436    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:05:32.615180 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611439    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:05:32.615180 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611441    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:05:32.615180 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.611447    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 21:05:32.615180 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611595    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:05:32.615180 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611602    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:05:32.615180 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611605    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:05:32.615180 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611608    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:05:32.615180 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611610    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:05:32.615180 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611613    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:05:32.615180 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611616    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:05:32.615180 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611618    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:05:32.615180 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611621    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:05:32.615180 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611624    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:05:32.615180 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611626    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:05:32.615551 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611629    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:05:32.615551 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611633    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:05:32.615551 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611637    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:05:32.615551 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611640    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:05:32.615551 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611643    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:05:32.615551 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611646    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:05:32.615551 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611648    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:05:32.615551 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611651    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:05:32.615551 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611653    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:05:32.615551 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611656    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:05:32.615551 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611658    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:05:32.615551 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611661    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:05:32.615551 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611663    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:05:32.615551 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611666    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:05:32.615551 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611668    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:05:32.615551 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611671    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:05:32.615551 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611673    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:05:32.615551 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611695    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:05:32.615551 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611700    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:05:32.615551 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611703    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:05:32.616035 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611707    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:05:32.616035 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611710    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:05:32.616035 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611712    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:05:32.616035 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611715    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:05:32.616035 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611717    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:05:32.616035 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611720    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:05:32.616035 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611722    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:05:32.616035 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611731    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:05:32.616035 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611733    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:05:32.616035 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611737    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:05:32.616035 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611740    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:05:32.616035 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611742    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:05:32.616035 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611745    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:05:32.616035 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611748    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:05:32.616035 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611750    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:05:32.616035 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611753    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:05:32.616035 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611756    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:05:32.616035 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611758    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:05:32.616035 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611761    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:05:32.616473 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611763    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:05:32.616473 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611767    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:05:32.616473 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611770    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:05:32.616473 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611774    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:05:32.616473 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611777    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:05:32.616473 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611779    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:05:32.616473 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611782    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:05:32.616473 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611785    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:05:32.616473 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611787    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:05:32.616473 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611790    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:05:32.616473 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611792    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:05:32.616473 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611800    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:05:32.616473 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611803    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:05:32.616473 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611806    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:05:32.616473 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611808    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:05:32.616473 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611810    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:05:32.616473 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611813    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:05:32.616473 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611816    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:05:32.616473 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611818    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:05:32.616927 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611820    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:05:32.616927 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611823    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:05:32.616927 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611825    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:05:32.616927 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611827    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:05:32.616927 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611830    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:05:32.616927 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611832    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:05:32.616927 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611835    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:05:32.616927 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611837    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:05:32.616927 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611840    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:05:32.616927 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611842    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:05:32.616927 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611845    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:05:32.616927 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611847    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:05:32.616927 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611850    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:05:32.616927 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611852    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:05:32.616927 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611854    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:05:32.616927 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611857    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:05:32.616927 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611859    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:05:32.616927 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611861    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:05:32.616927 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611864    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:05:32.616927 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611866    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:05:32.617378 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611869    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:05:32.617378 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611871    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:05:32.617378 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611874    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:05:32.617378 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611876    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:05:32.617378 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611878    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:05:32.617378 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611887    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:05:32.617378 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611890    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:05:32.617378 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611892    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:05:32.617378 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611895    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:05:32.617378 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:32.611897    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:05:32.617378 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.611903    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 21:05:32.617378 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.613115    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 21:05:32.617378 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.615606    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 21:05:32.617378 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.616726    2578 server.go:1019] "Starting client certificate rotation"
Sep 11 21:05:32.617736 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.616824    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 21:05:32.617736 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.616867    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 21:05:32.619619 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.619604    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 21:05:32.620721 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.620704    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 21:05:32.633647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.633627    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 11 21:05:32.637044 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.637029    2578 log.go:25] "Validated CRI v1 image API"
Sep 11 21:05:32.638365 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.638346    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 21:05:32.640995 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.640972    2578 fs.go:135] Filesystem UUIDs: map[0d8f2b40-e76c-4fed-8631-50d496648adb:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fc6160ad-5a05-496d-8dd3-d4f501b9fe04:/dev/nvme0n1p4]
Sep 11 21:05:32.641081 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.640993    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 21:05:32.642897 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.642876    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 21:05:32.646542 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.646398    2578 manager.go:216] Machine: {Timestamp:2026-09-11 21:05:32.645702232 +0000 UTC m=+0.374808619 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199774 MemoryCapacity:33164468224 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e155ef9cd254d7ca9c8c68455783c SystemUUID:ec2e155e-f9cd-254d-7ca9-c8c68455783c BootID:d3f3db66-d904-4ad4-80b6-94f782e4699d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5c:9b:3c:f1:6b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5c:9b:3c:f1:6b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:cf:28:3d:a6:cc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164468224 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 21:05:32.646542 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.646532    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 21:05:32.646727 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.646631    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 21:05:32.650354 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.650323    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 21:05:32.650490 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.650356    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-224.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 21:05:32.651261 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.651251    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 21:05:32.651310 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.651264    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 21:05:32.651310 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.651272    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 21:05:32.652109 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.652098    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 21:05:32.662292 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.662267    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ll6n4"
Sep 11 21:05:32.674073 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.674052    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ll6n4"
Sep 11 21:05:32.798056 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.798029    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 21:05:32.803935 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.803917    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 21:05:32.804060 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.804050    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 21:05:32.804116 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.804084    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 21:05:32.804116 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.804104    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 21:05:32.804194 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.804120    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 21:05:32.804194 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.804130    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 11 21:05:32.804194 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.804139    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 21:05:32.804701 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.804672    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 21:05:32.804773 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.804713    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 21:05:32.809004 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.808984    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 21:05:32.811843 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.811827    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 21:05:32.811922 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.811856    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 21:05:32.813939 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.813926    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 21:05:32.813988 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.813942    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 21:05:32.813988 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.813948    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 21:05:32.813988 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.813953    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 21:05:32.813988 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.813962    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 21:05:32.813988 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.813967    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 21:05:32.813988 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.813980    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 21:05:32.813988 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.813985    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 21:05:32.813988 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.813992    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 21:05:32.814267 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.813998    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 21:05:32.814267 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.814006    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 21:05:32.814267 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.814017    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 21:05:32.814267 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.814040    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 21:05:32.814267 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.814045    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 21:05:32.816754 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.816735    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:05:32.817995 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.817982    2578 server.go:1268] "Started kubelet"
Sep 11 21:05:32.818112 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.818061    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 21:05:32.818223 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.818067    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 21:05:32.818322 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.818242    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 21:05:32.818744 ip-10-0-142-224 systemd[1]: Started Kubernetes Kubelet.
Sep 11 21:05:32.819463 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.819222    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 21:05:32.819530 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.819331    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:05:32.819807 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.819673    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 21:05:32.823047 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.823031    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-142-224.ec2.internal" not found
Sep 11 21:05:32.823810 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:32.823795    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 21:05:32.827017 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.827003    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 21:05:32.827664 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.827648    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 21:05:32.827766 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:32.827737    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-224.ec2.internal\" not found"
Sep 11 21:05:32.828360 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.828336    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 21:05:32.828360 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.828338    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 21:05:32.828360 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.828363    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 21:05:32.828520 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.828398    2578 factory.go:55] Registering systemd factory
Sep 11 21:05:32.828520 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.828443    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 11 21:05:32.828520 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.828464    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 21:05:32.828520 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.828474    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 21:05:32.828666 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.828626    2578 factory.go:153] Registering CRI-O factory
Sep 11 21:05:32.828666 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.828636    2578 factory.go:223] Registration of the crio container factory successfully
Sep 11 21:05:32.828753 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.828696    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 21:05:32.828753 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.828710    2578 factory.go:103] Registering Raw factory
Sep 11 21:05:32.828753 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.828725    2578 manager.go:1196] Started watching for new ooms in manager
Sep 11 21:05:32.829152 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.829141    2578 manager.go:349] Starting recovery of all containers
Sep 11 21:05:32.830577 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.830558    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:05:32.834495 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:32.834470    2578 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-224.ec2.internal\" not found" node="ip-10-0-142-224.ec2.internal"
Sep 11 21:05:32.842422 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.842403    2578 manager.go:354] Recovery completed
Sep 11 21:05:32.843882 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.843811    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-142-224.ec2.internal" not found
Sep 11 21:05:32.846448 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.846436    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:05:32.849490 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.849475    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-224.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:05:32.849548 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.849504    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-224.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:05:32.849548 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.849517    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-224.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:05:32.850009 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.849997    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:05:32.850009 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.850007    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 21:05:32.850104 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.850017    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 21:05:32.850104 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.850034    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 21:05:32.850767 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.850750    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-224.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:05:32.850836 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.850776    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-224.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:05:32.850836 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.850789    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-224.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:05:32.852257 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.852246    2578 policy_none.go:49] "None policy: Start"
Sep 11 21:05:32.852305 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.852260    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 21:05:32.852305 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.852271    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 21:05:32.854230 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.854218    2578 policy_none.go:47] "Start"
Sep 11 21:05:32.889279 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.889262    2578 manager.go:337] "Starting Device Plugin manager"
Sep 11 21:05:32.908524 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:32.889308    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 21:05:32.908524 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.889322    2578 server.go:87] "Starting device plugin registration server"
Sep 11 21:05:32.908524 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.889653    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 21:05:32.908524 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.889664    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 21:05:32.908524 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.889733    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 21:05:32.908524 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.889805    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 21:05:32.908524 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.889814    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 21:05:32.908524 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:32.890426    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 21:05:32.908524 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:32.890479    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-224.ec2.internal\" not found"
Sep 11 21:05:32.908524 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.894707    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 21:05:32.908524 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.895808    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 21:05:32.908524 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.895854    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 21:05:32.908524 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.895872    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 21:05:32.908524 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:32.895912    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 21:05:32.908524 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.898185    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:05:32.908524 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.908522    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-142-224.ec2.internal" not found
Sep 11 21:05:32.989821 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.989791    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:05:32.990936 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.990920    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-224.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:05:32.991046 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.990952    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-224.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:05:32.991046 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.990996    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-224.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:05:32.991046 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.991034    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-224.ec2.internal"
Sep 11 21:05:32.996277 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.996259    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-224.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-224.ec2.internal"]
Sep 11 21:05:32.996364 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.996329    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:05:32.997227 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.997213    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-224.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:05:32.997341 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.997239    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-224.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:05:32.997341 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.997264    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-224.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:05:32.999465 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.999450    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:05:32.999632 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.999618    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-224.ec2.internal"
Sep 11 21:05:32.999669 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.999664    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:05:32.999943 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:32.999916    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-224.ec2.internal"
Sep 11 21:05:33.000248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:33.000223    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-224.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:05:33.000340 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:33.000255    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-224.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:05:33.000340 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:33.000231    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-224.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:05:33.000340 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:33.000268    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-224.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:05:33.000340 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:33.000285    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-224.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:05:33.000340 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:33.000298    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-224.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:05:33.002559 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:33.002542    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-224.ec2.internal"
Sep 11 21:05:33.019268 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:33.019251    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-224.ec2.internal"
Sep 11 21:05:33.022986 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:33.022969    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-224.ec2.internal"
Sep 11 21:05:33.029027 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:33.029011    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-224.ec2.internal"
Sep 11 21:05:33.036312 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:33.036295    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 21:05:33.037437 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:33.037424    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 21:05:33.044860 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:33.044842    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 21:05:33.044944 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:33.044883    2578 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-224.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-224.ec2.internal"
Sep 11 21:05:33.044944 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:33.044899    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-224.ec2.internal"
Sep 11 21:05:33.058487 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:33.058472    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 21:05:33.058562 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:33.058505    2578 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-224.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-224.ec2.internal"
Sep 11 21:05:33.129865 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:33.129797    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0f7f776ba9d1d40ea4f5268b9e77f2fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-224.ec2.internal\" (UID: \"0f7f776ba9d1d40ea4f5268b9e77f2fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-224.ec2.internal"
Sep 11 21:05:33.129865 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:33.129841    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f7f776ba9d1d40ea4f5268b9e77f2fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-224.ec2.internal\" (UID: \"0f7f776ba9d1d40ea4f5268b9e77f2fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-224.ec2.internal"
Sep 11 21:05:33.129865 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:33.129864    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f18dae482bb48fb8ae2e58c513d48e19-config\") pod \"kube-apiserver-proxy-ip-10-0-142-224.ec2.internal\" (UID: \"f18dae482bb48fb8ae2e58c513d48e19\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-224.ec2.internal"
Sep 11 21:05:34.256070 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.256039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0f7f776ba9d1d40ea4f5268b9e77f2fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-224.ec2.internal\" (UID: \"0f7f776ba9d1d40ea4f5268b9e77f2fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-224.ec2.internal"
Sep 11 21:05:34.256070 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.256073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f7f776ba9d1d40ea4f5268b9e77f2fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-224.ec2.internal\" (UID: \"0f7f776ba9d1d40ea4f5268b9e77f2fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-224.ec2.internal"
Sep 11 21:05:34.256245 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.256090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f18dae482bb48fb8ae2e58c513d48e19-config\") pod \"kube-apiserver-proxy-ip-10-0-142-224.ec2.internal\" (UID: \"f18dae482bb48fb8ae2e58c513d48e19\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-224.ec2.internal"
Sep 11 21:05:34.256245 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.256124    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f18dae482bb48fb8ae2e58c513d48e19-config\") pod \"kube-apiserver-proxy-ip-10-0-142-224.ec2.internal\" (UID: \"f18dae482bb48fb8ae2e58c513d48e19\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-224.ec2.internal"
Sep 11 21:05:34.256245 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.256136    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f7f776ba9d1d40ea4f5268b9e77f2fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-224.ec2.internal\" (UID: \"0f7f776ba9d1d40ea4f5268b9e77f2fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-224.ec2.internal"
Sep 11 21:05:34.256245 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.256158    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0f7f776ba9d1d40ea4f5268b9e77f2fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-224.ec2.internal\" (UID: \"0f7f776ba9d1d40ea4f5268b9e77f2fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-224.ec2.internal"
Sep 11 21:05:34.365554 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.365520    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-224.ec2.internal"
Sep 11 21:05:34.367688 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.367666    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-224.ec2.internal"
Sep 11 21:05:34.641548 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.641462    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 21:05:34.642042 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.641605    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 21:05:34.642042 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.641626    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 21:05:34.642042 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.641638    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 21:05:34.829973 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.829924    2578 apiserver.go:52] "Watching apiserver"
Sep 11 21:05:34.830145 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.829920    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 21:00:33 +0000 UTC" deadline="2028-04-17 19:16:50.911186359 +0000 UTC"
Sep 11 21:05:34.830145 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.830033    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14014h11m16.081157136s"
Sep 11 21:05:34.840218 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.840190    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 21:05:34.840534 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.840513    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-mbqxp","kube-system/kube-apiserver-proxy-ip-10-0-142-224.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm","openshift-cluster-node-tuning-operator/tuned-r2lqc","openshift-dns/node-resolver-chmvt","openshift-multus/multus-ngfwx","openshift-multus/network-metrics-daemon-4gct2","openshift-network-diagnostics/network-check-target-cjtsr","kube-system/konnectivity-agent-fx2dp","openshift-image-registry/node-ca-2dd4w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-224.ec2.internal","openshift-multus/multus-additional-cni-plugins-sptwp","openshift-network-operator/iptables-alerter-wl7nn","openshift-ovn-kubernetes/ovnkube-node-hk624"]
Sep 11 21:05:34.842984 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.842970    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:34.843092 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:34.843075    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbqxp" podUID="a06b3567-7898-44a3-b981-5d5180c6130e"
Sep 11 21:05:34.845058 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.845040    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm"
Sep 11 21:05:34.847101 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.847083    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.847365 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.847349    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xf542\""
Sep 11 21:05:34.847443 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.847350    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 21:05:34.847443 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.847389    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 21:05:34.847443 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.847403    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 21:05:34.848940 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.848924    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:05:34.849126 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.849110    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q8xbd\""
Sep 11 21:05:34.849343 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.849330    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-chmvt"
Sep 11 21:05:34.849573 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.849557    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 21:05:34.851136 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.851121    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 21:05:34.851338 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.851325    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 21:05:34.851408 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.851340    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.851460 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.851417    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qzc66\""
Sep 11 21:05:34.852660 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.852648    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 21:05:34.853333 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.853313    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 21:05:34.853439 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.853420    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 21:05:34.853495 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.853473    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:05:34.853561 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:34.853543    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4gct2" podUID="03071acd-860e-4da8-b813-079c6af540b6"
Sep 11 21:05:34.853815 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.853799    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 21:05:34.853893 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.853858    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 21:05:34.854096 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.854079    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pjmdx\""
Sep 11 21:05:34.855704 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.855685    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:05:34.855799 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:34.855754    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjtsr" podUID="c94aef67-aa61-440f-9e7a-a8fc58611ca8"
Sep 11 21:05:34.858096 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.857835    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fx2dp"
Sep 11 21:05:34.858879 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.858814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/281570ce-f91a-4db6-b516-dbe6a0664e64-device-dir\") pod \"aws-ebs-csi-driver-node-wjrmm\" (UID: \"281570ce-f91a-4db6-b516-dbe6a0664e64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm"
Sep 11 21:05:34.858879 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.858862    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-etc-sysconfig\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.859030 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.858890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-lib-modules\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.859030 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.858944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-etc-kubernetes\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.859030 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.858970    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdm9p\" (UniqueName: \"kubernetes.io/projected/03071acd-860e-4da8-b813-079c6af540b6-kube-api-access-tdm9p\") pod \"network-metrics-daemon-4gct2\" (UID: \"03071acd-860e-4da8-b813-079c6af540b6\") " pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:05:34.859030 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.858999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/281570ce-f91a-4db6-b516-dbe6a0664e64-sys-fs\") pod \"aws-ebs-csi-driver-node-wjrmm\" (UID: \"281570ce-f91a-4db6-b516-dbe6a0664e64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm"
Sep 11 21:05:34.859205 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-run\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.859205 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859086    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-multus-socket-dir-parent\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.859205 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859139    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-host-var-lib-cni-bin\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.859205 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859162    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/281570ce-f91a-4db6-b516-dbe6a0664e64-registration-dir\") pod \"aws-ebs-csi-driver-node-wjrmm\" (UID: \"281570ce-f91a-4db6-b516-dbe6a0664e64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm"
Sep 11 21:05:34.859205 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/281570ce-f91a-4db6-b516-dbe6a0664e64-etc-selinux\") pod \"aws-ebs-csi-driver-node-wjrmm\" (UID: \"281570ce-f91a-4db6-b516-dbe6a0664e64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm"
Sep 11 21:05:34.859433 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859227    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-host\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.859433 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-cnibin\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.859433 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859276    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-host-run-k8s-cni-cncf-io\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.859433 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859303    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mcn4\" (UniqueName: \"kubernetes.io/projected/281570ce-f91a-4db6-b516-dbe6a0664e64-kube-api-access-6mcn4\") pod \"aws-ebs-csi-driver-node-wjrmm\" (UID: \"281570ce-f91a-4db6-b516-dbe6a0664e64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm"
Sep 11 21:05:34.859433 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859323    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a195b677-21de-4546-808e-ea0d6d54704e-tmp\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.859433 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859362    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkqs7\" (UniqueName: \"kubernetes.io/projected/7c9e9fa2-dc0c-4832-9841-fe2468faea5a-kube-api-access-xkqs7\") pod \"node-resolver-chmvt\" (UID: \"7c9e9fa2-dc0c-4832-9841-fe2468faea5a\") " pod="openshift-dns/node-resolver-chmvt"
Sep 11 21:05:34.859433 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859396    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a06b3567-7898-44a3-b981-5d5180c6130e-dbus\") pod \"global-pull-secret-syncer-mbqxp\" (UID: \"a06b3567-7898-44a3-b981-5d5180c6130e\") " pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:34.859433 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859423    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-host-run-multus-certs\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.859733 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859449    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/281570ce-f91a-4db6-b516-dbe6a0664e64-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wjrmm\" (UID: \"281570ce-f91a-4db6-b516-dbe6a0664e64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm"
Sep 11 21:05:34.859733 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859472    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-etc-systemd\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.859733 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859502    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-system-cni-dir\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.859733 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859535    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/674ba88c-8fb0-440e-b4e9-9ee972de3473-cni-binary-copy\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.859733 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-host-var-lib-kubelet\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.859733 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859598    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44l7n\" (UniqueName: \"kubernetes.io/projected/c94aef67-aa61-440f-9e7a-a8fc58611ca8-kube-api-access-44l7n\") pod \"network-check-target-cjtsr\" (UID: \"c94aef67-aa61-440f-9e7a-a8fc58611ca8\") " pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:05:34.859733 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859626    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-etc-modprobe-d\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.859733 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-etc-sysctl-d\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.859733 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859673    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-var-lib-kubelet\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.859733 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859695    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7c9e9fa2-dc0c-4832-9841-fe2468faea5a-hosts-file\") pod \"node-resolver-chmvt\" (UID: \"7c9e9fa2-dc0c-4832-9841-fe2468faea5a\") " pod="openshift-dns/node-resolver-chmvt"
Sep 11 21:05:34.859733 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859718    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a06b3567-7898-44a3-b981-5d5180c6130e-original-pull-secret\") pod \"global-pull-secret-syncer-mbqxp\" (UID: \"a06b3567-7898-44a3-b981-5d5180c6130e\") " pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:34.860221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-hostroot\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.860221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859791    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/674ba88c-8fb0-440e-b4e9-9ee972de3473-multus-daemon-config\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.860221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03071acd-860e-4da8-b813-079c6af540b6-metrics-certs\") pod \"network-metrics-daemon-4gct2\" (UID: \"03071acd-860e-4da8-b813-079c6af540b6\") " pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:05:34.860221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859929    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-etc-kubernetes\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.860221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-sys\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.860221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.859995    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a195b677-21de-4546-808e-ea0d6d54704e-etc-tuned\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.860221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.860046    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2dd4w"
Sep 11 21:05:34.860221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.860063    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjx99\" (UniqueName: \"kubernetes.io/projected/a195b677-21de-4546-808e-ea0d6d54704e-kube-api-access-vjx99\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.860221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.860112    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a06b3567-7898-44a3-b981-5d5180c6130e-kubelet-config\") pod \"global-pull-secret-syncer-mbqxp\" (UID: \"a06b3567-7898-44a3-b981-5d5180c6130e\") " pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:34.860221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.860138    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 21:05:34.860221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.860153    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-multus-cni-dir\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.860221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.860174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-host-run-netns\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.860221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.860192    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 21:05:34.860221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.860213    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-host-var-lib-cni-multus\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.860814 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.860241    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-etc-sysctl-conf\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.860814 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.860268    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c9e9fa2-dc0c-4832-9841-fe2468faea5a-tmp-dir\") pod \"node-resolver-chmvt\" (UID: \"7c9e9fa2-dc0c-4832-9841-fe2468faea5a\") " pod="openshift-dns/node-resolver-chmvt"
Sep 11 21:05:34.860814 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.860322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-multus-conf-dir\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.860814 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.860349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln9ng\" (UniqueName: \"kubernetes.io/projected/674ba88c-8fb0-440e-b4e9-9ee972de3473-kube-api-access-ln9ng\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.860814 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.860371    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/281570ce-f91a-4db6-b516-dbe6a0664e64-socket-dir\") pod \"aws-ebs-csi-driver-node-wjrmm\" (UID: \"281570ce-f91a-4db6-b516-dbe6a0664e64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm"
Sep 11 21:05:34.860814 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.860630    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mtxvm\""
Sep 11 21:05:34.862607 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.862514    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sptwp"
Sep 11 21:05:34.862607 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.862602    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 21:05:34.862919 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.862644    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 21:05:34.862919 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.862806    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 21:05:34.862919 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.862834    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xfxkc\""
Sep 11 21:05:34.864604 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.864556    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 21:05:34.864689 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.864631    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 21:05:34.864742 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.864720    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wl7nn"
Sep 11 21:05:34.864914 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.864886    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tmkkr\""
Sep 11 21:05:34.867414 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.867388    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hx7qn\""
Sep 11 21:05:34.867774 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.867595    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 21:05:34.867774 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.867650    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:34.867774 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.867435    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 21:05:34.869569 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.867652    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:05:34.872388 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.872366    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 21:05:34.872479 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.872401    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 21:05:34.872479 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.872457    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 21:05:34.872589 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.872484    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 21:05:34.872589 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.872537    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 21:05:34.872589 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.872545    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 21:05:34.872764 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.872747    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rcb92\""
Sep 11 21:05:34.872799 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.872780    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 21:05:34.887700 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.887558    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8wb78"
Sep 11 21:05:34.896628 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.896595    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8wb78"
Sep 11 21:05:34.954557 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.954528    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 21:05:34.961588 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.961563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/281570ce-f91a-4db6-b516-dbe6a0664e64-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wjrmm\" (UID: \"281570ce-f91a-4db6-b516-dbe6a0664e64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm"
Sep 11 21:05:34.961703 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.961601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-etc-systemd\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.961703 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.961626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/674ba88c-8fb0-440e-b4e9-9ee972de3473-cni-binary-copy\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.961870 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.961723    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-etc-systemd\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.961870 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.961736    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44l7n\" (UniqueName: \"kubernetes.io/projected/c94aef67-aa61-440f-9e7a-a8fc58611ca8-kube-api-access-44l7n\") pod \"network-check-target-cjtsr\" (UID: \"c94aef67-aa61-440f-9e7a-a8fc58611ca8\") " pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:05:34.961870 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.961769    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/281570ce-f91a-4db6-b516-dbe6a0664e64-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wjrmm\" (UID: \"281570ce-f91a-4db6-b516-dbe6a0664e64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm"
Sep 11 21:05:34.961870 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.961793    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5c8ceec8-5ea9-41ef-864c-c6e65ea6f483-serviceca\") pod \"node-ca-2dd4w\" (UID: \"5c8ceec8-5ea9-41ef-864c-c6e65ea6f483\") " pod="openshift-image-registry/node-ca-2dd4w"
Sep 11 21:05:34.961870 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.961835    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-host-run-netns\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:34.962079 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.961869    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8f5b49a6-5bb6-40bb-ba24-91886289da00-ovnkube-config\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:34.962079 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.961914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-etc-modprobe-d\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.962079 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.961941    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a06b3567-7898-44a3-b981-5d5180c6130e-original-pull-secret\") pod \"global-pull-secret-syncer-mbqxp\" (UID: \"a06b3567-7898-44a3-b981-5d5180c6130e\") " pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:34.962079 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.961964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-hostroot\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.962079 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962022    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-hostroot\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.962079 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/674ba88c-8fb0-440e-b4e9-9ee972de3473-multus-daemon-config\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.962079 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962071    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-etc-modprobe-d\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.962380 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:34.962089    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:05:34.962380 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:34.962181    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a06b3567-7898-44a3-b981-5d5180c6130e-original-pull-secret podName:a06b3567-7898-44a3-b981-5d5180c6130e nodeName:}" failed. No retries permitted until 2026-09-11 21:05:35.462142667 +0000 UTC m=+2.166111953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a06b3567-7898-44a3-b981-5d5180c6130e-original-pull-secret") pod "global-pull-secret-syncer-mbqxp" (UID: "a06b3567-7898-44a3-b981-5d5180c6130e") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:05:34.962380 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:34.962187    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:05:34.962380 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962076    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03071acd-860e-4da8-b813-079c6af540b6-metrics-certs\") pod \"network-metrics-daemon-4gct2\" (UID: \"03071acd-860e-4da8-b813-079c6af540b6\") " pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:05:34.962380 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962214    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/674ba88c-8fb0-440e-b4e9-9ee972de3473-cni-binary-copy\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.962380 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962230    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-var-lib-openvswitch\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:34.962380 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:34.962237    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03071acd-860e-4da8-b813-079c6af540b6-metrics-certs podName:03071acd-860e-4da8-b813-079c6af540b6 nodeName:}" failed. No retries permitted until 2026-09-11 21:05:35.462219958 +0000 UTC m=+2.166189241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03071acd-860e-4da8-b813-079c6af540b6-metrics-certs") pod "network-metrics-daemon-4gct2" (UID: "03071acd-860e-4da8-b813-079c6af540b6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:05:34.962380 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-sys\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.962380 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962292    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjx99\" (UniqueName: \"kubernetes.io/projected/a195b677-21de-4546-808e-ea0d6d54704e-kube-api-access-vjx99\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.962380 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962341    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-multus-cni-dir\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.962380 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962346    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-sys\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.962380 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962365    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c3adb7a-bb53-45a7-a118-aa8478dcf7bc-konnectivity-ca\") pod \"konnectivity-agent-fx2dp\" (UID: \"1c3adb7a-bb53-45a7-a118-aa8478dcf7bc\") " pod="kube-system/konnectivity-agent-fx2dp"
Sep 11 21:05:34.962964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962388    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-host-kubelet\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:34.962964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962413    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-multus-cni-dir\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.962964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962426    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-run-ovn\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:34.962964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962451    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8f5b49a6-5bb6-40bb-ba24-91886289da00-env-overrides\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:34.962964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962495    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-etc-sysctl-conf\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.962964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962547    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-multus-conf-dir\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.962964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-etc-openvswitch\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:34.962964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962600    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-multus-conf-dir\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.962964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962605    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-etc-sysconfig\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.962964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962553    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/674ba88c-8fb0-440e-b4e9-9ee972de3473-multus-daemon-config\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.962964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962615    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-etc-sysctl-conf\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.962964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-etc-kubernetes\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.962964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962643    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-etc-sysconfig\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.962964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962652    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-etc-kubernetes\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.962964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962657    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdm9p\" (UniqueName: \"kubernetes.io/projected/03071acd-860e-4da8-b813-079c6af540b6-kube-api-access-tdm9p\") pod \"network-metrics-daemon-4gct2\" (UID: \"03071acd-860e-4da8-b813-079c6af540b6\") " pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:05:34.962964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962682    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx2dc\" (UniqueName: \"kubernetes.io/projected/5c8ceec8-5ea9-41ef-864c-c6e65ea6f483-kube-api-access-zx2dc\") pod \"node-ca-2dd4w\" (UID: \"5c8ceec8-5ea9-41ef-864c-c6e65ea6f483\") " pod="openshift-image-registry/node-ca-2dd4w"
Sep 11 21:05:34.962964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962710    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c15a91fd-755a-457c-aa8a-ac6b811025e4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sptwp\" (UID: \"c15a91fd-755a-457c-aa8a-ac6b811025e4\") " pod="openshift-multus/multus-additional-cni-plugins-sptwp"
Sep 11 21:05:34.963606 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962733    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c15a91fd-755a-457c-aa8a-ac6b811025e4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sptwp\" (UID: \"c15a91fd-755a-457c-aa8a-ac6b811025e4\") " pod="openshift-multus/multus-additional-cni-plugins-sptwp"
Sep 11 21:05:34.963606 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962755    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-run-openvswitch\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:34.963606 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-node-log\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:34.963606 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/281570ce-f91a-4db6-b516-dbe6a0664e64-registration-dir\") pod \"aws-ebs-csi-driver-node-wjrmm\" (UID: \"281570ce-f91a-4db6-b516-dbe6a0664e64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm"
Sep 11 21:05:34.963606 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962877    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-cnibin\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.963606 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962909    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/281570ce-f91a-4db6-b516-dbe6a0664e64-registration-dir\") pod \"aws-ebs-csi-driver-node-wjrmm\" (UID: \"281570ce-f91a-4db6-b516-dbe6a0664e64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm"
Sep 11 21:05:34.963606 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-host-run-k8s-cni-cncf-io\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.963606 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962942    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-host-cni-bin\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:34.963606 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962951    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-host-run-k8s-cni-cncf-io\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.963606 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.962978    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkqs7\" (UniqueName: \"kubernetes.io/projected/7c9e9fa2-dc0c-4832-9841-fe2468faea5a-kube-api-access-xkqs7\") pod \"node-resolver-chmvt\" (UID: \"7c9e9fa2-dc0c-4832-9841-fe2468faea5a\") " pod="openshift-dns/node-resolver-chmvt"
Sep 11 21:05:34.963606 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a06b3567-7898-44a3-b981-5d5180c6130e-dbus\") pod \"global-pull-secret-syncer-mbqxp\" (UID: \"a06b3567-7898-44a3-b981-5d5180c6130e\") " pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:34.963606 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963136    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-host-run-multus-certs\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.963606 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a06b3567-7898-44a3-b981-5d5180c6130e-dbus\") pod \"global-pull-secret-syncer-mbqxp\" (UID: \"a06b3567-7898-44a3-b981-5d5180c6130e\") " pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:34.963606 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-host-run-multus-certs\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.963606 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963407    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-cnibin\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.963606 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c3adb7a-bb53-45a7-a118-aa8478dcf7bc-agent-certs\") pod \"konnectivity-agent-fx2dp\" (UID: \"1c3adb7a-bb53-45a7-a118-aa8478dcf7bc\") " pod="kube-system/konnectivity-agent-fx2dp"
Sep 11 21:05:34.963606 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963428    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c8ceec8-5ea9-41ef-864c-c6e65ea6f483-host\") pod \"node-ca-2dd4w\" (UID: \"5c8ceec8-5ea9-41ef-864c-c6e65ea6f483\") " pod="openshift-image-registry/node-ca-2dd4w"
Sep 11 21:05:34.964255 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-system-cni-dir\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.964255 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963457    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-host-var-lib-kubelet\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.964255 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963473    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92f1cdb8-aec7-478d-9b69-59fa54dbdd3c-iptables-alerter-script\") pod \"iptables-alerter-wl7nn\" (UID: \"92f1cdb8-aec7-478d-9b69-59fa54dbdd3c\") " pod="openshift-network-operator/iptables-alerter-wl7nn"
Sep 11 21:05:34.964255 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-etc-sysctl-d\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.964255 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963532    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-host-var-lib-kubelet\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.964255 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963534    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-system-cni-dir\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.964255 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-var-lib-kubelet\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.964255 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7c9e9fa2-dc0c-4832-9841-fe2468faea5a-hosts-file\") pod \"node-resolver-chmvt\" (UID: \"7c9e9fa2-dc0c-4832-9841-fe2468faea5a\") " pod="openshift-dns/node-resolver-chmvt"
Sep 11 21:05:34.964255 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963601    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-systemd-units\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:34.964255 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963614    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-etc-sysctl-d\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.964255 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963630    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-var-lib-kubelet\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.964255 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-etc-kubernetes\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.964255 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963662    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7c9e9fa2-dc0c-4832-9841-fe2468faea5a-hosts-file\") pod \"node-resolver-chmvt\" (UID: \"7c9e9fa2-dc0c-4832-9841-fe2468faea5a\") " pod="openshift-dns/node-resolver-chmvt"
Sep 11 21:05:34.964255 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963664    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a195b677-21de-4546-808e-ea0d6d54704e-etc-tuned\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.964255 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963709    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a06b3567-7898-44a3-b981-5d5180c6130e-kubelet-config\") pod \"global-pull-secret-syncer-mbqxp\" (UID: \"a06b3567-7898-44a3-b981-5d5180c6130e\") " pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:34.964255 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963739    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-etc-kubernetes\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.964255 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963776    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-host-run-netns\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.964255 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963804    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a06b3567-7898-44a3-b981-5d5180c6130e-kubelet-config\") pod \"global-pull-secret-syncer-mbqxp\" (UID: \"a06b3567-7898-44a3-b981-5d5180c6130e\") " pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:34.965003 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963814    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-host-var-lib-cni-multus\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.965003 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963861    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-host-var-lib-cni-multus\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.965003 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln9ng\" (UniqueName: \"kubernetes.io/projected/674ba88c-8fb0-440e-b4e9-9ee972de3473-kube-api-access-ln9ng\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.965003 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963831    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-host-run-netns\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.965003 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.963977    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4flf\" (UniqueName: \"kubernetes.io/projected/92f1cdb8-aec7-478d-9b69-59fa54dbdd3c-kube-api-access-n4flf\") pod \"iptables-alerter-wl7nn\" (UID: \"92f1cdb8-aec7-478d-9b69-59fa54dbdd3c\") " pod="openshift-network-operator/iptables-alerter-wl7nn"
Sep 11 21:05:34.965003 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964017    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t7zw\" (UniqueName: \"kubernetes.io/projected/c15a91fd-755a-457c-aa8a-ac6b811025e4-kube-api-access-7t7zw\") pod \"multus-additional-cni-plugins-sptwp\" (UID: \"c15a91fd-755a-457c-aa8a-ac6b811025e4\") " pod="openshift-multus/multus-additional-cni-plugins-sptwp"
Sep 11 21:05:34.965003 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c9e9fa2-dc0c-4832-9841-fe2468faea5a-tmp-dir\") pod \"node-resolver-chmvt\" (UID: \"7c9e9fa2-dc0c-4832-9841-fe2468faea5a\") " pod="openshift-dns/node-resolver-chmvt"
Sep 11 21:05:34.965003 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964090    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c15a91fd-755a-457c-aa8a-ac6b811025e4-cnibin\") pod \"multus-additional-cni-plugins-sptwp\" (UID: \"c15a91fd-755a-457c-aa8a-ac6b811025e4\") " pod="openshift-multus/multus-additional-cni-plugins-sptwp"
Sep 11 21:05:34.965003 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c15a91fd-755a-457c-aa8a-ac6b811025e4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sptwp\" (UID: \"c15a91fd-755a-457c-aa8a-ac6b811025e4\") " pod="openshift-multus/multus-additional-cni-plugins-sptwp"
Sep 11 21:05:34.965003 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-host-slash\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:34.965003 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/281570ce-f91a-4db6-b516-dbe6a0664e64-socket-dir\") pod \"aws-ebs-csi-driver-node-wjrmm\" (UID: \"281570ce-f91a-4db6-b516-dbe6a0664e64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm"
Sep 11 21:05:34.965003 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/281570ce-f91a-4db6-b516-dbe6a0664e64-device-dir\") pod \"aws-ebs-csi-driver-node-wjrmm\" (UID: \"281570ce-f91a-4db6-b516-dbe6a0664e64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm"
Sep 11 21:05:34.965003 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964221    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-lib-modules\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.965003 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c15a91fd-755a-457c-aa8a-ac6b811025e4-cni-binary-copy\") pod \"multus-additional-cni-plugins-sptwp\" (UID: \"c15a91fd-755a-457c-aa8a-ac6b811025e4\") " pod="openshift-multus/multus-additional-cni-plugins-sptwp"
Sep 11 21:05:34.965003 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964253    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 21:05:34.965003 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964269    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-run-systemd\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:34.965003 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964292    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-log-socket\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:34.965738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964320    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c9e9fa2-dc0c-4832-9841-fe2468faea5a-tmp-dir\") pod \"node-resolver-chmvt\" (UID: \"7c9e9fa2-dc0c-4832-9841-fe2468faea5a\") " pod="openshift-dns/node-resolver-chmvt"
Sep 11 21:05:34.965738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964274    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/281570ce-f91a-4db6-b516-dbe6a0664e64-device-dir\") pod \"aws-ebs-csi-driver-node-wjrmm\" (UID: \"281570ce-f91a-4db6-b516-dbe6a0664e64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm"
Sep 11 21:05:34.965738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964319    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:34.965738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/281570ce-f91a-4db6-b516-dbe6a0664e64-sys-fs\") pod \"aws-ebs-csi-driver-node-wjrmm\" (UID: \"281570ce-f91a-4db6-b516-dbe6a0664e64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm"
Sep 11 21:05:34.965738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964399    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-run\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.965738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-multus-socket-dir-parent\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.965738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964435    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-lib-modules\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.965738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964473    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/281570ce-f91a-4db6-b516-dbe6a0664e64-sys-fs\") pod \"aws-ebs-csi-driver-node-wjrmm\" (UID: \"281570ce-f91a-4db6-b516-dbe6a0664e64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm"
Sep 11 21:05:34.965738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964474    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-multus-socket-dir-parent\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.965738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964491    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-host-var-lib-cni-bin\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.965738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964515    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/674ba88c-8fb0-440e-b4e9-9ee972de3473-host-var-lib-cni-bin\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:34.965738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964473    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-run\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.965738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964543    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-host-cni-netd\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:34.965738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f5b49a6-5bb6-40bb-ba24-91886289da00-ovn-node-metrics-cert\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:34.965738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964583    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/281570ce-f91a-4db6-b516-dbe6a0664e64-socket-dir\") pod \"aws-ebs-csi-driver-node-wjrmm\" (UID: \"281570ce-f91a-4db6-b516-dbe6a0664e64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm"
Sep 11 21:05:34.965738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/281570ce-f91a-4db6-b516-dbe6a0664e64-etc-selinux\") pod \"aws-ebs-csi-driver-node-wjrmm\" (UID: \"281570ce-f91a-4db6-b516-dbe6a0664e64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm"
Sep 11 21:05:34.965738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964650    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-host\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.966531 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92f1cdb8-aec7-478d-9b69-59fa54dbdd3c-host-slash\") pod \"iptables-alerter-wl7nn\" (UID: \"92f1cdb8-aec7-478d-9b69-59fa54dbdd3c\") " pod="openshift-network-operator/iptables-alerter-wl7nn"
Sep 11 21:05:34.966531 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c15a91fd-755a-457c-aa8a-ac6b811025e4-system-cni-dir\") pod \"multus-additional-cni-plugins-sptwp\" (UID: \"c15a91fd-755a-457c-aa8a-ac6b811025e4\") " pod="openshift-multus/multus-additional-cni-plugins-sptwp"
Sep 11 21:05:34.966531 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964709    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a195b677-21de-4546-808e-ea0d6d54704e-host\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.966531 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964728    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-host-run-ovn-kubernetes\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:34.966531 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964729    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/281570ce-f91a-4db6-b516-dbe6a0664e64-etc-selinux\") pod \"aws-ebs-csi-driver-node-wjrmm\" (UID: \"281570ce-f91a-4db6-b516-dbe6a0664e64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm"
Sep 11 21:05:34.966531 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964761    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k55hn\" (UniqueName: \"kubernetes.io/projected/8f5b49a6-5bb6-40bb-ba24-91886289da00-kube-api-access-k55hn\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:34.966531 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964793    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mcn4\" (UniqueName: \"kubernetes.io/projected/281570ce-f91a-4db6-b516-dbe6a0664e64-kube-api-access-6mcn4\") pod \"aws-ebs-csi-driver-node-wjrmm\" (UID: \"281570ce-f91a-4db6-b516-dbe6a0664e64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm"
Sep 11 21:05:34.966531 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a195b677-21de-4546-808e-ea0d6d54704e-tmp\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.966531 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.964932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8f5b49a6-5bb6-40bb-ba24-91886289da00-ovnkube-script-lib\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:34.967856 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.967834    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a195b677-21de-4546-808e-ea0d6d54704e-tmp\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.967856 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.967845    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a195b677-21de-4546-808e-ea0d6d54704e-etc-tuned\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.971429 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:34.971401    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:05:34.971429 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:34.971434    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:05:34.971643 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:34.971448    2578 projected.go:196] Error preparing data for projected volume kube-api-access-44l7n for pod openshift-network-diagnostics/network-check-target-cjtsr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:05:34.971643 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:34.971515    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c94aef67-aa61-440f-9e7a-a8fc58611ca8-kube-api-access-44l7n podName:c94aef67-aa61-440f-9e7a-a8fc58611ca8 nodeName:}" failed. No retries permitted until 2026-09-11 21:05:35.471496735 +0000 UTC m=+2.175466019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-44l7n" (UniqueName: "kubernetes.io/projected/c94aef67-aa61-440f-9e7a-a8fc58611ca8-kube-api-access-44l7n") pod "network-check-target-cjtsr" (UID: "c94aef67-aa61-440f-9e7a-a8fc58611ca8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:05:34.972865 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.972841    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjx99\" (UniqueName: \"kubernetes.io/projected/a195b677-21de-4546-808e-ea0d6d54704e-kube-api-access-vjx99\") pod \"tuned-r2lqc\" (UID: \"a195b677-21de-4546-808e-ea0d6d54704e\") " pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:34.974207 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.974180    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mcn4\" (UniqueName: \"kubernetes.io/projected/281570ce-f91a-4db6-b516-dbe6a0664e64-kube-api-access-6mcn4\") pod \"aws-ebs-csi-driver-node-wjrmm\" (UID: \"281570ce-f91a-4db6-b516-dbe6a0664e64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm"
Sep 11 21:05:34.974284 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.974251    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkqs7\" (UniqueName: \"kubernetes.io/projected/7c9e9fa2-dc0c-4832-9841-fe2468faea5a-kube-api-access-xkqs7\") pod \"node-resolver-chmvt\" (UID: \"7c9e9fa2-dc0c-4832-9841-fe2468faea5a\") " pod="openshift-dns/node-resolver-chmvt"
Sep 11 21:05:34.974864 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.974808    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdm9p\" (UniqueName: \"kubernetes.io/projected/03071acd-860e-4da8-b813-079c6af540b6-kube-api-access-tdm9p\") pod \"network-metrics-daemon-4gct2\" (UID: \"03071acd-860e-4da8-b813-079c6af540b6\") " pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:05:34.974980 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:34.974964    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln9ng\" (UniqueName: \"kubernetes.io/projected/674ba88c-8fb0-440e-b4e9-9ee972de3473-kube-api-access-ln9ng\") pod \"multus-ngfwx\" (UID: \"674ba88c-8fb0-440e-b4e9-9ee972de3473\") " pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:35.066006 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.065859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-systemd-units\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.066097 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066026    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4flf\" (UniqueName: \"kubernetes.io/projected/92f1cdb8-aec7-478d-9b69-59fa54dbdd3c-kube-api-access-n4flf\") pod \"iptables-alerter-wl7nn\" (UID: \"92f1cdb8-aec7-478d-9b69-59fa54dbdd3c\") " pod="openshift-network-operator/iptables-alerter-wl7nn"
Sep 11 21:05:35.066097 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066045    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t7zw\" (UniqueName: \"kubernetes.io/projected/c15a91fd-755a-457c-aa8a-ac6b811025e4-kube-api-access-7t7zw\") pod \"multus-additional-cni-plugins-sptwp\" (UID: \"c15a91fd-755a-457c-aa8a-ac6b811025e4\") " pod="openshift-multus/multus-additional-cni-plugins-sptwp"
Sep 11 21:05:35.066097 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c15a91fd-755a-457c-aa8a-ac6b811025e4-cnibin\") pod \"multus-additional-cni-plugins-sptwp\" (UID: \"c15a91fd-755a-457c-aa8a-ac6b811025e4\") " pod="openshift-multus/multus-additional-cni-plugins-sptwp"
Sep 11 21:05:35.066245 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.065964    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-systemd-units\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.066245 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066129    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c15a91fd-755a-457c-aa8a-ac6b811025e4-cnibin\") pod \"multus-additional-cni-plugins-sptwp\" (UID: \"c15a91fd-755a-457c-aa8a-ac6b811025e4\") " pod="openshift-multus/multus-additional-cni-plugins-sptwp"
Sep 11 21:05:35.066245 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c15a91fd-755a-457c-aa8a-ac6b811025e4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sptwp\" (UID: \"c15a91fd-755a-457c-aa8a-ac6b811025e4\") " pod="openshift-multus/multus-additional-cni-plugins-sptwp"
Sep 11 21:05:35.066245 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066184    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-host-slash\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.066245 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066222    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-host-slash\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.066472 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c15a91fd-755a-457c-aa8a-ac6b811025e4-cni-binary-copy\") pod \"multus-additional-cni-plugins-sptwp\" (UID: \"c15a91fd-755a-457c-aa8a-ac6b811025e4\") " pod="openshift-multus/multus-additional-cni-plugins-sptwp"
Sep 11 21:05:35.066472 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066331    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-run-systemd\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.066472 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066352    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-log-socket\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.066472 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066392    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-log-socket\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.066472 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066398    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-run-systemd\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.066472 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066449    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.066739 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-host-cni-netd\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.066739 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f5b49a6-5bb6-40bb-ba24-91886289da00-ovn-node-metrics-cert\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.066739 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066529    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92f1cdb8-aec7-478d-9b69-59fa54dbdd3c-host-slash\") pod \"iptables-alerter-wl7nn\" (UID: \"92f1cdb8-aec7-478d-9b69-59fa54dbdd3c\") " pod="openshift-network-operator/iptables-alerter-wl7nn"
Sep 11 21:05:35.066739 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066531    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.066739 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c15a91fd-755a-457c-aa8a-ac6b811025e4-system-cni-dir\") pod \"multus-additional-cni-plugins-sptwp\" (UID: \"c15a91fd-755a-457c-aa8a-ac6b811025e4\") " pod="openshift-multus/multus-additional-cni-plugins-sptwp"
Sep 11 21:05:35.066739 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-host-run-ovn-kubernetes\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.066739 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066602    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92f1cdb8-aec7-478d-9b69-59fa54dbdd3c-host-slash\") pod \"iptables-alerter-wl7nn\" (UID: \"92f1cdb8-aec7-478d-9b69-59fa54dbdd3c\") " pod="openshift-network-operator/iptables-alerter-wl7nn"
Sep 11 21:05:35.066739 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066607    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-host-run-ovn-kubernetes\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.066739 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k55hn\" (UniqueName: \"kubernetes.io/projected/8f5b49a6-5bb6-40bb-ba24-91886289da00-kube-api-access-k55hn\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.066739 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066627    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c15a91fd-755a-457c-aa8a-ac6b811025e4-system-cni-dir\") pod \"multus-additional-cni-plugins-sptwp\" (UID: \"c15a91fd-755a-457c-aa8a-ac6b811025e4\") " pod="openshift-multus/multus-additional-cni-plugins-sptwp"
Sep 11 21:05:35.066739 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066639    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c15a91fd-755a-457c-aa8a-ac6b811025e4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sptwp\" (UID: \"c15a91fd-755a-457c-aa8a-ac6b811025e4\") " pod="openshift-multus/multus-additional-cni-plugins-sptwp"
Sep 11 21:05:35.066739 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066645    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-host-cni-netd\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.066739 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8f5b49a6-5bb6-40bb-ba24-91886289da00-ovnkube-script-lib\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.066739 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066707    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5c8ceec8-5ea9-41ef-864c-c6e65ea6f483-serviceca\") pod \"node-ca-2dd4w\" (UID: \"5c8ceec8-5ea9-41ef-864c-c6e65ea6f483\") " pod="openshift-image-registry/node-ca-2dd4w"
Sep 11 21:05:35.066739 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-host-run-netns\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.066739 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8f5b49a6-5bb6-40bb-ba24-91886289da00-ovnkube-config\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.067543 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-var-lib-openvswitch\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.067543 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066793    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c15a91fd-755a-457c-aa8a-ac6b811025e4-cni-binary-copy\") pod \"multus-additional-cni-plugins-sptwp\" (UID: \"c15a91fd-755a-457c-aa8a-ac6b811025e4\") " pod="openshift-multus/multus-additional-cni-plugins-sptwp"
Sep 11 21:05:35.067543 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066797    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-host-run-netns\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.067543 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066844    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-var-lib-openvswitch\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.067543 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066930    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c3adb7a-bb53-45a7-a118-aa8478dcf7bc-konnectivity-ca\") pod \"konnectivity-agent-fx2dp\" (UID: \"1c3adb7a-bb53-45a7-a118-aa8478dcf7bc\") " pod="kube-system/konnectivity-agent-fx2dp"
Sep 11 21:05:35.067543 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-host-kubelet\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.067543 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.066988    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-run-ovn\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.067543 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067011    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8f5b49a6-5bb6-40bb-ba24-91886289da00-env-overrides\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.067543 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067022    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-host-kubelet\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.067543 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-etc-openvswitch\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.067543 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067065    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-run-ovn\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.067543 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067222    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5c8ceec8-5ea9-41ef-864c-c6e65ea6f483-serviceca\") pod \"node-ca-2dd4w\" (UID: \"5c8ceec8-5ea9-41ef-864c-c6e65ea6f483\") " pod="openshift-image-registry/node-ca-2dd4w"
Sep 11 21:05:35.067543 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx2dc\" (UniqueName: \"kubernetes.io/projected/5c8ceec8-5ea9-41ef-864c-c6e65ea6f483-kube-api-access-zx2dc\") pod \"node-ca-2dd4w\" (UID: \"5c8ceec8-5ea9-41ef-864c-c6e65ea6f483\") " pod="openshift-image-registry/node-ca-2dd4w"
Sep 11 21:05:35.067543 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067274    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-etc-openvswitch\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.067543 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067273    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c15a91fd-755a-457c-aa8a-ac6b811025e4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sptwp\" (UID: \"c15a91fd-755a-457c-aa8a-ac6b811025e4\") " pod="openshift-multus/multus-additional-cni-plugins-sptwp"
Sep 11 21:05:35.067543 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067305    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8f5b49a6-5bb6-40bb-ba24-91886289da00-ovnkube-config\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.067543 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067313    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8f5b49a6-5bb6-40bb-ba24-91886289da00-ovnkube-script-lib\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.068221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067308    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c15a91fd-755a-457c-aa8a-ac6b811025e4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sptwp\" (UID: \"c15a91fd-755a-457c-aa8a-ac6b811025e4\") " pod="openshift-multus/multus-additional-cni-plugins-sptwp"
Sep 11 21:05:35.068221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067348    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-run-openvswitch\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.068221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-node-log\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.068221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067401    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-host-cni-bin\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.068221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067401    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c15a91fd-755a-457c-aa8a-ac6b811025e4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sptwp\" (UID: \"c15a91fd-755a-457c-aa8a-ac6b811025e4\") " pod="openshift-multus/multus-additional-cni-plugins-sptwp"
Sep 11 21:05:35.068221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067428    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c3adb7a-bb53-45a7-a118-aa8478dcf7bc-agent-certs\") pod \"konnectivity-agent-fx2dp\" (UID: \"1c3adb7a-bb53-45a7-a118-aa8478dcf7bc\") " pod="kube-system/konnectivity-agent-fx2dp"
Sep 11 21:05:35.068221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067438    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-run-openvswitch\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.068221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067450    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c8ceec8-5ea9-41ef-864c-c6e65ea6f483-host\") pod \"node-ca-2dd4w\" (UID: \"5c8ceec8-5ea9-41ef-864c-c6e65ea6f483\") " pod="openshift-image-registry/node-ca-2dd4w"
Sep 11 21:05:35.068221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067451    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-node-log\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.068221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067472    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8f5b49a6-5bb6-40bb-ba24-91886289da00-env-overrides\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.068221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067476    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f5b49a6-5bb6-40bb-ba24-91886289da00-host-cni-bin\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.068221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067497    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92f1cdb8-aec7-478d-9b69-59fa54dbdd3c-iptables-alerter-script\") pod \"iptables-alerter-wl7nn\" (UID: \"92f1cdb8-aec7-478d-9b69-59fa54dbdd3c\") " pod="openshift-network-operator/iptables-alerter-wl7nn"
Sep 11 21:05:35.068221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067501    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c3adb7a-bb53-45a7-a118-aa8478dcf7bc-konnectivity-ca\") pod \"konnectivity-agent-fx2dp\" (UID: \"1c3adb7a-bb53-45a7-a118-aa8478dcf7bc\") " pod="kube-system/konnectivity-agent-fx2dp"
Sep 11 21:05:35.068221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067565    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c8ceec8-5ea9-41ef-864c-c6e65ea6f483-host\") pod \"node-ca-2dd4w\" (UID: \"5c8ceec8-5ea9-41ef-864c-c6e65ea6f483\") " pod="openshift-image-registry/node-ca-2dd4w"
Sep 11 21:05:35.068221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067720    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c15a91fd-755a-457c-aa8a-ac6b811025e4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sptwp\" (UID: \"c15a91fd-755a-457c-aa8a-ac6b811025e4\") " pod="openshift-multus/multus-additional-cni-plugins-sptwp"
Sep 11 21:05:35.068221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.067994    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92f1cdb8-aec7-478d-9b69-59fa54dbdd3c-iptables-alerter-script\") pod \"iptables-alerter-wl7nn\" (UID: \"92f1cdb8-aec7-478d-9b69-59fa54dbdd3c\") " pod="openshift-network-operator/iptables-alerter-wl7nn"
Sep 11 21:05:35.069105 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.069088    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f5b49a6-5bb6-40bb-ba24-91886289da00-ovn-node-metrics-cert\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.069432 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.069418    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c3adb7a-bb53-45a7-a118-aa8478dcf7bc-agent-certs\") pod \"konnectivity-agent-fx2dp\" (UID: \"1c3adb7a-bb53-45a7-a118-aa8478dcf7bc\") " pod="kube-system/konnectivity-agent-fx2dp"
Sep 11 21:05:35.074042 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:35.074019    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f7f776ba9d1d40ea4f5268b9e77f2fd.slice/crio-1388f5b3c615d0987203997d60a86038f76aac8ac8617bdbf8710f41b4830aa4 WatchSource:0}: Error finding container 1388f5b3c615d0987203997d60a86038f76aac8ac8617bdbf8710f41b4830aa4: Status 404 returned error can't find the container with id 1388f5b3c615d0987203997d60a86038f76aac8ac8617bdbf8710f41b4830aa4
Sep 11 21:05:35.074413 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.074397    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k55hn\" (UniqueName: \"kubernetes.io/projected/8f5b49a6-5bb6-40bb-ba24-91886289da00-kube-api-access-k55hn\") pod \"ovnkube-node-hk624\" (UID: \"8f5b49a6-5bb6-40bb-ba24-91886289da00\") " pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.074748 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.074723    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4flf\" (UniqueName: \"kubernetes.io/projected/92f1cdb8-aec7-478d-9b69-59fa54dbdd3c-kube-api-access-n4flf\") pod \"iptables-alerter-wl7nn\" (UID: \"92f1cdb8-aec7-478d-9b69-59fa54dbdd3c\") " pod="openshift-network-operator/iptables-alerter-wl7nn"
Sep 11 21:05:35.074815 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:35.074774    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf18dae482bb48fb8ae2e58c513d48e19.slice/crio-e93671c0849f3deb0efd7f2c6963080a1923bcc5fcfaa2ba2392518fb9c22140 WatchSource:0}: Error finding container e93671c0849f3deb0efd7f2c6963080a1923bcc5fcfaa2ba2392518fb9c22140: Status 404 returned error can't find the container with id e93671c0849f3deb0efd7f2c6963080a1923bcc5fcfaa2ba2392518fb9c22140
Sep 11 21:05:35.078492 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.078470    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t7zw\" (UniqueName: \"kubernetes.io/projected/c15a91fd-755a-457c-aa8a-ac6b811025e4-kube-api-access-7t7zw\") pod \"multus-additional-cni-plugins-sptwp\" (UID: \"c15a91fd-755a-457c-aa8a-ac6b811025e4\") " pod="openshift-multus/multus-additional-cni-plugins-sptwp"
Sep 11 21:05:35.078492 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.078480    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx2dc\" (UniqueName: \"kubernetes.io/projected/5c8ceec8-5ea9-41ef-864c-c6e65ea6f483-kube-api-access-zx2dc\") pod \"node-ca-2dd4w\" (UID: \"5c8ceec8-5ea9-41ef-864c-c6e65ea6f483\") " pod="openshift-image-registry/node-ca-2dd4w"
Sep 11 21:05:35.079364 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.079351    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:05:35.163929 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.163841    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm"
Sep 11 21:05:35.173533 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:35.173506    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod281570ce_f91a_4db6_b516_dbe6a0664e64.slice/crio-c30d7ad2a175c313b96b5a4e7ed1d3dfdb561dd86a4f4a5d453ffcb848156096 WatchSource:0}: Error finding container c30d7ad2a175c313b96b5a4e7ed1d3dfdb561dd86a4f4a5d453ffcb848156096: Status 404 returned error can't find the container with id c30d7ad2a175c313b96b5a4e7ed1d3dfdb561dd86a4f4a5d453ffcb848156096
Sep 11 21:05:35.177786 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.177764    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r2lqc"
Sep 11 21:05:35.185505 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:35.185486    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda195b677_21de_4546_808e_ea0d6d54704e.slice/crio-55f108235bdb337b5a5cdefb65eccffe29a8f0af4590097a1f6da0d69dd93485 WatchSource:0}: Error finding container 55f108235bdb337b5a5cdefb65eccffe29a8f0af4590097a1f6da0d69dd93485: Status 404 returned error can't find the container with id 55f108235bdb337b5a5cdefb65eccffe29a8f0af4590097a1f6da0d69dd93485
Sep 11 21:05:35.188304 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.188290    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-chmvt"
Sep 11 21:05:35.192055 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.192039    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ngfwx"
Sep 11 21:05:35.196373 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:35.196352    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c9e9fa2_dc0c_4832_9841_fe2468faea5a.slice/crio-d957d1a503c1b37505e1a7288b84e4fe104a013ddb538a9890ca9e74d24cd10d WatchSource:0}: Error finding container d957d1a503c1b37505e1a7288b84e4fe104a013ddb538a9890ca9e74d24cd10d: Status 404 returned error can't find the container with id d957d1a503c1b37505e1a7288b84e4fe104a013ddb538a9890ca9e74d24cd10d
Sep 11 21:05:35.200031 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:35.200012    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod674ba88c_8fb0_440e_b4e9_9ee972de3473.slice/crio-e6319b93a5aded2b51bee45a6163a02a1cfd09b2ee69e3fc42f4cd9990cdd2d7 WatchSource:0}: Error finding container e6319b93a5aded2b51bee45a6163a02a1cfd09b2ee69e3fc42f4cd9990cdd2d7: Status 404 returned error can't find the container with id e6319b93a5aded2b51bee45a6163a02a1cfd09b2ee69e3fc42f4cd9990cdd2d7
Sep 11 21:05:35.209752 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.209734    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fx2dp"
Sep 11 21:05:35.218296 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:35.218273    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3adb7a_bb53_45a7_a118_aa8478dcf7bc.slice/crio-17b996ea4ea410eae56a6d2005b02a5c5f811ad50155b20990b6c745203541cc WatchSource:0}: Error finding container 17b996ea4ea410eae56a6d2005b02a5c5f811ad50155b20990b6c745203541cc: Status 404 returned error can't find the container with id 17b996ea4ea410eae56a6d2005b02a5c5f811ad50155b20990b6c745203541cc
Sep 11 21:05:35.224829 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.224793    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2dd4w"
Sep 11 21:05:35.234450 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:35.234426    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8ceec8_5ea9_41ef_864c_c6e65ea6f483.slice/crio-6cf024beba907e152ac3b4f79c2bf3fa3af5217e3a3ed990a02496314c58b2dc WatchSource:0}: Error finding container 6cf024beba907e152ac3b4f79c2bf3fa3af5217e3a3ed990a02496314c58b2dc: Status 404 returned error can't find the container with id 6cf024beba907e152ac3b4f79c2bf3fa3af5217e3a3ed990a02496314c58b2dc
Sep 11 21:05:35.247103 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.247087    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sptwp"
Sep 11 21:05:35.255311 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:35.255289    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15a91fd_755a_457c_aa8a_ac6b811025e4.slice/crio-0ce043800a40d1bf1b2cc212e546816db2290d90a230f864d80234bdfa0ddbc7 WatchSource:0}: Error finding container 0ce043800a40d1bf1b2cc212e546816db2290d90a230f864d80234bdfa0ddbc7: Status 404 returned error can't find the container with id 0ce043800a40d1bf1b2cc212e546816db2290d90a230f864d80234bdfa0ddbc7
Sep 11 21:05:35.262832 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.262802    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wl7nn"
Sep 11 21:05:35.269544 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.269526    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:05:35.271446 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:35.271425    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f1cdb8_aec7_478d_9b69_59fa54dbdd3c.slice/crio-60b5b5d0c44312f22291fc3234e729c3a0fd2320616768d7abbac66d5f6964cf WatchSource:0}: Error finding container 60b5b5d0c44312f22291fc3234e729c3a0fd2320616768d7abbac66d5f6964cf: Status 404 returned error can't find the container with id 60b5b5d0c44312f22291fc3234e729c3a0fd2320616768d7abbac66d5f6964cf
Sep 11 21:05:35.277539 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:05:35.277515    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5b49a6_5bb6_40bb_ba24_91886289da00.slice/crio-ba3e99f410b4afdfafd27558244055f5031d8b6d7f9a59529edbd25af9944302 WatchSource:0}: Error finding container ba3e99f410b4afdfafd27558244055f5031d8b6d7f9a59529edbd25af9944302: Status 404 returned error can't find the container with id ba3e99f410b4afdfafd27558244055f5031d8b6d7f9a59529edbd25af9944302
Sep 11 21:05:35.279200 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:35.279182    2578 kuberuntime_manager.go:1459] "Unhandled Error" err="container ovn-controller start failed in pod ovnkube-node-hk624_openshift-ovn-kubernetes(8f5b49a6-5bb6-40bb-ba24-91886289da00): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 11 21:05:35.469494 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.469415    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a06b3567-7898-44a3-b981-5d5180c6130e-original-pull-secret\") pod \"global-pull-secret-syncer-mbqxp\" (UID: \"a06b3567-7898-44a3-b981-5d5180c6130e\") " pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:35.469494 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.469452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03071acd-860e-4da8-b813-079c6af540b6-metrics-certs\") pod \"network-metrics-daemon-4gct2\" (UID: \"03071acd-860e-4da8-b813-079c6af540b6\") " pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:05:35.469682 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:35.469663    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:05:35.469754 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:35.469742    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03071acd-860e-4da8-b813-079c6af540b6-metrics-certs podName:03071acd-860e-4da8-b813-079c6af540b6 nodeName:}" failed. No retries permitted until 2026-09-11 21:05:36.469723469 +0000 UTC m=+3.173692757 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03071acd-860e-4da8-b813-079c6af540b6-metrics-certs") pod "network-metrics-daemon-4gct2" (UID: "03071acd-860e-4da8-b813-079c6af540b6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:05:35.469754 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:35.469743    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:05:35.469843 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:35.469781    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a06b3567-7898-44a3-b981-5d5180c6130e-original-pull-secret podName:a06b3567-7898-44a3-b981-5d5180c6130e nodeName:}" failed. No retries permitted until 2026-09-11 21:05:36.469774077 +0000 UTC m=+3.173743344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a06b3567-7898-44a3-b981-5d5180c6130e-original-pull-secret") pod "global-pull-secret-syncer-mbqxp" (UID: "a06b3567-7898-44a3-b981-5d5180c6130e") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:05:35.519013 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.518972    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:05:35.570302 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.570268    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44l7n\" (UniqueName: \"kubernetes.io/projected/c94aef67-aa61-440f-9e7a-a8fc58611ca8-kube-api-access-44l7n\") pod \"network-check-target-cjtsr\" (UID: \"c94aef67-aa61-440f-9e7a-a8fc58611ca8\") " pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:05:35.570489 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:35.570472    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:05:35.570558 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:35.570497    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:05:35.570558 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:35.570510    2578 projected.go:196] Error preparing data for projected volume kube-api-access-44l7n for pod openshift-network-diagnostics/network-check-target-cjtsr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:05:35.570662 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:35.570574    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c94aef67-aa61-440f-9e7a-a8fc58611ca8-kube-api-access-44l7n podName:c94aef67-aa61-440f-9e7a-a8fc58611ca8 nodeName:}" failed. No retries permitted until 2026-09-11 21:05:36.570556438 +0000 UTC m=+3.274525725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-44l7n" (UniqueName: "kubernetes.io/projected/c94aef67-aa61-440f-9e7a-a8fc58611ca8-kube-api-access-44l7n") pod "network-check-target-cjtsr" (UID: "c94aef67-aa61-440f-9e7a-a8fc58611ca8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:05:35.905050 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.899877    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 21:00:34 +0000 UTC" deadline="2028-02-28 00:43:57.179624824 +0000 UTC"
Sep 11 21:05:35.905050 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.899930    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12819h38m21.279698201s"
Sep 11 21:05:35.935455 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.935394    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hk624" event={"ID":"8f5b49a6-5bb6-40bb-ba24-91886289da00","Type":"ContainerStarted","Data":"ba3e99f410b4afdfafd27558244055f5031d8b6d7f9a59529edbd25af9944302"}
Sep 11 21:05:35.935607 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.935518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wl7nn" event={"ID":"92f1cdb8-aec7-478d-9b69-59fa54dbdd3c","Type":"ContainerStarted","Data":"60b5b5d0c44312f22291fc3234e729c3a0fd2320616768d7abbac66d5f6964cf"}
Sep 11 21:05:35.936503 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.936474    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:05:35.936890 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.936864    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sptwp" event={"ID":"c15a91fd-755a-457c-aa8a-ac6b811025e4","Type":"ContainerStarted","Data":"0ce043800a40d1bf1b2cc212e546816db2290d90a230f864d80234bdfa0ddbc7"}
Sep 11 21:05:35.941891 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.941870    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fx2dp" event={"ID":"1c3adb7a-bb53-45a7-a118-aa8478dcf7bc","Type":"ContainerStarted","Data":"17b996ea4ea410eae56a6d2005b02a5c5f811ad50155b20990b6c745203541cc"}
Sep 11 21:05:35.951351 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.951325    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ngfwx" event={"ID":"674ba88c-8fb0-440e-b4e9-9ee972de3473","Type":"ContainerStarted","Data":"e6319b93a5aded2b51bee45a6163a02a1cfd09b2ee69e3fc42f4cd9990cdd2d7"}
Sep 11 21:05:35.953254 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.953228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-chmvt" event={"ID":"7c9e9fa2-dc0c-4832-9841-fe2468faea5a","Type":"ContainerStarted","Data":"d957d1a503c1b37505e1a7288b84e4fe104a013ddb538a9890ca9e74d24cd10d"}
Sep 11 21:05:35.956400 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.956373    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r2lqc" event={"ID":"a195b677-21de-4546-808e-ea0d6d54704e","Type":"ContainerStarted","Data":"55f108235bdb337b5a5cdefb65eccffe29a8f0af4590097a1f6da0d69dd93485"}
Sep 11 21:05:35.959127 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.959103    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm" event={"ID":"281570ce-f91a-4db6-b516-dbe6a0664e64","Type":"ContainerStarted","Data":"c30d7ad2a175c313b96b5a4e7ed1d3dfdb561dd86a4f4a5d453ffcb848156096"}
Sep 11 21:05:35.962497 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.962473    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2dd4w" event={"ID":"5c8ceec8-5ea9-41ef-864c-c6e65ea6f483","Type":"ContainerStarted","Data":"6cf024beba907e152ac3b4f79c2bf3fa3af5217e3a3ed990a02496314c58b2dc"}
Sep 11 21:05:35.967918 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.967746    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-224.ec2.internal" event={"ID":"f18dae482bb48fb8ae2e58c513d48e19","Type":"ContainerStarted","Data":"e93671c0849f3deb0efd7f2c6963080a1923bcc5fcfaa2ba2392518fb9c22140"}
Sep 11 21:05:35.970910 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:35.970866    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-224.ec2.internal" event={"ID":"0f7f776ba9d1d40ea4f5268b9e77f2fd","Type":"ContainerStarted","Data":"1388f5b3c615d0987203997d60a86038f76aac8ac8617bdbf8710f41b4830aa4"}
Sep 11 21:05:36.040559 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:36.040229    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:05:36.478198 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:36.478162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a06b3567-7898-44a3-b981-5d5180c6130e-original-pull-secret\") pod \"global-pull-secret-syncer-mbqxp\" (UID: \"a06b3567-7898-44a3-b981-5d5180c6130e\") " pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:36.478376 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:36.478214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03071acd-860e-4da8-b813-079c6af540b6-metrics-certs\") pod \"network-metrics-daemon-4gct2\" (UID: \"03071acd-860e-4da8-b813-079c6af540b6\") " pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:05:36.478439 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:36.478400    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:05:36.478503 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:36.478461    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03071acd-860e-4da8-b813-079c6af540b6-metrics-certs podName:03071acd-860e-4da8-b813-079c6af540b6 nodeName:}" failed. No retries permitted until 2026-09-11 21:05:38.478442987 +0000 UTC m=+5.182412268 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03071acd-860e-4da8-b813-079c6af540b6-metrics-certs") pod "network-metrics-daemon-4gct2" (UID: "03071acd-860e-4da8-b813-079c6af540b6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:05:36.478569 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:36.478531    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:05:36.478612 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:36.478567    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a06b3567-7898-44a3-b981-5d5180c6130e-original-pull-secret podName:a06b3567-7898-44a3-b981-5d5180c6130e nodeName:}" failed. No retries permitted until 2026-09-11 21:05:38.478556366 +0000 UTC m=+5.182525652 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a06b3567-7898-44a3-b981-5d5180c6130e-original-pull-secret") pod "global-pull-secret-syncer-mbqxp" (UID: "a06b3567-7898-44a3-b981-5d5180c6130e") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:05:36.579599 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:36.579560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44l7n\" (UniqueName: \"kubernetes.io/projected/c94aef67-aa61-440f-9e7a-a8fc58611ca8-kube-api-access-44l7n\") pod \"network-check-target-cjtsr\" (UID: \"c94aef67-aa61-440f-9e7a-a8fc58611ca8\") " pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:05:36.579788 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:36.579771    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:05:36.579882 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:36.579802    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:05:36.579882 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:36.579815    2578 projected.go:196] Error preparing data for projected volume kube-api-access-44l7n for pod openshift-network-diagnostics/network-check-target-cjtsr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:05:36.579986 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:36.579889    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c94aef67-aa61-440f-9e7a-a8fc58611ca8-kube-api-access-44l7n podName:c94aef67-aa61-440f-9e7a-a8fc58611ca8 nodeName:}" failed. No retries permitted until 2026-09-11 21:05:38.579870143 +0000 UTC m=+5.283839415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-44l7n" (UniqueName: "kubernetes.io/projected/c94aef67-aa61-440f-9e7a-a8fc58611ca8-kube-api-access-44l7n") pod "network-check-target-cjtsr" (UID: "c94aef67-aa61-440f-9e7a-a8fc58611ca8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:05:36.901065 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:36.900975    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 21:00:34 +0000 UTC" deadline="2028-02-17 07:42:37.628411203 +0000 UTC"
Sep 11 21:05:36.901065 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:36.901020    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12562h37m0.727396035s"
Sep 11 21:05:36.921489 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:36.921462    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:05:36.921978 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:36.921609    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjtsr" podUID="c94aef67-aa61-440f-9e7a-a8fc58611ca8"
Sep 11 21:05:36.922045 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:36.922032    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:36.922142 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:36.922118    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbqxp" podUID="a06b3567-7898-44a3-b981-5d5180c6130e"
Sep 11 21:05:36.922210 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:36.922188    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:05:36.922317 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:36.922265    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4gct2" podUID="03071acd-860e-4da8-b813-079c6af540b6"
Sep 11 21:05:38.494888 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:38.494851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a06b3567-7898-44a3-b981-5d5180c6130e-original-pull-secret\") pod \"global-pull-secret-syncer-mbqxp\" (UID: \"a06b3567-7898-44a3-b981-5d5180c6130e\") " pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:38.494888 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:38.494893    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03071acd-860e-4da8-b813-079c6af540b6-metrics-certs\") pod \"network-metrics-daemon-4gct2\" (UID: \"03071acd-860e-4da8-b813-079c6af540b6\") " pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:05:38.495499 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:38.495011    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:05:38.495499 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:38.495067    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03071acd-860e-4da8-b813-079c6af540b6-metrics-certs podName:03071acd-860e-4da8-b813-079c6af540b6 nodeName:}" failed. No retries permitted until 2026-09-11 21:05:42.495052398 +0000 UTC m=+9.199021665 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03071acd-860e-4da8-b813-079c6af540b6-metrics-certs") pod "network-metrics-daemon-4gct2" (UID: "03071acd-860e-4da8-b813-079c6af540b6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:05:38.495499 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:38.495461    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:05:38.495660 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:38.495509    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a06b3567-7898-44a3-b981-5d5180c6130e-original-pull-secret podName:a06b3567-7898-44a3-b981-5d5180c6130e nodeName:}" failed. No retries permitted until 2026-09-11 21:05:42.495491197 +0000 UTC m=+9.199460464 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a06b3567-7898-44a3-b981-5d5180c6130e-original-pull-secret") pod "global-pull-secret-syncer-mbqxp" (UID: "a06b3567-7898-44a3-b981-5d5180c6130e") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:05:38.596570 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:38.596137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44l7n\" (UniqueName: \"kubernetes.io/projected/c94aef67-aa61-440f-9e7a-a8fc58611ca8-kube-api-access-44l7n\") pod \"network-check-target-cjtsr\" (UID: \"c94aef67-aa61-440f-9e7a-a8fc58611ca8\") " pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:05:38.596570 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:38.596279    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:05:38.596570 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:38.596292    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:05:38.596570 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:38.596301    2578 projected.go:196] Error preparing data for projected volume kube-api-access-44l7n for pod openshift-network-diagnostics/network-check-target-cjtsr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:05:38.596570 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:38.596351    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c94aef67-aa61-440f-9e7a-a8fc58611ca8-kube-api-access-44l7n podName:c94aef67-aa61-440f-9e7a-a8fc58611ca8 nodeName:}" failed. No retries permitted until 2026-09-11 21:05:42.59633229 +0000 UTC m=+9.300301557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-44l7n" (UniqueName: "kubernetes.io/projected/c94aef67-aa61-440f-9e7a-a8fc58611ca8-kube-api-access-44l7n") pod "network-check-target-cjtsr" (UID: "c94aef67-aa61-440f-9e7a-a8fc58611ca8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:05:38.922289 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:38.921837    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:38.922289 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:38.921882    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:05:38.922289 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:38.921965    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbqxp" podUID="a06b3567-7898-44a3-b981-5d5180c6130e"
Sep 11 21:05:38.922289 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:38.922036    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:05:38.922289 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:38.922042    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4gct2" podUID="03071acd-860e-4da8-b813-079c6af540b6"
Sep 11 21:05:38.922289 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:38.922120    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjtsr" podUID="c94aef67-aa61-440f-9e7a-a8fc58611ca8"
Sep 11 21:05:40.921293 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:40.921254    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:40.921931 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:40.921304    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:05:40.921931 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:40.921392    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbqxp" podUID="a06b3567-7898-44a3-b981-5d5180c6130e"
Sep 11 21:05:40.921931 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:40.921254    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:05:40.921931 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:40.921507    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4gct2" podUID="03071acd-860e-4da8-b813-079c6af540b6"
Sep 11 21:05:40.921931 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:40.921588    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjtsr" podUID="c94aef67-aa61-440f-9e7a-a8fc58611ca8"
Sep 11 21:05:42.526146 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:42.526112    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a06b3567-7898-44a3-b981-5d5180c6130e-original-pull-secret\") pod \"global-pull-secret-syncer-mbqxp\" (UID: \"a06b3567-7898-44a3-b981-5d5180c6130e\") " pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:42.526146 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:42.526157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03071acd-860e-4da8-b813-079c6af540b6-metrics-certs\") pod \"network-metrics-daemon-4gct2\" (UID: \"03071acd-860e-4da8-b813-079c6af540b6\") " pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:05:42.526772 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:42.526316    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:05:42.526772 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:42.526371    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03071acd-860e-4da8-b813-079c6af540b6-metrics-certs podName:03071acd-860e-4da8-b813-079c6af540b6 nodeName:}" failed. No retries permitted until 2026-09-11 21:05:50.526355608 +0000 UTC m=+17.230324875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03071acd-860e-4da8-b813-079c6af540b6-metrics-certs") pod "network-metrics-daemon-4gct2" (UID: "03071acd-860e-4da8-b813-079c6af540b6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:05:42.526772 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:42.526748    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:05:42.526967 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:42.526838    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a06b3567-7898-44a3-b981-5d5180c6130e-original-pull-secret podName:a06b3567-7898-44a3-b981-5d5180c6130e nodeName:}" failed. No retries permitted until 2026-09-11 21:05:50.526802511 +0000 UTC m=+17.230771795 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a06b3567-7898-44a3-b981-5d5180c6130e-original-pull-secret") pod "global-pull-secret-syncer-mbqxp" (UID: "a06b3567-7898-44a3-b981-5d5180c6130e") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:05:42.627574 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:42.627484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44l7n\" (UniqueName: \"kubernetes.io/projected/c94aef67-aa61-440f-9e7a-a8fc58611ca8-kube-api-access-44l7n\") pod \"network-check-target-cjtsr\" (UID: \"c94aef67-aa61-440f-9e7a-a8fc58611ca8\") " pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:05:42.627749 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:42.627657    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:05:42.627749 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:42.627679    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:05:42.627749 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:42.627690    2578 projected.go:196] Error preparing data for projected volume kube-api-access-44l7n for pod openshift-network-diagnostics/network-check-target-cjtsr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:05:42.627749 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:42.627747    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c94aef67-aa61-440f-9e7a-a8fc58611ca8-kube-api-access-44l7n podName:c94aef67-aa61-440f-9e7a-a8fc58611ca8 nodeName:}" failed. No retries permitted until 2026-09-11 21:05:50.627730491 +0000 UTC m=+17.331699772 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-44l7n" (UniqueName: "kubernetes.io/projected/c94aef67-aa61-440f-9e7a-a8fc58611ca8-kube-api-access-44l7n") pod "network-check-target-cjtsr" (UID: "c94aef67-aa61-440f-9e7a-a8fc58611ca8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:05:42.922242 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:42.922160    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:05:42.922407 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:42.922160    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:42.922407 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:42.922303    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4gct2" podUID="03071acd-860e-4da8-b813-079c6af540b6"
Sep 11 21:05:42.922407 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:42.922160    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:05:42.922407 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:42.922396    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbqxp" podUID="a06b3567-7898-44a3-b981-5d5180c6130e"
Sep 11 21:05:42.922601 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:42.922430    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjtsr" podUID="c94aef67-aa61-440f-9e7a-a8fc58611ca8"
Sep 11 21:05:44.921830 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:44.921778    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:05:44.922238 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:44.921835    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:44.922238 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:44.921835    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:05:44.922238 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:44.921921    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjtsr" podUID="c94aef67-aa61-440f-9e7a-a8fc58611ca8"
Sep 11 21:05:44.922238 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:44.922036    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbqxp" podUID="a06b3567-7898-44a3-b981-5d5180c6130e"
Sep 11 21:05:44.922238 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:44.922123    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4gct2" podUID="03071acd-860e-4da8-b813-079c6af540b6"
Sep 11 21:05:46.922093 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:46.922059    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:05:46.922093 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:46.922085    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:46.922587 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:46.922085    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:05:46.922587 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:46.922185    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjtsr" podUID="c94aef67-aa61-440f-9e7a-a8fc58611ca8"
Sep 11 21:05:46.922587 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:46.922264    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbqxp" podUID="a06b3567-7898-44a3-b981-5d5180c6130e"
Sep 11 21:05:46.922587 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:46.922334    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4gct2" podUID="03071acd-860e-4da8-b813-079c6af540b6"
Sep 11 21:05:48.922222 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:48.922042    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:05:48.922916 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:48.922334    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4gct2" podUID="03071acd-860e-4da8-b813-079c6af540b6"
Sep 11 21:05:48.922916 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:48.922088    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:05:48.922916 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:48.922088    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:48.922916 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:48.922464    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjtsr" podUID="c94aef67-aa61-440f-9e7a-a8fc58611ca8"
Sep 11 21:05:48.922916 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:48.922563    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbqxp" podUID="a06b3567-7898-44a3-b981-5d5180c6130e"
Sep 11 21:05:49.001546 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:49.001470    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hk624" event={"ID":"8f5b49a6-5bb6-40bb-ba24-91886289da00","Type":"ContainerStarted","Data":"29ef82aaed78870a2e623a55f37bc43b2e6435e2c5049da96c65a5e0d8286e2f"}
Sep 11 21:05:49.001697 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:49.001546    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hk624" event={"ID":"8f5b49a6-5bb6-40bb-ba24-91886289da00","Type":"ContainerStarted","Data":"920b6d8d0d19eb1def3e998a7106de59f16e3507831df6572a78c94dfc614a1f"}
Sep 11 21:05:49.002947 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:49.002923    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ngfwx" event={"ID":"674ba88c-8fb0-440e-b4e9-9ee972de3473","Type":"ContainerStarted","Data":"2c53ba4b9ccb1ec34dd430586235fd6c5e086aa39df0d9a8425f01f5aab9d3c3"}
Sep 11 21:05:49.004338 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:49.004315    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r2lqc" event={"ID":"a195b677-21de-4546-808e-ea0d6d54704e","Type":"ContainerStarted","Data":"3b9eb040bf5e0e78bc2f6a1a80332712e6273b8da0d855657e9338651b6c5760"}
Sep 11 21:05:49.005893 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:49.005868    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-224.ec2.internal" event={"ID":"f18dae482bb48fb8ae2e58c513d48e19","Type":"ContainerStarted","Data":"232447896fe5c6d87357fedfcc64cc8ab62fedc44acf208aed53e2ab976c8a3c"}
Sep 11 21:05:49.027465 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:49.027382    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ngfwx" podStartSLOduration=1.909339065 podStartE2EDuration="15.027367719s" totalImagesPullingTime="13.118028654s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:05:49.027237051 +0000 UTC m=+15.731206340" watchObservedRunningTime="2026-09-11 21:05:49.027367719 +0000 UTC m=+15.731337010"
Sep 11 21:05:49.049648 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:49.049590    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-r2lqc" podStartSLOduration=2.40877458 podStartE2EDuration="15.049570942s" totalImagesPullingTime="12.640796362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:05:49.049338702 +0000 UTC m=+15.753307994" watchObservedRunningTime="2026-09-11 21:05:49.049570942 +0000 UTC m=+15.753540233"
Sep 11 21:05:49.065708 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:49.065667    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-224.ec2.internal" podStartSLOduration=15.065653371 podStartE2EDuration="15.065653371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:05:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:05:49.065520242 +0000 UTC m=+15.769489531" watchObservedRunningTime="2026-09-11 21:05:49.065653371 +0000 UTC m=+15.769622660"
Sep 11 21:05:50.010687 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:50.010649    2578 generic.go:356] "Generic (PLEG): container finished" podID="0f7f776ba9d1d40ea4f5268b9e77f2fd" containerID="91e9af61507326d026de778a39fe87a8e66528848c5a62bb25f59e03999111b7" exitCode=0
Sep 11 21:05:50.011279 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:50.010741    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-224.ec2.internal" event={"ID":"0f7f776ba9d1d40ea4f5268b9e77f2fd","Type":"ContainerDied","Data":"91e9af61507326d026de778a39fe87a8e66528848c5a62bb25f59e03999111b7"}
Sep 11 21:05:50.013445 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:50.013230    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wl7nn" event={"ID":"92f1cdb8-aec7-478d-9b69-59fa54dbdd3c","Type":"ContainerStarted","Data":"a49678c429e615606d2c921a5571819108c6cf55035842fe89695dfe043a4e27"}
Sep 11 21:05:50.017268 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:50.017239    2578 generic.go:356] "Generic (PLEG): container finished" podID="c15a91fd-755a-457c-aa8a-ac6b811025e4" containerID="8a9033735e06e3873ae35c1fa5ecd8ba5f6f4f54b3db94da757ad9d792b11786" exitCode=0
Sep 11 21:05:50.017372 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:50.017316    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sptwp" event={"ID":"c15a91fd-755a-457c-aa8a-ac6b811025e4","Type":"ContainerDied","Data":"8a9033735e06e3873ae35c1fa5ecd8ba5f6f4f54b3db94da757ad9d792b11786"}
Sep 11 21:05:50.020563 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:50.020529    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fx2dp" event={"ID":"1c3adb7a-bb53-45a7-a118-aa8478dcf7bc","Type":"ContainerStarted","Data":"f75bbf491442e868518e14bcb50631857089554d6038b1e4216f1a67ca4fca37"}
Sep 11 21:05:50.025355 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:50.025304    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-chmvt" event={"ID":"7c9e9fa2-dc0c-4832-9841-fe2468faea5a","Type":"ContainerStarted","Data":"ced80aea3fea03085c9e00ff7cbdccc2306af8894e4049f92318b22bdd456258"}
Sep 11 21:05:50.028328 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:50.028303    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm" event={"ID":"281570ce-f91a-4db6-b516-dbe6a0664e64","Type":"ContainerStarted","Data":"09db515b9d27f5fcad471f47f10eb8d1c7ba3ae905d52d965cd3c3edd6519c8c"}
Sep 11 21:05:50.030731 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:50.030706    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2dd4w" event={"ID":"5c8ceec8-5ea9-41ef-864c-c6e65ea6f483","Type":"ContainerStarted","Data":"c0e7bd5a8efd023e927213a936aec428ec30829b6919c1ac1864bd12c334bb66"}
Sep 11 21:05:50.045808 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:50.045762    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-chmvt" podStartSLOduration=3.427849792 podStartE2EDuration="16.045748645s" totalImagesPullingTime="12.617898853s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:05:50.045554374 +0000 UTC m=+16.749523665" watchObservedRunningTime="2026-09-11 21:05:50.045748645 +0000 UTC m=+16.749717929"
Sep 11 21:05:50.061068 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:50.061020    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fx2dp" podStartSLOduration=3.4668796459999998 podStartE2EDuration="16.061003328s" totalImagesPullingTime="12.594123682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:05:50.060524438 +0000 UTC m=+16.764493729" watchObservedRunningTime="2026-09-11 21:05:50.061003328 +0000 UTC m=+16.764972618"
Sep 11 21:05:50.161460 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:50.161400    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wl7nn" podStartSLOduration=3.622483957 podStartE2EDuration="16.161376896s" totalImagesPullingTime="12.538892939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:05:50.110795867 +0000 UTC m=+16.814765155" watchObservedRunningTime="2026-09-11 21:05:50.161376896 +0000 UTC m=+16.865346180"
Sep 11 21:05:50.162356 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:50.162313    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2dd4w" podStartSLOduration=3.584159461 podStartE2EDuration="16.162299123s" totalImagesPullingTime="12.578139662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:05:50.161498092 +0000 UTC m=+16.865467386" watchObservedRunningTime="2026-09-11 21:05:50.162299123 +0000 UTC m=+16.866268412"
Sep 11 21:05:50.590111 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:50.590040    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a06b3567-7898-44a3-b981-5d5180c6130e-original-pull-secret\") pod \"global-pull-secret-syncer-mbqxp\" (UID: \"a06b3567-7898-44a3-b981-5d5180c6130e\") " pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:50.590111 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:50.590094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03071acd-860e-4da8-b813-079c6af540b6-metrics-certs\") pod \"network-metrics-daemon-4gct2\" (UID: \"03071acd-860e-4da8-b813-079c6af540b6\") " pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:05:50.590450 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:50.590257    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:05:50.590450 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:50.590319    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03071acd-860e-4da8-b813-079c6af540b6-metrics-certs podName:03071acd-860e-4da8-b813-079c6af540b6 nodeName:}" failed. No retries permitted until 2026-09-11 21:06:06.590301301 +0000 UTC m=+33.294270572 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03071acd-860e-4da8-b813-079c6af540b6-metrics-certs") pod "network-metrics-daemon-4gct2" (UID: "03071acd-860e-4da8-b813-079c6af540b6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:05:50.590735 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:50.590715    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:05:50.590839 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:50.590770    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a06b3567-7898-44a3-b981-5d5180c6130e-original-pull-secret podName:a06b3567-7898-44a3-b981-5d5180c6130e nodeName:}" failed. No retries permitted until 2026-09-11 21:06:06.590755338 +0000 UTC m=+33.294724608 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a06b3567-7898-44a3-b981-5d5180c6130e-original-pull-secret") pod "global-pull-secret-syncer-mbqxp" (UID: "a06b3567-7898-44a3-b981-5d5180c6130e") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:05:50.691099 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:50.691067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44l7n\" (UniqueName: \"kubernetes.io/projected/c94aef67-aa61-440f-9e7a-a8fc58611ca8-kube-api-access-44l7n\") pod \"network-check-target-cjtsr\" (UID: \"c94aef67-aa61-440f-9e7a-a8fc58611ca8\") " pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:05:50.691303 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:50.691282    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:05:50.691384 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:50.691314    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:05:50.691384 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:50.691326    2578 projected.go:196] Error preparing data for projected volume kube-api-access-44l7n for pod openshift-network-diagnostics/network-check-target-cjtsr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:05:50.691384 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:50.691382    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c94aef67-aa61-440f-9e7a-a8fc58611ca8-kube-api-access-44l7n podName:c94aef67-aa61-440f-9e7a-a8fc58611ca8 nodeName:}" failed. No retries permitted until 2026-09-11 21:06:06.691368605 +0000 UTC m=+33.395337890 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-44l7n" (UniqueName: "kubernetes.io/projected/c94aef67-aa61-440f-9e7a-a8fc58611ca8-kube-api-access-44l7n") pod "network-check-target-cjtsr" (UID: "c94aef67-aa61-440f-9e7a-a8fc58611ca8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:05:50.921254 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:50.921228    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:05:50.921364 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:50.921265    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:50.921364 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:50.921265    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:05:50.921364 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:50.921354    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjtsr" podUID="c94aef67-aa61-440f-9e7a-a8fc58611ca8"
Sep 11 21:05:50.921523 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:50.921498    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbqxp" podUID="a06b3567-7898-44a3-b981-5d5180c6130e"
Sep 11 21:05:50.921665 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:50.921640    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4gct2" podUID="03071acd-860e-4da8-b813-079c6af540b6"
Sep 11 21:05:51.023451 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:51.023423    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 21:05:51.034333 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:51.034306    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm" event={"ID":"281570ce-f91a-4db6-b516-dbe6a0664e64","Type":"ContainerStarted","Data":"1f36d945a9833ff6d2efe111a2ad79a064dc896fe973de033d7ed9ba0927b349"}
Sep 11 21:05:51.036332 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:51.036307    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-224.ec2.internal" event={"ID":"0f7f776ba9d1d40ea4f5268b9e77f2fd","Type":"ContainerStarted","Data":"93163bb74246ad3f1c82855c858321ae6c96ec3e12cac28e05a9d6efd3be8f24"}
Sep 11 21:05:51.056223 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:51.056187    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-224.ec2.internal" podStartSLOduration=17.056174946 podStartE2EDuration="17.056174946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:05:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:05:51.05584298 +0000 UTC m=+17.759812266" watchObservedRunningTime="2026-09-11 21:05:51.056174946 +0000 UTC m=+17.760144234"
Sep 11 21:05:51.924651 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:51.924333    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T21:05:51.023447925Z","UUID":"085998e1-07ab-49d7-8489-eca41b41144e","Handler":null,"Name":"","Endpoint":""}
Sep 11 21:05:51.927689 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:51.927668    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 21:05:51.927860 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:51.927716    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 21:05:52.922104 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:52.922069    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:05:52.922104 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:52.922108    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:52.922669 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:52.922174    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:05:52.922669 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:52.922196    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjtsr" podUID="c94aef67-aa61-440f-9e7a-a8fc58611ca8"
Sep 11 21:05:52.922669 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:52.922263    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4gct2" podUID="03071acd-860e-4da8-b813-079c6af540b6"
Sep 11 21:05:52.922669 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:52.922337    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbqxp" podUID="a06b3567-7898-44a3-b981-5d5180c6130e"
Sep 11 21:05:53.043444 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:53.043403    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm" event={"ID":"281570ce-f91a-4db6-b516-dbe6a0664e64","Type":"ContainerStarted","Data":"413ba4ee252d83df8f7cad42dd9275b59586b38fb1c768afe58f2d749aa13472"}
Sep 11 21:05:53.067720 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:53.067668    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjrmm" podStartSLOduration=3.490855831 podStartE2EDuration="19.06765059s" totalImagesPullingTime="15.576794759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:05:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:05:53.067450403 +0000 UTC m=+19.771419693" watchObservedRunningTime="2026-09-11 21:05:53.06765059 +0000 UTC m=+19.771619879"
Sep 11 21:05:54.406760 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:54.406721    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fx2dp"
Sep 11 21:05:54.407470 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:54.407347    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fx2dp"
Sep 11 21:05:54.921394 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:54.921363    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:05:54.921579 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:54.921416    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:05:54.921579 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:54.921511    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4gct2" podUID="03071acd-860e-4da8-b813-079c6af540b6"
Sep 11 21:05:54.921733 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:54.921621    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:54.921733 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:54.921720    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbqxp" podUID="a06b3567-7898-44a3-b981-5d5180c6130e"
Sep 11 21:05:54.921860 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:54.921801    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjtsr" podUID="c94aef67-aa61-440f-9e7a-a8fc58611ca8"
Sep 11 21:05:55.049421 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:55.049042    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fx2dp"
Sep 11 21:05:55.049591 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:55.049577    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fx2dp"
Sep 11 21:05:56.053001 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:56.052967    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sptwp" event={"ID":"c15a91fd-755a-457c-aa8a-ac6b811025e4","Type":"ContainerStarted","Data":"9cdb3038aea1fc456df05ae86187484e219391700793459e0126d54faf361850"}
Sep 11 21:05:56.922186 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:56.922152    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:05:56.922368 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:56.922152    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:56.922368 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:56.922274    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjtsr" podUID="c94aef67-aa61-440f-9e7a-a8fc58611ca8"
Sep 11 21:05:56.922368 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:56.922283    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:05:56.922368 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:56.922359    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbqxp" podUID="a06b3567-7898-44a3-b981-5d5180c6130e"
Sep 11 21:05:56.922518 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:56.922451    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4gct2" podUID="03071acd-860e-4da8-b813-079c6af540b6"
Sep 11 21:05:58.921627 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:58.921582    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:05:58.922190 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:58.921694    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:05:58.922190 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:58.921705    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjtsr" podUID="c94aef67-aa61-440f-9e7a-a8fc58611ca8"
Sep 11 21:05:58.922190 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:58.921720    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:05:58.922190 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:58.921789    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbqxp" podUID="a06b3567-7898-44a3-b981-5d5180c6130e"
Sep 11 21:05:58.922190 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:05:58.921882    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4gct2" podUID="03071acd-860e-4da8-b813-079c6af540b6"
Sep 11 21:05:59.062150 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:59.062121    2578 generic.go:356] "Generic (PLEG): container finished" podID="c15a91fd-755a-457c-aa8a-ac6b811025e4" containerID="9cdb3038aea1fc456df05ae86187484e219391700793459e0126d54faf361850" exitCode=0
Sep 11 21:05:59.062324 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:05:59.062176    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sptwp" event={"ID":"c15a91fd-755a-457c-aa8a-ac6b811025e4","Type":"ContainerDied","Data":"9cdb3038aea1fc456df05ae86187484e219391700793459e0126d54faf361850"}
Sep 11 21:06:00.067468 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:00.067394    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hk624" event={"ID":"8f5b49a6-5bb6-40bb-ba24-91886289da00","Type":"ContainerStarted","Data":"fe9d2556b6b8a9e5a2595ee9cbffe9383bbd67a34ca500504f4124d033e1e9d2"}
Sep 11 21:06:00.067864 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:00.067522    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hk624" event={"ID":"8f5b49a6-5bb6-40bb-ba24-91886289da00","Type":"ContainerStarted","Data":"c92944d0744188c35805bd08a0815f08319d9a513a7a7749afbe81b1b7f447c5"}
Sep 11 21:06:00.921525 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:00.921495    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:06:00.921643 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:00.921588    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:06:00.921643 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:00.921594    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjtsr" podUID="c94aef67-aa61-440f-9e7a-a8fc58611ca8"
Sep 11 21:06:00.921738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:00.921596    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:06:00.921738 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:00.921684    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbqxp" podUID="a06b3567-7898-44a3-b981-5d5180c6130e"
Sep 11 21:06:00.921806 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:00.921751    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4gct2" podUID="03071acd-860e-4da8-b813-079c6af540b6"
Sep 11 21:06:01.071390 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:01.071308    2578 generic.go:356] "Generic (PLEG): container finished" podID="c15a91fd-755a-457c-aa8a-ac6b811025e4" containerID="14c3efcaf13b7c7d511a221c8bd54f67224f11652083f149fa330643365e85f6" exitCode=0
Sep 11 21:06:01.071390 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:01.071365    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sptwp" event={"ID":"c15a91fd-755a-457c-aa8a-ac6b811025e4","Type":"ContainerDied","Data":"14c3efcaf13b7c7d511a221c8bd54f67224f11652083f149fa330643365e85f6"}
Sep 11 21:06:02.076391 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:02.076196    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hk624" event={"ID":"8f5b49a6-5bb6-40bb-ba24-91886289da00","Type":"ContainerStarted","Data":"0194cb5a9081276d6ea560b301fe1347993224ab8c64235b31fee87c7bdd01bc"}
Sep 11 21:06:02.921362 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:02.921331    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:06:02.921528 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:02.921421    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:06:02.921528 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:02.921425    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjtsr" podUID="c94aef67-aa61-440f-9e7a-a8fc58611ca8"
Sep 11 21:06:02.921528 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:02.921446    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:06:02.921528 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:02.921522    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbqxp" podUID="a06b3567-7898-44a3-b981-5d5180c6130e"
Sep 11 21:06:02.921670 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:02.921597    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4gct2" podUID="03071acd-860e-4da8-b813-079c6af540b6"
Sep 11 21:06:03.080714 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:03.080682    2578 generic.go:356] "Generic (PLEG): container finished" podID="c15a91fd-755a-457c-aa8a-ac6b811025e4" containerID="a207f7894b13c97fcb62f08e83d4b0fd13a6ce3aac8a13f40d2264ec8d39f7ce" exitCode=0
Sep 11 21:06:03.081105 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:03.080740    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sptwp" event={"ID":"c15a91fd-755a-457c-aa8a-ac6b811025e4","Type":"ContainerDied","Data":"a207f7894b13c97fcb62f08e83d4b0fd13a6ce3aac8a13f40d2264ec8d39f7ce"}
Sep 11 21:06:03.989212 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:03.989169    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1de7a8eeb36fff2326d9f3af3f515b1b617c93675b1149831949cab0cf40a2c6\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-hk624" podUID="8f5b49a6-5bb6-40bb-ba24-91886289da00"
Sep 11 21:06:04.087486 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:04.087445    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hk624" event={"ID":"8f5b49a6-5bb6-40bb-ba24-91886289da00","Type":"ContainerStarted","Data":"bc97cfbf261e34ce97ddf26f763717213808e02bbdf33c340b622b75cc36ce09"}
Sep 11 21:06:04.087961 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:04.087922    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:06:04.088017 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:04.087990    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:06:04.088061 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:04.088034    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:06:04.108317 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:04.108281    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:06:04.108929 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:04.108906    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:06:04.921576 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:04.921343    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:06:04.921779 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:04.921364    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:06:04.921779 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:04.921669    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjtsr" podUID="c94aef67-aa61-440f-9e7a-a8fc58611ca8"
Sep 11 21:06:04.921779 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:04.921422    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:06:04.921779 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:04.921768    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbqxp" podUID="a06b3567-7898-44a3-b981-5d5180c6130e"
Sep 11 21:06:04.921941 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:04.921836    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4gct2" podUID="03071acd-860e-4da8-b813-079c6af540b6"
Sep 11 21:06:05.094778 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:05.094741    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hk624" event={"ID":"8f5b49a6-5bb6-40bb-ba24-91886289da00","Type":"ContainerStarted","Data":"6a27b9bf2db5127eae7e59b4200c0a2d67116bac05450f79d2b6d123f83566f2"}
Sep 11 21:06:05.095327 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:05.094839    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hk624" event={"ID":"8f5b49a6-5bb6-40bb-ba24-91886289da00","Type":"ContainerStarted","Data":"099d8e679d544eeb223d0859c39fa239a85c32472a7c236d894a39bddf81acbe"}
Sep 11 21:06:05.125249 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:05.125179    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hk624" podStartSLOduration=6.978799599 podStartE2EDuration="31.125158401s" totalImagesPullingTime="24.146358802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:05:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-11 21:06:05.12510204 +0000 UTC m=+31.829071326" watchObservedRunningTime="2026-09-11 21:06:05.125158401 +0000 UTC m=+31.829127692"
Sep 11 21:06:05.880322 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:05.880287    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mbqxp"]
Sep 11 21:06:05.880495 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:05.880441    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:06:05.880563 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:05.880540    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbqxp" podUID="a06b3567-7898-44a3-b981-5d5180c6130e"
Sep 11 21:06:05.884872 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:05.884815    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cjtsr"]
Sep 11 21:06:05.885034 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:05.884991    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:06:05.885140 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:05.885101    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjtsr" podUID="c94aef67-aa61-440f-9e7a-a8fc58611ca8"
Sep 11 21:06:05.885578 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:05.885554    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4gct2"]
Sep 11 21:06:05.885680 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:05.885663    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:06:05.885771 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:05.885745    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4gct2" podUID="03071acd-860e-4da8-b813-079c6af540b6"
Sep 11 21:06:06.599858 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:06.599809    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a06b3567-7898-44a3-b981-5d5180c6130e-original-pull-secret\") pod \"global-pull-secret-syncer-mbqxp\" (UID: \"a06b3567-7898-44a3-b981-5d5180c6130e\") " pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:06:06.599858 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:06.599863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03071acd-860e-4da8-b813-079c6af540b6-metrics-certs\") pod \"network-metrics-daemon-4gct2\" (UID: \"03071acd-860e-4da8-b813-079c6af540b6\") " pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:06:06.600630 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:06.599970    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:06:06.600630 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:06.599973    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:06:06.600630 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:06.600030    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03071acd-860e-4da8-b813-079c6af540b6-metrics-certs podName:03071acd-860e-4da8-b813-079c6af540b6 nodeName:}" failed. No retries permitted until 2026-09-11 21:06:38.600013443 +0000 UTC m=+65.303982732 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03071acd-860e-4da8-b813-079c6af540b6-metrics-certs") pod "network-metrics-daemon-4gct2" (UID: "03071acd-860e-4da8-b813-079c6af540b6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:06:06.600630 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:06.600051    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a06b3567-7898-44a3-b981-5d5180c6130e-original-pull-secret podName:a06b3567-7898-44a3-b981-5d5180c6130e nodeName:}" failed. No retries permitted until 2026-09-11 21:06:38.600040913 +0000 UTC m=+65.304010180 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a06b3567-7898-44a3-b981-5d5180c6130e-original-pull-secret") pod "global-pull-secret-syncer-mbqxp" (UID: "a06b3567-7898-44a3-b981-5d5180c6130e") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:06:06.700786 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:06.700749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44l7n\" (UniqueName: \"kubernetes.io/projected/c94aef67-aa61-440f-9e7a-a8fc58611ca8-kube-api-access-44l7n\") pod \"network-check-target-cjtsr\" (UID: \"c94aef67-aa61-440f-9e7a-a8fc58611ca8\") " pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:06:06.701058 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:06.700945    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:06:06.701058 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:06.700971    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:06:06.701058 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:06.700986    2578 projected.go:196] Error preparing data for projected volume kube-api-access-44l7n for pod openshift-network-diagnostics/network-check-target-cjtsr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:06:06.701209 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:06.701060    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c94aef67-aa61-440f-9e7a-a8fc58611ca8-kube-api-access-44l7n podName:c94aef67-aa61-440f-9e7a-a8fc58611ca8 nodeName:}" failed. No retries permitted until 2026-09-11 21:06:38.701038569 +0000 UTC m=+65.405007839 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-44l7n" (UniqueName: "kubernetes.io/projected/c94aef67-aa61-440f-9e7a-a8fc58611ca8-kube-api-access-44l7n") pod "network-check-target-cjtsr" (UID: "c94aef67-aa61-440f-9e7a-a8fc58611ca8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:06:07.921746 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:07.921655    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:06:07.921746 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:07.921704    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:06:07.922399 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:07.921672    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:06:07.922399 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:07.921777    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbqxp" podUID="a06b3567-7898-44a3-b981-5d5180c6130e"
Sep 11 21:06:07.922399 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:07.921849    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4gct2" podUID="03071acd-860e-4da8-b813-079c6af540b6"
Sep 11 21:06:07.922399 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:07.921919    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjtsr" podUID="c94aef67-aa61-440f-9e7a-a8fc58611ca8"
Sep 11 21:06:08.103998 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:08.103969    2578 generic.go:356] "Generic (PLEG): container finished" podID="c15a91fd-755a-457c-aa8a-ac6b811025e4" containerID="8ed39be1a62bf950a9cd276cf28ddd71d4d65507e88544627db1f8beb4524345" exitCode=0
Sep 11 21:06:08.104139 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:08.104018    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sptwp" event={"ID":"c15a91fd-755a-457c-aa8a-ac6b811025e4","Type":"ContainerDied","Data":"8ed39be1a62bf950a9cd276cf28ddd71d4d65507e88544627db1f8beb4524345"}
Sep 11 21:06:09.109758 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:09.109549    2578 generic.go:356] "Generic (PLEG): container finished" podID="c15a91fd-755a-457c-aa8a-ac6b811025e4" containerID="2152ff9559deaa39222a6d7f66e02650325de9d354e8fbb2810463433bd696a7" exitCode=0
Sep 11 21:06:09.109758 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:09.109621    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sptwp" event={"ID":"c15a91fd-755a-457c-aa8a-ac6b811025e4","Type":"ContainerDied","Data":"2152ff9559deaa39222a6d7f66e02650325de9d354e8fbb2810463433bd696a7"}
Sep 11 21:06:09.921229 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:09.921187    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:06:09.921229 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:09.921219    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:06:09.921229 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:09.921231    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:06:09.921669 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:09.921331    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4gct2" podUID="03071acd-860e-4da8-b813-079c6af540b6"
Sep 11 21:06:09.921669 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:09.921450    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjtsr" podUID="c94aef67-aa61-440f-9e7a-a8fc58611ca8"
Sep 11 21:06:09.921669 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:09.921551    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbqxp" podUID="a06b3567-7898-44a3-b981-5d5180c6130e"
Sep 11 21:06:10.115251 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:10.115214    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sptwp" event={"ID":"c15a91fd-755a-457c-aa8a-ac6b811025e4","Type":"ContainerStarted","Data":"5ca533187131184ab4908df96c31380ce58e2546c58684fa9492110c6ce3a244"}
Sep 11 21:06:10.142782 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:10.142730    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sptwp" podStartSLOduration=11.053581478 podStartE2EDuration="36.142717635s" totalImagesPullingTime="25.089136157s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:05:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:06:10.142545509 +0000 UTC m=+36.846514797" watchObservedRunningTime="2026-09-11 21:06:10.142717635 +0000 UTC m=+36.846686923"
Sep 11 21:06:11.921393 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:11.921352    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:06:11.921880 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:11.921370    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:06:11.921880 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:11.921459    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbqxp" podUID="a06b3567-7898-44a3-b981-5d5180c6130e"
Sep 11 21:06:11.921880 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:11.921498    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:06:11.921880 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:11.921624    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4gct2" podUID="03071acd-860e-4da8-b813-079c6af540b6"
Sep 11 21:06:11.921880 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:11.921717    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjtsr" podUID="c94aef67-aa61-440f-9e7a-a8fc58611ca8"
Sep 11 21:06:12.155418 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.155348    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-224.ec2.internal" event="NodeReady"
Sep 11 21:06:12.155538 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.155468    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 21:06:12.217015 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.216984    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hgf4n"]
Sep 11 21:06:12.232621 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.232594    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lhdrr"]
Sep 11 21:06:12.232886 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.232863    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hgf4n"
Sep 11 21:06:12.235025 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.235002    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 21:06:12.235176 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.235157    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f5khd\""
Sep 11 21:06:12.235176 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.235170    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 21:06:12.246264 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.246245    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hgf4n"]
Sep 11 21:06:12.246344 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.246276    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lhdrr"]
Sep 11 21:06:12.246393 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.246372    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lhdrr"
Sep 11 21:06:12.248744 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.248724    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 21:06:12.248864 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.248756    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 21:06:12.248924 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.248815    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 21:06:12.248989 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.248969    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rvmq7\""
Sep 11 21:06:12.340256 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.340224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv9l5\" (UniqueName: \"kubernetes.io/projected/a4135cd6-e9bc-45f9-978d-eead761ac9df-kube-api-access-mv9l5\") pod \"dns-default-hgf4n\" (UID: \"a4135cd6-e9bc-45f9-978d-eead761ac9df\") " pod="openshift-dns/dns-default-hgf4n"
Sep 11 21:06:12.340256 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.340259    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4135cd6-e9bc-45f9-978d-eead761ac9df-metrics-tls\") pod \"dns-default-hgf4n\" (UID: \"a4135cd6-e9bc-45f9-978d-eead761ac9df\") " pod="openshift-dns/dns-default-hgf4n"
Sep 11 21:06:12.340476 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.340277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4135cd6-e9bc-45f9-978d-eead761ac9df-tmp-dir\") pod \"dns-default-hgf4n\" (UID: \"a4135cd6-e9bc-45f9-978d-eead761ac9df\") " pod="openshift-dns/dns-default-hgf4n"
Sep 11 21:06:12.340476 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.340302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2af5c92-e222-4927-8013-0764ee9a4e8d-cert\") pod \"ingress-canary-lhdrr\" (UID: \"d2af5c92-e222-4927-8013-0764ee9a4e8d\") " pod="openshift-ingress-canary/ingress-canary-lhdrr"
Sep 11 21:06:12.340476 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.340362    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a4135cd6-e9bc-45f9-978d-eead761ac9df-config-volume\") pod \"dns-default-hgf4n\" (UID: \"a4135cd6-e9bc-45f9-978d-eead761ac9df\") " pod="openshift-dns/dns-default-hgf4n"
Sep 11 21:06:12.340476 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.340460    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8gw2\" (UniqueName: \"kubernetes.io/projected/d2af5c92-e222-4927-8013-0764ee9a4e8d-kube-api-access-r8gw2\") pod \"ingress-canary-lhdrr\" (UID: \"d2af5c92-e222-4927-8013-0764ee9a4e8d\") " pod="openshift-ingress-canary/ingress-canary-lhdrr"
Sep 11 21:06:12.441658 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.441623    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv9l5\" (UniqueName: \"kubernetes.io/projected/a4135cd6-e9bc-45f9-978d-eead761ac9df-kube-api-access-mv9l5\") pod \"dns-default-hgf4n\" (UID: \"a4135cd6-e9bc-45f9-978d-eead761ac9df\") " pod="openshift-dns/dns-default-hgf4n"
Sep 11 21:06:12.441801 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.441676    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4135cd6-e9bc-45f9-978d-eead761ac9df-metrics-tls\") pod \"dns-default-hgf4n\" (UID: \"a4135cd6-e9bc-45f9-978d-eead761ac9df\") " pod="openshift-dns/dns-default-hgf4n"
Sep 11 21:06:12.441801 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.441703    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4135cd6-e9bc-45f9-978d-eead761ac9df-tmp-dir\") pod \"dns-default-hgf4n\" (UID: \"a4135cd6-e9bc-45f9-978d-eead761ac9df\") " pod="openshift-dns/dns-default-hgf4n"
Sep 11 21:06:12.441801 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.441725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2af5c92-e222-4927-8013-0764ee9a4e8d-cert\") pod \"ingress-canary-lhdrr\" (UID: \"d2af5c92-e222-4927-8013-0764ee9a4e8d\") " pod="openshift-ingress-canary/ingress-canary-lhdrr"
Sep 11 21:06:12.441801 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.441749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a4135cd6-e9bc-45f9-978d-eead761ac9df-config-volume\") pod \"dns-default-hgf4n\" (UID: \"a4135cd6-e9bc-45f9-978d-eead761ac9df\") " pod="openshift-dns/dns-default-hgf4n"
Sep 11 21:06:12.441801 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.441787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8gw2\" (UniqueName: \"kubernetes.io/projected/d2af5c92-e222-4927-8013-0764ee9a4e8d-kube-api-access-r8gw2\") pod \"ingress-canary-lhdrr\" (UID: \"d2af5c92-e222-4927-8013-0764ee9a4e8d\") " pod="openshift-ingress-canary/ingress-canary-lhdrr"
Sep 11 21:06:12.442117 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:12.441914    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:06:12.442117 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:12.441984    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4135cd6-e9bc-45f9-978d-eead761ac9df-metrics-tls podName:a4135cd6-e9bc-45f9-978d-eead761ac9df nodeName:}" failed. No retries permitted until 2026-09-11 21:06:12.941962047 +0000 UTC m=+39.645931328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4135cd6-e9bc-45f9-978d-eead761ac9df-metrics-tls") pod "dns-default-hgf4n" (UID: "a4135cd6-e9bc-45f9-978d-eead761ac9df") : secret "dns-default-metrics-tls" not found
Sep 11 21:06:12.442117 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.442106    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4135cd6-e9bc-45f9-978d-eead761ac9df-tmp-dir\") pod \"dns-default-hgf4n\" (UID: \"a4135cd6-e9bc-45f9-978d-eead761ac9df\") " pod="openshift-dns/dns-default-hgf4n"
Sep 11 21:06:12.442423 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.442406    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a4135cd6-e9bc-45f9-978d-eead761ac9df-config-volume\") pod \"dns-default-hgf4n\" (UID: \"a4135cd6-e9bc-45f9-978d-eead761ac9df\") " pod="openshift-dns/dns-default-hgf4n"
Sep 11 21:06:12.445876 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.445853    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2af5c92-e222-4927-8013-0764ee9a4e8d-cert\") pod \"ingress-canary-lhdrr\" (UID: \"d2af5c92-e222-4927-8013-0764ee9a4e8d\") " pod="openshift-ingress-canary/ingress-canary-lhdrr"
Sep 11 21:06:12.451143 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.451123    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8gw2\" (UniqueName: \"kubernetes.io/projected/d2af5c92-e222-4927-8013-0764ee9a4e8d-kube-api-access-r8gw2\") pod \"ingress-canary-lhdrr\" (UID: \"d2af5c92-e222-4927-8013-0764ee9a4e8d\") " pod="openshift-ingress-canary/ingress-canary-lhdrr"
Sep 11 21:06:12.459428 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.459396    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv9l5\" (UniqueName: \"kubernetes.io/projected/a4135cd6-e9bc-45f9-978d-eead761ac9df-kube-api-access-mv9l5\") pod \"dns-default-hgf4n\" (UID: \"a4135cd6-e9bc-45f9-978d-eead761ac9df\") " pod="openshift-dns/dns-default-hgf4n"
Sep 11 21:06:12.554680 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.554643    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lhdrr"
Sep 11 21:06:12.872591 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.872347    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lhdrr"]
Sep 11 21:06:12.876676 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:06:12.876648    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2af5c92_e222_4927_8013_0764ee9a4e8d.slice/crio-7df25888c40f81b61eb166f78552ec490691a451b98c816d8df90eea1e7070d6 WatchSource:0}: Error finding container 7df25888c40f81b61eb166f78552ec490691a451b98c816d8df90eea1e7070d6: Status 404 returned error can't find the container with id 7df25888c40f81b61eb166f78552ec490691a451b98c816d8df90eea1e7070d6
Sep 11 21:06:12.946288 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:12.946246    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4135cd6-e9bc-45f9-978d-eead761ac9df-metrics-tls\") pod \"dns-default-hgf4n\" (UID: \"a4135cd6-e9bc-45f9-978d-eead761ac9df\") " pod="openshift-dns/dns-default-hgf4n"
Sep 11 21:06:12.946743 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:12.946393    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:06:12.946743 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:12.946465    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4135cd6-e9bc-45f9-978d-eead761ac9df-metrics-tls podName:a4135cd6-e9bc-45f9-978d-eead761ac9df nodeName:}" failed. No retries permitted until 2026-09-11 21:06:13.946443999 +0000 UTC m=+40.650413285 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4135cd6-e9bc-45f9-978d-eead761ac9df-metrics-tls") pod "dns-default-hgf4n" (UID: "a4135cd6-e9bc-45f9-978d-eead761ac9df") : secret "dns-default-metrics-tls" not found
Sep 11 21:06:13.122911 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:13.122797    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lhdrr" event={"ID":"d2af5c92-e222-4927-8013-0764ee9a4e8d","Type":"ContainerStarted","Data":"7df25888c40f81b61eb166f78552ec490691a451b98c816d8df90eea1e7070d6"}
Sep 11 21:06:13.922421 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:13.922388    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:06:13.922641 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:13.922464    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:06:13.922641 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:13.922504    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:06:13.938944 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:13.938916    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 21:06:13.939129 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:13.939112    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mpnkz\""
Sep 11 21:06:13.940333 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:13.940299    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 21:06:13.941048 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:13.941032    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 21:06:13.941151 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:13.941101    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 21:06:13.942046 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:13.941961    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mwzls\""
Sep 11 21:06:13.954324 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:13.954297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4135cd6-e9bc-45f9-978d-eead761ac9df-metrics-tls\") pod \"dns-default-hgf4n\" (UID: \"a4135cd6-e9bc-45f9-978d-eead761ac9df\") " pod="openshift-dns/dns-default-hgf4n"
Sep 11 21:06:13.954809 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:13.954463    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:06:13.954809 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:13.954531    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4135cd6-e9bc-45f9-978d-eead761ac9df-metrics-tls podName:a4135cd6-e9bc-45f9-978d-eead761ac9df nodeName:}" failed. No retries permitted until 2026-09-11 21:06:15.954516655 +0000 UTC m=+42.658485925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4135cd6-e9bc-45f9-978d-eead761ac9df-metrics-tls") pod "dns-default-hgf4n" (UID: "a4135cd6-e9bc-45f9-978d-eead761ac9df") : secret "dns-default-metrics-tls" not found
Sep 11 21:06:15.969676 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:15.969635    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4135cd6-e9bc-45f9-978d-eead761ac9df-metrics-tls\") pod \"dns-default-hgf4n\" (UID: \"a4135cd6-e9bc-45f9-978d-eead761ac9df\") " pod="openshift-dns/dns-default-hgf4n"
Sep 11 21:06:15.970312 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:15.969787    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:06:15.970312 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:15.969872    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4135cd6-e9bc-45f9-978d-eead761ac9df-metrics-tls podName:a4135cd6-e9bc-45f9-978d-eead761ac9df nodeName:}" failed. No retries permitted until 2026-09-11 21:06:19.969853497 +0000 UTC m=+46.673822764 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4135cd6-e9bc-45f9-978d-eead761ac9df-metrics-tls") pod "dns-default-hgf4n" (UID: "a4135cd6-e9bc-45f9-978d-eead761ac9df") : secret "dns-default-metrics-tls" not found
Sep 11 21:06:16.130668 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:16.130635    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lhdrr" event={"ID":"d2af5c92-e222-4927-8013-0764ee9a4e8d","Type":"ContainerStarted","Data":"e0feb4d4f0830c3fa51c64799fad760e8ab61ff636e4300c421e77d423d217c4"}
Sep 11 21:06:19.997919 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:19.997880    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4135cd6-e9bc-45f9-978d-eead761ac9df-metrics-tls\") pod \"dns-default-hgf4n\" (UID: \"a4135cd6-e9bc-45f9-978d-eead761ac9df\") " pod="openshift-dns/dns-default-hgf4n"
Sep 11 21:06:19.998391 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:19.998028    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:06:19.998391 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:19.998103    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4135cd6-e9bc-45f9-978d-eead761ac9df-metrics-tls podName:a4135cd6-e9bc-45f9-978d-eead761ac9df nodeName:}" failed. No retries permitted until 2026-09-11 21:06:27.998081702 +0000 UTC m=+54.702050969 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4135cd6-e9bc-45f9-978d-eead761ac9df-metrics-tls") pod "dns-default-hgf4n" (UID: "a4135cd6-e9bc-45f9-978d-eead761ac9df") : secret "dns-default-metrics-tls" not found
Sep 11 21:06:20.600497 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:20.600436    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lhdrr" podStartSLOduration=5.933978326 podStartE2EDuration="8.600418826s" totalImagesPullingTime="2.6664405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:06:16.149235479 +0000 UTC m=+42.853204770" watchObservedRunningTime="2026-09-11 21:06:20.600418826 +0000 UTC m=+47.304388118"
Sep 11 21:06:20.600673 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:20.600650    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-mggdp"]
Sep 11 21:06:20.604937 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:20.604917    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mggdp"
Sep 11 21:06:20.607458 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:20.607431    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 11 21:06:20.607458 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:20.607429    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 11 21:06:20.608103 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:20.608083    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2z2lf\""
Sep 11 21:06:20.611749 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:20.611544    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-mggdp"]
Sep 11 21:06:20.702426 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:20.702391    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqq7f\" (UniqueName: \"kubernetes.io/projected/6f098d43-3053-4e83-ae6a-7c4bd9a09a97-kube-api-access-hqq7f\") pod \"migrator-8b4d697c5-mggdp\" (UID: \"6f098d43-3053-4e83-ae6a-7c4bd9a09a97\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mggdp"
Sep 11 21:06:20.803806 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:20.803780    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqq7f\" (UniqueName: \"kubernetes.io/projected/6f098d43-3053-4e83-ae6a-7c4bd9a09a97-kube-api-access-hqq7f\") pod \"migrator-8b4d697c5-mggdp\" (UID: \"6f098d43-3053-4e83-ae6a-7c4bd9a09a97\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mggdp"
Sep 11 21:06:20.813034 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:20.813011    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqq7f\" (UniqueName: \"kubernetes.io/projected/6f098d43-3053-4e83-ae6a-7c4bd9a09a97-kube-api-access-hqq7f\") pod \"migrator-8b4d697c5-mggdp\" (UID: \"6f098d43-3053-4e83-ae6a-7c4bd9a09a97\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mggdp"
Sep 11 21:06:20.914744 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:20.914651    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mggdp"
Sep 11 21:06:21.093023 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:21.092834    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-c28km"]
Sep 11 21:06:21.098621 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:21.098598    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:21.101134 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:21.101109    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 21:06:21.101313 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:21.101289    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 21:06:21.101393 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:21.101372    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gq824\""
Sep 11 21:06:21.101728 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:21.101713    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 21:06:21.101777 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:21.101756    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 21:06:21.109454 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:21.109433    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c28km"]
Sep 11 21:06:21.186990 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:21.186963    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-mggdp"]
Sep 11 21:06:21.190289 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:06:21.190264    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f098d43_3053_4e83_ae6a_7c4bd9a09a97.slice/crio-cf6185a69c762021a8284b8710043c16eb04b393e8ea93d7285ff90fa9c88a66 WatchSource:0}: Error finding container cf6185a69c762021a8284b8710043c16eb04b393e8ea93d7285ff90fa9c88a66: Status 404 returned error can't find the container with id cf6185a69c762021a8284b8710043c16eb04b393e8ea93d7285ff90fa9c88a66
Sep 11 21:06:21.206796 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:21.206763    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5cb0d0f-a841-4121-b7bf-d6d9628faded-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:21.206929 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:21.206800    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5cb0d0f-a841-4121-b7bf-d6d9628faded-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:21.206929 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:21.206846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5cb0d0f-a841-4121-b7bf-d6d9628faded-data-volume\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:21.206992 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:21.206952    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spkvw\" (UniqueName: \"kubernetes.io/projected/e5cb0d0f-a841-4121-b7bf-d6d9628faded-kube-api-access-spkvw\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:21.207024 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:21.206995    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e5cb0d0f-a841-4121-b7bf-d6d9628faded-crio-socket\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:21.307321 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:21.307283    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e5cb0d0f-a841-4121-b7bf-d6d9628faded-crio-socket\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:21.307483 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:21.307331    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5cb0d0f-a841-4121-b7bf-d6d9628faded-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:21.307483 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:21.307359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5cb0d0f-a841-4121-b7bf-d6d9628faded-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:21.307483 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:21.307399    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5cb0d0f-a841-4121-b7bf-d6d9628faded-data-volume\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:21.307584 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:21.307486    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:06:21.307584 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:21.307517    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e5cb0d0f-a841-4121-b7bf-d6d9628faded-crio-socket\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:21.307584 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:21.307525    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:06:21.307584 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:21.307565    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5cb0d0f-a841-4121-b7bf-d6d9628faded-kube-rbac-proxy-cm podName:e5cb0d0f-a841-4121-b7bf-d6d9628faded nodeName:}" failed. No retries permitted until 2026-09-11 21:06:21.807549569 +0000 UTC m=+48.511518837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e5cb0d0f-a841-4121-b7bf-d6d9628faded-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c28km" (UID: "e5cb0d0f-a841-4121-b7bf-d6d9628faded") : configmap "kube-rbac-proxy" not found
Sep 11 21:06:21.307757 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:21.307592    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5cb0d0f-a841-4121-b7bf-d6d9628faded-insights-runtime-extractor-tls podName:e5cb0d0f-a841-4121-b7bf-d6d9628faded nodeName:}" failed. No retries permitted until 2026-09-11 21:06:21.807586028 +0000 UTC m=+48.511555295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e5cb0d0f-a841-4121-b7bf-d6d9628faded-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c28km" (UID: "e5cb0d0f-a841-4121-b7bf-d6d9628faded") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:06:21.307757 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:21.307606    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spkvw\" (UniqueName: \"kubernetes.io/projected/e5cb0d0f-a841-4121-b7bf-d6d9628faded-kube-api-access-spkvw\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:21.307757 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:21.307703    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e5cb0d0f-a841-4121-b7bf-d6d9628faded-data-volume\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:21.316325 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:21.316292    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spkvw\" (UniqueName: \"kubernetes.io/projected/e5cb0d0f-a841-4121-b7bf-d6d9628faded-kube-api-access-spkvw\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:21.811270 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:21.811225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5cb0d0f-a841-4121-b7bf-d6d9628faded-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:21.811270 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:21.811279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5cb0d0f-a841-4121-b7bf-d6d9628faded-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:21.811468 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:21.811350    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:06:21.811468 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:21.811425    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5cb0d0f-a841-4121-b7bf-d6d9628faded-kube-rbac-proxy-cm podName:e5cb0d0f-a841-4121-b7bf-d6d9628faded nodeName:}" failed. No retries permitted until 2026-09-11 21:06:22.811410557 +0000 UTC m=+49.515379828 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e5cb0d0f-a841-4121-b7bf-d6d9628faded-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c28km" (UID: "e5cb0d0f-a841-4121-b7bf-d6d9628faded") : configmap "kube-rbac-proxy" not found
Sep 11 21:06:21.811468 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:21.811434    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:06:21.811579 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:21.811478    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5cb0d0f-a841-4121-b7bf-d6d9628faded-insights-runtime-extractor-tls podName:e5cb0d0f-a841-4121-b7bf-d6d9628faded nodeName:}" failed. No retries permitted until 2026-09-11 21:06:22.811462461 +0000 UTC m=+49.515431742 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e5cb0d0f-a841-4121-b7bf-d6d9628faded-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c28km" (UID: "e5cb0d0f-a841-4121-b7bf-d6d9628faded") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:06:22.146456 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:22.146367    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mggdp" event={"ID":"6f098d43-3053-4e83-ae6a-7c4bd9a09a97","Type":"ContainerStarted","Data":"cf6185a69c762021a8284b8710043c16eb04b393e8ea93d7285ff90fa9c88a66"}
Sep 11 21:06:22.819296 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:22.819197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5cb0d0f-a841-4121-b7bf-d6d9628faded-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:22.819296 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:22.819240    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5cb0d0f-a841-4121-b7bf-d6d9628faded-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:22.819515 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:22.819326    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:06:22.819515 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:22.819346    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:06:22.819515 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:22.819389    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5cb0d0f-a841-4121-b7bf-d6d9628faded-insights-runtime-extractor-tls podName:e5cb0d0f-a841-4121-b7bf-d6d9628faded nodeName:}" failed. No retries permitted until 2026-09-11 21:06:24.819375898 +0000 UTC m=+51.523345187 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e5cb0d0f-a841-4121-b7bf-d6d9628faded-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c28km" (UID: "e5cb0d0f-a841-4121-b7bf-d6d9628faded") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:06:22.819515 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:22.819405    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5cb0d0f-a841-4121-b7bf-d6d9628faded-kube-rbac-proxy-cm podName:e5cb0d0f-a841-4121-b7bf-d6d9628faded nodeName:}" failed. No retries permitted until 2026-09-11 21:06:24.819397492 +0000 UTC m=+51.523366759 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e5cb0d0f-a841-4121-b7bf-d6d9628faded-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c28km" (UID: "e5cb0d0f-a841-4121-b7bf-d6d9628faded") : configmap "kube-rbac-proxy" not found
Sep 11 21:06:23.150303 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:23.150224    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mggdp" event={"ID":"6f098d43-3053-4e83-ae6a-7c4bd9a09a97","Type":"ContainerStarted","Data":"29226e5badae9a0a0d380d3a28d5db91484259a07ec4cba0ec055bf7568255f8"}
Sep 11 21:06:23.150303 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:23.150274    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mggdp" event={"ID":"6f098d43-3053-4e83-ae6a-7c4bd9a09a97","Type":"ContainerStarted","Data":"9d3abc43157703284b78bcf90f11a10a083df808f91f4360cd1843b99302a9b0"}
Sep 11 21:06:23.173904 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:23.173858    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mggdp" podStartSLOduration=1.855421957 podStartE2EDuration="3.173842893s" totalImagesPullingTime="1.318420936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:06:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:06:23.172575862 +0000 UTC m=+49.876545152" watchObservedRunningTime="2026-09-11 21:06:23.173842893 +0000 UTC m=+49.877812173"
Sep 11 21:06:23.254925 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:23.254899    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-chmvt_7c9e9fa2-dc0c-4832-9841-fe2468faea5a/dns-node-resolver/0.log"
Sep 11 21:06:23.787197 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:23.787165    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-wpwdh"]
Sep 11 21:06:23.790084 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:23.790063    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-wpwdh"
Sep 11 21:06:23.797043 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:23.797022    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 11 21:06:23.797161 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:23.797101    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 11 21:06:23.797161 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:23.797132    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 11 21:06:23.797433 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:23.797415    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 11 21:06:23.797811 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:23.797795    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ntq68\""
Sep 11 21:06:23.808329 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:23.808302    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-wpwdh"]
Sep 11 21:06:23.927380 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:23.927352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/67aa0b9a-f707-48ba-979a-ec0e58c91461-signing-key\") pod \"service-ca-568869fdc5-wpwdh\" (UID: \"67aa0b9a-f707-48ba-979a-ec0e58c91461\") " pod="openshift-service-ca/service-ca-568869fdc5-wpwdh"
Sep 11 21:06:23.927570 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:23.927455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/67aa0b9a-f707-48ba-979a-ec0e58c91461-signing-cabundle\") pod \"service-ca-568869fdc5-wpwdh\" (UID: \"67aa0b9a-f707-48ba-979a-ec0e58c91461\") " pod="openshift-service-ca/service-ca-568869fdc5-wpwdh"
Sep 11 21:06:23.927570 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:23.927484    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffqm9\" (UniqueName: \"kubernetes.io/projected/67aa0b9a-f707-48ba-979a-ec0e58c91461-kube-api-access-ffqm9\") pod \"service-ca-568869fdc5-wpwdh\" (UID: \"67aa0b9a-f707-48ba-979a-ec0e58c91461\") " pod="openshift-service-ca/service-ca-568869fdc5-wpwdh"
Sep 11 21:06:24.028477 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:24.028437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/67aa0b9a-f707-48ba-979a-ec0e58c91461-signing-cabundle\") pod \"service-ca-568869fdc5-wpwdh\" (UID: \"67aa0b9a-f707-48ba-979a-ec0e58c91461\") " pod="openshift-service-ca/service-ca-568869fdc5-wpwdh"
Sep 11 21:06:24.028477 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:24.028479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffqm9\" (UniqueName: \"kubernetes.io/projected/67aa0b9a-f707-48ba-979a-ec0e58c91461-kube-api-access-ffqm9\") pod \"service-ca-568869fdc5-wpwdh\" (UID: \"67aa0b9a-f707-48ba-979a-ec0e58c91461\") " pod="openshift-service-ca/service-ca-568869fdc5-wpwdh"
Sep 11 21:06:24.028671 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:24.028651    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/67aa0b9a-f707-48ba-979a-ec0e58c91461-signing-key\") pod \"service-ca-568869fdc5-wpwdh\" (UID: \"67aa0b9a-f707-48ba-979a-ec0e58c91461\") " pod="openshift-service-ca/service-ca-568869fdc5-wpwdh"
Sep 11 21:06:24.029170 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:24.029142    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/67aa0b9a-f707-48ba-979a-ec0e58c91461-signing-cabundle\") pod \"service-ca-568869fdc5-wpwdh\" (UID: \"67aa0b9a-f707-48ba-979a-ec0e58c91461\") " pod="openshift-service-ca/service-ca-568869fdc5-wpwdh"
Sep 11 21:06:24.031217 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:24.031196    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/67aa0b9a-f707-48ba-979a-ec0e58c91461-signing-key\") pod \"service-ca-568869fdc5-wpwdh\" (UID: \"67aa0b9a-f707-48ba-979a-ec0e58c91461\") " pod="openshift-service-ca/service-ca-568869fdc5-wpwdh"
Sep 11 21:06:24.037080 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:24.037056    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffqm9\" (UniqueName: \"kubernetes.io/projected/67aa0b9a-f707-48ba-979a-ec0e58c91461-kube-api-access-ffqm9\") pod \"service-ca-568869fdc5-wpwdh\" (UID: \"67aa0b9a-f707-48ba-979a-ec0e58c91461\") " pod="openshift-service-ca/service-ca-568869fdc5-wpwdh"
Sep 11 21:06:24.054772 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:24.054710    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2dd4w_5c8ceec8-5ea9-41ef-864c-c6e65ea6f483/node-ca/0.log"
Sep 11 21:06:24.098380 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:24.098308    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-wpwdh"
Sep 11 21:06:24.377509 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:24.377406    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-wpwdh"]
Sep 11 21:06:24.380864 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:06:24.380812    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67aa0b9a_f707_48ba_979a_ec0e58c91461.slice/crio-2f1d1bcb9f22a52380e17c4ab9a13cc1f633d5c28321e530558b1082310ae9ef WatchSource:0}: Error finding container 2f1d1bcb9f22a52380e17c4ab9a13cc1f633d5c28321e530558b1082310ae9ef: Status 404 returned error can't find the container with id 2f1d1bcb9f22a52380e17c4ab9a13cc1f633d5c28321e530558b1082310ae9ef
Sep 11 21:06:24.834098 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:24.834051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5cb0d0f-a841-4121-b7bf-d6d9628faded-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:24.834270 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:24.834105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5cb0d0f-a841-4121-b7bf-d6d9628faded-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:24.834270 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:24.834183    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:06:24.834339 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:24.834282    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5cb0d0f-a841-4121-b7bf-d6d9628faded-kube-rbac-proxy-cm podName:e5cb0d0f-a841-4121-b7bf-d6d9628faded nodeName:}" failed. No retries permitted until 2026-09-11 21:06:28.834264319 +0000 UTC m=+55.538233587 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e5cb0d0f-a841-4121-b7bf-d6d9628faded-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c28km" (UID: "e5cb0d0f-a841-4121-b7bf-d6d9628faded") : configmap "kube-rbac-proxy" not found
Sep 11 21:06:24.834339 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:24.834284    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:06:24.834447 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:24.834339    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5cb0d0f-a841-4121-b7bf-d6d9628faded-insights-runtime-extractor-tls podName:e5cb0d0f-a841-4121-b7bf-d6d9628faded nodeName:}" failed. No retries permitted until 2026-09-11 21:06:28.834324673 +0000 UTC m=+55.538293940 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e5cb0d0f-a841-4121-b7bf-d6d9628faded-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c28km" (UID: "e5cb0d0f-a841-4121-b7bf-d6d9628faded") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:06:25.052921 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:25.052892    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lhdrr_d2af5c92-e222-4927-8013-0764ee9a4e8d/serve-healthcheck-canary/0.log"
Sep 11 21:06:25.157280 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:25.157202    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-wpwdh" event={"ID":"67aa0b9a-f707-48ba-979a-ec0e58c91461","Type":"ContainerStarted","Data":"2f1d1bcb9f22a52380e17c4ab9a13cc1f633d5c28321e530558b1082310ae9ef"}
Sep 11 21:06:25.253997 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:25.253964    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-mggdp_6f098d43-3053-4e83-ae6a-7c4bd9a09a97/migrator/0.log"
Sep 11 21:06:25.452186 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:25.452159    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-mggdp_6f098d43-3053-4e83-ae6a-7c4bd9a09a97/graceful-termination/0.log"
Sep 11 21:06:27.164137 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:27.164101    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-wpwdh" event={"ID":"67aa0b9a-f707-48ba-979a-ec0e58c91461","Type":"ContainerStarted","Data":"7cbeaec90e2463403e88094f1d992c357f1bc51c758a779a9d497ac4fbd49b35"}
Sep 11 21:06:27.182194 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:27.182139    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-wpwdh" podStartSLOduration=2.078276619 podStartE2EDuration="4.182121774s" totalImagesPullingTime="2.103845155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:06:27.180995678 +0000 UTC m=+53.884964971" watchObservedRunningTime="2026-09-11 21:06:27.182121774 +0000 UTC m=+53.886091063"
Sep 11 21:06:28.061137 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:28.061100    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4135cd6-e9bc-45f9-978d-eead761ac9df-metrics-tls\") pod \"dns-default-hgf4n\" (UID: \"a4135cd6-e9bc-45f9-978d-eead761ac9df\") " pod="openshift-dns/dns-default-hgf4n"
Sep 11 21:06:28.061344 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:28.061236    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:06:28.061344 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:28.061298    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4135cd6-e9bc-45f9-978d-eead761ac9df-metrics-tls podName:a4135cd6-e9bc-45f9-978d-eead761ac9df nodeName:}" failed. No retries permitted until 2026-09-11 21:06:44.061279047 +0000 UTC m=+70.765248319 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4135cd6-e9bc-45f9-978d-eead761ac9df-metrics-tls") pod "dns-default-hgf4n" (UID: "a4135cd6-e9bc-45f9-978d-eead761ac9df") : secret "dns-default-metrics-tls" not found
Sep 11 21:06:28.867658 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:28.867615    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5cb0d0f-a841-4121-b7bf-d6d9628faded-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:28.867658 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:28.867661    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5cb0d0f-a841-4121-b7bf-d6d9628faded-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:28.868101 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:28.867753    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:06:28.868101 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:28.867774    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:06:28.868101 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:28.867839    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5cb0d0f-a841-4121-b7bf-d6d9628faded-kube-rbac-proxy-cm podName:e5cb0d0f-a841-4121-b7bf-d6d9628faded nodeName:}" failed. No retries permitted until 2026-09-11 21:06:36.867803934 +0000 UTC m=+63.571773215 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e5cb0d0f-a841-4121-b7bf-d6d9628faded-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c28km" (UID: "e5cb0d0f-a841-4121-b7bf-d6d9628faded") : configmap "kube-rbac-proxy" not found
Sep 11 21:06:28.868101 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:28.867851    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5cb0d0f-a841-4121-b7bf-d6d9628faded-insights-runtime-extractor-tls podName:e5cb0d0f-a841-4121-b7bf-d6d9628faded nodeName:}" failed. No retries permitted until 2026-09-11 21:06:36.867845848 +0000 UTC m=+63.571815115 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e5cb0d0f-a841-4121-b7bf-d6d9628faded-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c28km" (UID: "e5cb0d0f-a841-4121-b7bf-d6d9628faded") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:06:36.107666 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:36.107637    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hk624"
Sep 11 21:06:36.924063 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:36.924026    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5cb0d0f-a841-4121-b7bf-d6d9628faded-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:36.924063 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:36.924063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5cb0d0f-a841-4121-b7bf-d6d9628faded-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:36.924257 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:36.924152    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:06:36.924257 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:36.924200    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5cb0d0f-a841-4121-b7bf-d6d9628faded-insights-runtime-extractor-tls podName:e5cb0d0f-a841-4121-b7bf-d6d9628faded nodeName:}" failed. No retries permitted until 2026-09-11 21:06:52.924188068 +0000 UTC m=+79.628157335 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e5cb0d0f-a841-4121-b7bf-d6d9628faded-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c28km" (UID: "e5cb0d0f-a841-4121-b7bf-d6d9628faded") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:06:36.925189 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:36.925166    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e5cb0d0f-a841-4121-b7bf-d6d9628faded-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:38.636123 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:38.636085    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a06b3567-7898-44a3-b981-5d5180c6130e-original-pull-secret\") pod \"global-pull-secret-syncer-mbqxp\" (UID: \"a06b3567-7898-44a3-b981-5d5180c6130e\") " pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:06:38.636478 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:38.636131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03071acd-860e-4da8-b813-079c6af540b6-metrics-certs\") pod \"network-metrics-daemon-4gct2\" (UID: \"03071acd-860e-4da8-b813-079c6af540b6\") " pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:06:38.638743 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:38.638725    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 21:06:38.639188 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:38.639169    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 21:06:38.649242 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:38.649214    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a06b3567-7898-44a3-b981-5d5180c6130e-original-pull-secret\") pod \"global-pull-secret-syncer-mbqxp\" (UID: \"a06b3567-7898-44a3-b981-5d5180c6130e\") " pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:06:38.649376 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:38.649247    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03071acd-860e-4da8-b813-079c6af540b6-metrics-certs\") pod \"network-metrics-daemon-4gct2\" (UID: \"03071acd-860e-4da8-b813-079c6af540b6\") " pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:06:38.736580 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:38.736554    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44l7n\" (UniqueName: \"kubernetes.io/projected/c94aef67-aa61-440f-9e7a-a8fc58611ca8-kube-api-access-44l7n\") pod \"network-check-target-cjtsr\" (UID: \"c94aef67-aa61-440f-9e7a-a8fc58611ca8\") " pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:06:38.739224 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:38.739207    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 21:06:38.749603 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:38.749567    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 21:06:38.759930 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:38.759914    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44l7n\" (UniqueName: \"kubernetes.io/projected/c94aef67-aa61-440f-9e7a-a8fc58611ca8-kube-api-access-44l7n\") pod \"network-check-target-cjtsr\" (UID: \"c94aef67-aa61-440f-9e7a-a8fc58611ca8\") " pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:06:38.834786 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:38.834768    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbqxp"
Sep 11 21:06:38.845838 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:38.845809    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mpnkz\""
Sep 11 21:06:38.851212 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:38.851193    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mwzls\""
Sep 11 21:06:38.853208 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:38.853193    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4gct2"
Sep 11 21:06:38.859657 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:38.859639    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:06:39.108507 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:39.108481    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mbqxp"]
Sep 11 21:06:39.139627 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:39.139569    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4gct2"]
Sep 11 21:06:39.142311 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:06:39.142287    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03071acd_860e_4da8_b813_079c6af540b6.slice/crio-46ed9afcdc60904ccc2707f2d5a27fab3e629ac1ae8df063d4537be5a27ff105 WatchSource:0}: Error finding container 46ed9afcdc60904ccc2707f2d5a27fab3e629ac1ae8df063d4537be5a27ff105: Status 404 returned error can't find the container with id 46ed9afcdc60904ccc2707f2d5a27fab3e629ac1ae8df063d4537be5a27ff105
Sep 11 21:06:39.144594 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:39.144571    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cjtsr"]
Sep 11 21:06:39.148842 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:06:39.148806    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94aef67_aa61_440f_9e7a_a8fc58611ca8.slice/crio-969335c03e0b89e25bc021da70d52e902f5c1dc217ba2e748451ab5d844dfba5 WatchSource:0}: Error finding container 969335c03e0b89e25bc021da70d52e902f5c1dc217ba2e748451ab5d844dfba5: Status 404 returned error can't find the container with id 969335c03e0b89e25bc021da70d52e902f5c1dc217ba2e748451ab5d844dfba5
Sep 11 21:06:39.195060 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:39.195038    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4gct2" event={"ID":"03071acd-860e-4da8-b813-079c6af540b6","Type":"ContainerStarted","Data":"46ed9afcdc60904ccc2707f2d5a27fab3e629ac1ae8df063d4537be5a27ff105"}
Sep 11 21:06:39.196195 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:39.196173    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cjtsr" event={"ID":"c94aef67-aa61-440f-9e7a-a8fc58611ca8","Type":"ContainerStarted","Data":"969335c03e0b89e25bc021da70d52e902f5c1dc217ba2e748451ab5d844dfba5"}
Sep 11 21:06:39.197235 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:39.197215    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mbqxp" event={"ID":"a06b3567-7898-44a3-b981-5d5180c6130e","Type":"ContainerStarted","Data":"96a1de303c22c3d9a1e9eee10e39f79ed7993d32ebf3797e9df5bb623ba8fbf5"}
Sep 11 21:06:41.208380 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:41.208321    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4gct2" event={"ID":"03071acd-860e-4da8-b813-079c6af540b6","Type":"ContainerStarted","Data":"b9fe128f27267bf8fabb75a37e309e043a2d91886775872d1f79cfc9d29ed6be"}
Sep 11 21:06:41.208874 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:41.208408    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4gct2" event={"ID":"03071acd-860e-4da8-b813-079c6af540b6","Type":"ContainerStarted","Data":"1e42c2d237eab68a21add7b89825edb1be7dcc88bef98ff6d3e923845f65f3c1"}
Sep 11 21:06:41.227073 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:41.227004    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4gct2" podStartSLOduration=65.652335745 podStartE2EDuration="1m7.226985154s" totalImagesPullingTime="1.574649409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:06:41.226228843 +0000 UTC m=+67.930198133" watchObservedRunningTime="2026-09-11 21:06:41.226985154 +0000 UTC m=+67.930954442"
Sep 11 21:06:44.075581 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:44.075541    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4135cd6-e9bc-45f9-978d-eead761ac9df-metrics-tls\") pod \"dns-default-hgf4n\" (UID: \"a4135cd6-e9bc-45f9-978d-eead761ac9df\") " pod="openshift-dns/dns-default-hgf4n"
Sep 11 21:06:44.077716 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:44.077699    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4135cd6-e9bc-45f9-978d-eead761ac9df-metrics-tls\") pod \"dns-default-hgf4n\" (UID: \"a4135cd6-e9bc-45f9-978d-eead761ac9df\") " pod="openshift-dns/dns-default-hgf4n"
Sep 11 21:06:44.221467 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:44.221437    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cjtsr" event={"ID":"c94aef67-aa61-440f-9e7a-a8fc58611ca8","Type":"ContainerStarted","Data":"89ef524ebe2776dee17d28de199e1b7ca250492e9764b676a145b47c259899ad"}
Sep 11 21:06:44.221635 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:44.221508    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:06:44.222994 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:44.222960    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mbqxp" event={"ID":"a06b3567-7898-44a3-b981-5d5180c6130e","Type":"ContainerStarted","Data":"8a070e9bd55bfba7b90f919362da642b88cdd5344ba935288c3e8724c081c9cc"}
Sep 11 21:06:44.238909 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:44.238866    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cjtsr" podStartSLOduration=65.749458711 podStartE2EDuration="1m10.238853032s" totalImagesPullingTime="4.489394321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:06:44.238073485 +0000 UTC m=+70.942042777" watchObservedRunningTime="2026-09-11 21:06:44.238853032 +0000 UTC m=+70.942822321"
Sep 11 21:06:44.254082 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:44.254041    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mbqxp" podStartSLOduration=65.722376262 podStartE2EDuration="1m10.254030231s" totalImagesPullingTime="4.531653969s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:06:44.25298396 +0000 UTC m=+70.956953250" watchObservedRunningTime="2026-09-11 21:06:44.254030231 +0000 UTC m=+70.957999519"
Sep 11 21:06:44.344029 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:44.343975    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f5khd\""
Sep 11 21:06:44.352240 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:44.352225    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hgf4n"
Sep 11 21:06:44.624195 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:44.624130    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hgf4n"]
Sep 11 21:06:44.627922 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:06:44.627897    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4135cd6_e9bc_45f9_978d_eead761ac9df.slice/crio-8e64484b010373d853463f669efd7a9c4086cf7dd48a878e27c9eb6032d597a1 WatchSource:0}: Error finding container 8e64484b010373d853463f669efd7a9c4086cf7dd48a878e27c9eb6032d597a1: Status 404 returned error can't find the container with id 8e64484b010373d853463f669efd7a9c4086cf7dd48a878e27c9eb6032d597a1
Sep 11 21:06:45.227803 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:45.227768    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hgf4n" event={"ID":"a4135cd6-e9bc-45f9-978d-eead761ac9df","Type":"ContainerStarted","Data":"8e64484b010373d853463f669efd7a9c4086cf7dd48a878e27c9eb6032d597a1"}
Sep 11 21:06:46.233775 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:46.233738    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hgf4n" event={"ID":"a4135cd6-e9bc-45f9-978d-eead761ac9df","Type":"ContainerStarted","Data":"0eb423c875ccb01b43dc6cd8878781a0e66429afb687c5bf8d4d5036af49f799"}
Sep 11 21:06:47.238810 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:47.238774    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hgf4n" event={"ID":"a4135cd6-e9bc-45f9-978d-eead761ac9df","Type":"ContainerStarted","Data":"e71693276e73e949f665d90816d1ec73633be1b45bd12ceae475846bd249b56e"}
Sep 11 21:06:47.239190 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:47.238888    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hgf4n"
Sep 11 21:06:47.259324 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:47.259267    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hgf4n" podStartSLOduration=33.795018552 podStartE2EDuration="35.259129657s" totalImagesPullingTime="1.464111105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:06:47.258681455 +0000 UTC m=+73.962650744" watchObservedRunningTime="2026-09-11 21:06:47.259129657 +0000 UTC m=+73.963098948"
Sep 11 21:06:49.483916 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.483885    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-868576d4cb-jfsgj"]
Sep 11 21:06:49.489013 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.488990    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-868576d4cb-jfsgj"
Sep 11 21:06:49.491832 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.491792    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 21:06:49.492210 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.492186    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 21:06:49.492883 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.492864    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 21:06:49.492947 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.492937    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 11 21:06:49.503876 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.503854    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj"]
Sep 11 21:06:49.507076 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.507062    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj"
Sep 11 21:06:49.509469 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.509449    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 11 21:06:49.509595 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.509577    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 11 21:06:49.509734 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.509707    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 11 21:06:49.509797 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.509762    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 11 21:06:49.510262 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.510242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bebc5ca8-5cdf-41db-bda5-ed32fa7fa05d-klusterlet-config\") pod \"klusterlet-addon-workmgr-868576d4cb-jfsgj\" (UID: \"bebc5ca8-5cdf-41db-bda5-ed32fa7fa05d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-868576d4cb-jfsgj"
Sep 11 21:06:49.510338 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.510273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x7dm\" (UniqueName: \"kubernetes.io/projected/bebc5ca8-5cdf-41db-bda5-ed32fa7fa05d-kube-api-access-6x7dm\") pod \"klusterlet-addon-workmgr-868576d4cb-jfsgj\" (UID: \"bebc5ca8-5cdf-41db-bda5-ed32fa7fa05d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-868576d4cb-jfsgj"
Sep 11 21:06:49.510384 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.510353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bebc5ca8-5cdf-41db-bda5-ed32fa7fa05d-tmp\") pod \"klusterlet-addon-workmgr-868576d4cb-jfsgj\" (UID: \"bebc5ca8-5cdf-41db-bda5-ed32fa7fa05d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-868576d4cb-jfsgj"
Sep 11 21:06:49.514595 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.514572    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-868576d4cb-jfsgj"]
Sep 11 21:06:49.532971 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.532949    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj"]
Sep 11 21:06:49.594182 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.594153    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5859b9dc4-62dsf"]
Sep 11 21:06:49.597279 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.597262    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:49.600251 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.600235    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 21:06:49.600328 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.600311    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 21:06:49.600942 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.600930    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j64xm\""
Sep 11 21:06:49.602354 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.602339    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 21:06:49.610693 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.610672    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/88de7057-f51a-4a6b-b9b9-15203dba6448-ca\") pod \"cluster-proxy-proxy-agent-747f98bcc4-wftrj\" (UID: \"88de7057-f51a-4a6b-b9b9-15203dba6448\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj"
Sep 11 21:06:49.610803 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.610702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bebc5ca8-5cdf-41db-bda5-ed32fa7fa05d-klusterlet-config\") pod \"klusterlet-addon-workmgr-868576d4cb-jfsgj\" (UID: \"bebc5ca8-5cdf-41db-bda5-ed32fa7fa05d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-868576d4cb-jfsgj"
Sep 11 21:06:49.610803 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.610730    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/88de7057-f51a-4a6b-b9b9-15203dba6448-additional-service-ca\") pod \"cluster-proxy-proxy-agent-747f98bcc4-wftrj\" (UID: \"88de7057-f51a-4a6b-b9b9-15203dba6448\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj"
Sep 11 21:06:49.610803 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.610757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x7dm\" (UniqueName: \"kubernetes.io/projected/bebc5ca8-5cdf-41db-bda5-ed32fa7fa05d-kube-api-access-6x7dm\") pod \"klusterlet-addon-workmgr-868576d4cb-jfsgj\" (UID: \"bebc5ca8-5cdf-41db-bda5-ed32fa7fa05d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-868576d4cb-jfsgj"
Sep 11 21:06:49.611194 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.611169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkqpd\" (UniqueName: \"kubernetes.io/projected/88de7057-f51a-4a6b-b9b9-15203dba6448-kube-api-access-pkqpd\") pod \"cluster-proxy-proxy-agent-747f98bcc4-wftrj\" (UID: \"88de7057-f51a-4a6b-b9b9-15203dba6448\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj"
Sep 11 21:06:49.611280 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.611255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bebc5ca8-5cdf-41db-bda5-ed32fa7fa05d-tmp\") pod \"klusterlet-addon-workmgr-868576d4cb-jfsgj\" (UID: \"bebc5ca8-5cdf-41db-bda5-ed32fa7fa05d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-868576d4cb-jfsgj"
Sep 11 21:06:49.611340 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.611296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/88de7057-f51a-4a6b-b9b9-15203dba6448-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-747f98bcc4-wftrj\" (UID: \"88de7057-f51a-4a6b-b9b9-15203dba6448\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj"
Sep 11 21:06:49.611404 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.611353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/88de7057-f51a-4a6b-b9b9-15203dba6448-hub\") pod \"cluster-proxy-proxy-agent-747f98bcc4-wftrj\" (UID: \"88de7057-f51a-4a6b-b9b9-15203dba6448\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj"
Sep 11 21:06:49.611502 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.611472    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88de7057-f51a-4a6b-b9b9-15203dba6448-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-747f98bcc4-wftrj\" (UID: \"88de7057-f51a-4a6b-b9b9-15203dba6448\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj"
Sep 11 21:06:49.611739 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.611715    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bebc5ca8-5cdf-41db-bda5-ed32fa7fa05d-tmp\") pod \"klusterlet-addon-workmgr-868576d4cb-jfsgj\" (UID: \"bebc5ca8-5cdf-41db-bda5-ed32fa7fa05d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-868576d4cb-jfsgj"
Sep 11 21:06:49.613532 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.613508    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 21:06:49.614879 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.614856    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bebc5ca8-5cdf-41db-bda5-ed32fa7fa05d-klusterlet-config\") pod \"klusterlet-addon-workmgr-868576d4cb-jfsgj\" (UID: \"bebc5ca8-5cdf-41db-bda5-ed32fa7fa05d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-868576d4cb-jfsgj"
Sep 11 21:06:49.623145 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.623125    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5859b9dc4-62dsf"]
Sep 11 21:06:49.635703 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.635675    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x7dm\" (UniqueName: \"kubernetes.io/projected/bebc5ca8-5cdf-41db-bda5-ed32fa7fa05d-kube-api-access-6x7dm\") pod \"klusterlet-addon-workmgr-868576d4cb-jfsgj\" (UID: \"bebc5ca8-5cdf-41db-bda5-ed32fa7fa05d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-868576d4cb-jfsgj"
Sep 11 21:06:49.675592 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.675572    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5859b9dc4-62dsf"]
Sep 11 21:06:49.675729 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:06:49.675710    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-jpl46 registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-jpl46 registry-certificates registry-tls trusted-ca]: context canceled" pod="openshift-image-registry/image-registry-5859b9dc4-62dsf" podUID="c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4"
Sep 11 21:06:49.712509 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.712487    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpl46\" (UniqueName: \"kubernetes.io/projected/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-kube-api-access-jpl46\") pod \"image-registry-5859b9dc4-62dsf\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") " pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:49.712601 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.712524    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-registry-tls\") pod \"image-registry-5859b9dc4-62dsf\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") " pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:49.712601 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.712539    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-trusted-ca\") pod \"image-registry-5859b9dc4-62dsf\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") " pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:49.712601 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.712562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/88de7057-f51a-4a6b-b9b9-15203dba6448-additional-service-ca\") pod \"cluster-proxy-proxy-agent-747f98bcc4-wftrj\" (UID: \"88de7057-f51a-4a6b-b9b9-15203dba6448\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj"
Sep 11 21:06:49.712718 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.712621    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-image-registry-private-configuration\") pod \"image-registry-5859b9dc4-62dsf\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") " pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:49.712718 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.712674    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-ca-trust-extracted\") pod \"image-registry-5859b9dc4-62dsf\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") " pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:49.712789 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.712722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/88de7057-f51a-4a6b-b9b9-15203dba6448-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-747f98bcc4-wftrj\" (UID: \"88de7057-f51a-4a6b-b9b9-15203dba6448\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj"
Sep 11 21:06:49.712789 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.712750    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-installation-pull-secrets\") pod \"image-registry-5859b9dc4-62dsf\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") " pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:49.712789 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.712783    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-bound-sa-token\") pod \"image-registry-5859b9dc4-62dsf\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") " pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:49.712929 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.712840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/88de7057-f51a-4a6b-b9b9-15203dba6448-hub\") pod \"cluster-proxy-proxy-agent-747f98bcc4-wftrj\" (UID: \"88de7057-f51a-4a6b-b9b9-15203dba6448\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj"
Sep 11 21:06:49.712929 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.712873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88de7057-f51a-4a6b-b9b9-15203dba6448-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-747f98bcc4-wftrj\" (UID: \"88de7057-f51a-4a6b-b9b9-15203dba6448\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj"
Sep 11 21:06:49.712929 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.712899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-registry-certificates\") pod \"image-registry-5859b9dc4-62dsf\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") " pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:49.713076 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.712951    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkqpd\" (UniqueName: \"kubernetes.io/projected/88de7057-f51a-4a6b-b9b9-15203dba6448-kube-api-access-pkqpd\") pod \"cluster-proxy-proxy-agent-747f98bcc4-wftrj\" (UID: \"88de7057-f51a-4a6b-b9b9-15203dba6448\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj"
Sep 11 21:06:49.713076 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.712994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/88de7057-f51a-4a6b-b9b9-15203dba6448-ca\") pod \"cluster-proxy-proxy-agent-747f98bcc4-wftrj\" (UID: \"88de7057-f51a-4a6b-b9b9-15203dba6448\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj"
Sep 11 21:06:49.713214 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.713196    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/88de7057-f51a-4a6b-b9b9-15203dba6448-additional-service-ca\") pod \"cluster-proxy-proxy-agent-747f98bcc4-wftrj\" (UID: \"88de7057-f51a-4a6b-b9b9-15203dba6448\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj"
Sep 11 21:06:49.715028 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.715004    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/88de7057-f51a-4a6b-b9b9-15203dba6448-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-747f98bcc4-wftrj\" (UID: \"88de7057-f51a-4a6b-b9b9-15203dba6448\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj"
Sep 11 21:06:49.715128 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.715104    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/88de7057-f51a-4a6b-b9b9-15203dba6448-ca\") pod \"cluster-proxy-proxy-agent-747f98bcc4-wftrj\" (UID: \"88de7057-f51a-4a6b-b9b9-15203dba6448\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj"
Sep 11 21:06:49.715320 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.715301    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88de7057-f51a-4a6b-b9b9-15203dba6448-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-747f98bcc4-wftrj\" (UID: \"88de7057-f51a-4a6b-b9b9-15203dba6448\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj"
Sep 11 21:06:49.715372 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.715323    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/88de7057-f51a-4a6b-b9b9-15203dba6448-hub\") pod \"cluster-proxy-proxy-agent-747f98bcc4-wftrj\" (UID: \"88de7057-f51a-4a6b-b9b9-15203dba6448\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj"
Sep 11 21:06:49.722334 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.722317    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkqpd\" (UniqueName: \"kubernetes.io/projected/88de7057-f51a-4a6b-b9b9-15203dba6448-kube-api-access-pkqpd\") pod \"cluster-proxy-proxy-agent-747f98bcc4-wftrj\" (UID: \"88de7057-f51a-4a6b-b9b9-15203dba6448\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj"
Sep 11 21:06:49.799307 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.799257    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-868576d4cb-jfsgj"
Sep 11 21:06:49.814112 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.814088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-image-registry-private-configuration\") pod \"image-registry-5859b9dc4-62dsf\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") " pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:49.814212 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.814114    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-ca-trust-extracted\") pod \"image-registry-5859b9dc4-62dsf\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") " pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:49.814212 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.814139    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-installation-pull-secrets\") pod \"image-registry-5859b9dc4-62dsf\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") " pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:49.814212 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.814166    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-bound-sa-token\") pod \"image-registry-5859b9dc4-62dsf\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") " pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:49.814370 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.814260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-registry-certificates\") pod \"image-registry-5859b9dc4-62dsf\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") " pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:49.814370 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.814341    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpl46\" (UniqueName: \"kubernetes.io/projected/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-kube-api-access-jpl46\") pod \"image-registry-5859b9dc4-62dsf\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") " pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:49.814459 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.814387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-registry-tls\") pod \"image-registry-5859b9dc4-62dsf\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") " pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:49.814459 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.814405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-trusted-ca\") pod \"image-registry-5859b9dc4-62dsf\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") " pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:49.814555 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.814466    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-ca-trust-extracted\") pod \"image-registry-5859b9dc4-62dsf\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") " pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:49.815246 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.815227    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-registry-certificates\") pod \"image-registry-5859b9dc4-62dsf\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") " pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:49.815325 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.815297    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-trusted-ca\") pod \"image-registry-5859b9dc4-62dsf\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") " pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:49.817204 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.817172    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-image-registry-private-configuration\") pod \"image-registry-5859b9dc4-62dsf\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") " pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:49.817204 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.817191    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-installation-pull-secrets\") pod \"image-registry-5859b9dc4-62dsf\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") " pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:49.817204 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.817194    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-registry-tls\") pod \"image-registry-5859b9dc4-62dsf\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") " pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:49.824890 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.824837    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj"
Sep 11 21:06:49.825034 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.825012    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpl46\" (UniqueName: \"kubernetes.io/projected/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-kube-api-access-jpl46\") pod \"image-registry-5859b9dc4-62dsf\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") " pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:49.825115 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:49.825099    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-bound-sa-token\") pod \"image-registry-5859b9dc4-62dsf\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") " pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:50.072485 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.072419    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-868576d4cb-jfsgj"]
Sep 11 21:06:50.075637 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:06:50.075608    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbebc5ca8_5cdf_41db_bda5_ed32fa7fa05d.slice/crio-f65a055b822fa1e767e19dc5a0348c17253e41fc04f28e0bc0c3deec46bada8e WatchSource:0}: Error finding container f65a055b822fa1e767e19dc5a0348c17253e41fc04f28e0bc0c3deec46bada8e: Status 404 returned error can't find the container with id f65a055b822fa1e767e19dc5a0348c17253e41fc04f28e0bc0c3deec46bada8e
Sep 11 21:06:50.103988 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.103967    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj"]
Sep 11 21:06:50.105968 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:06:50.105947    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88de7057_f51a_4a6b_b9b9_15203dba6448.slice/crio-52ece0e0964318d4503e9ba6b069d824051602afb78c5350ba95d7b38e252db8 WatchSource:0}: Error finding container 52ece0e0964318d4503e9ba6b069d824051602afb78c5350ba95d7b38e252db8: Status 404 returned error can't find the container with id 52ece0e0964318d4503e9ba6b069d824051602afb78c5350ba95d7b38e252db8
Sep 11 21:06:50.252671 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.252642    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj" event={"ID":"88de7057-f51a-4a6b-b9b9-15203dba6448","Type":"ContainerStarted","Data":"52ece0e0964318d4503e9ba6b069d824051602afb78c5350ba95d7b38e252db8"}
Sep 11 21:06:50.253888 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.253867    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-868576d4cb-jfsgj" event={"ID":"bebc5ca8-5cdf-41db-bda5-ed32fa7fa05d","Type":"ContainerStarted","Data":"f65a055b822fa1e767e19dc5a0348c17253e41fc04f28e0bc0c3deec46bada8e"}
Sep 11 21:06:50.253972 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.253873    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:50.261868 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.261850    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:50.317264 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.317236    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-trusted-ca\") pod \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") "
Sep 11 21:06:50.317353 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.317277    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpl46\" (UniqueName: \"kubernetes.io/projected/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-kube-api-access-jpl46\") pod \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") "
Sep 11 21:06:50.317353 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.317312    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-registry-certificates\") pod \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") "
Sep 11 21:06:50.317459 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.317416    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-registry-tls\") pod \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") "
Sep 11 21:06:50.317459 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.317447    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-image-registry-private-configuration\") pod \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") "
Sep 11 21:06:50.317547 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.317472    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-bound-sa-token\") pod \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") "
Sep 11 21:06:50.317547 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.317496    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-ca-trust-extracted\") pod \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") "
Sep 11 21:06:50.317547 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.317536    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-installation-pull-secrets\") pod \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\" (UID: \"c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4\") "
Sep 11 21:06:50.317730 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.317682    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4" (UID: "c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:06:50.317777 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.317722    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4" (UID: "c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:06:50.319084 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.319053    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4" (UID: "c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:06:50.319622 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.319602    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-kube-api-access-jpl46" (OuterVolumeSpecName: "kube-api-access-jpl46") pod "c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4" (UID: "c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4"). InnerVolumeSpecName "kube-api-access-jpl46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:06:50.319706 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.319688    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4" (UID: "c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:06:50.320629 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.320613    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4" (UID: "c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:06:50.321190 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.321166    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4" (UID: "c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:06:50.321247 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.321185    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4" (UID: "c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:06:50.418757 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.418702    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-registry-certificates\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:06:50.418757 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.418724    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-registry-tls\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:06:50.418757 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.418734    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-image-registry-private-configuration\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:06:50.418757 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.418749    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-bound-sa-token\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:06:50.418757 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.418758    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-ca-trust-extracted\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:06:50.418968 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.418766    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-installation-pull-secrets\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:06:50.418968 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.418776    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-trusted-ca\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:06:50.418968 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:50.418784    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpl46\" (UniqueName: \"kubernetes.io/projected/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4-kube-api-access-jpl46\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:06:51.264254 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:51.259626    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5859b9dc4-62dsf"
Sep 11 21:06:51.304203 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:51.304136    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5859b9dc4-62dsf"]
Sep 11 21:06:51.309395 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:51.309343    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5859b9dc4-62dsf"]
Sep 11 21:06:51.927140 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:51.927076    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4" path="/var/lib/kubelet/pods/c5b8ed78-a9c9-4820-a4a8-0fe00faa2fd4/volumes"
Sep 11 21:06:52.938247 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:52.938194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5cb0d0f-a841-4121-b7bf-d6d9628faded-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:52.942861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:52.942709    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e5cb0d0f-a841-4121-b7bf-d6d9628faded-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c28km\" (UID: \"e5cb0d0f-a841-4121-b7bf-d6d9628faded\") " pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:53.210082 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:53.209998    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gq824\""
Sep 11 21:06:53.218299 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:53.218271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c28km"
Sep 11 21:06:56.086692 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:56.086629    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c28km"]
Sep 11 21:06:56.089495 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:06:56.089463    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5cb0d0f_a841_4121_b7bf_d6d9628faded.slice/crio-496c3cab090222bebceb03688e3d08b7a7d9bc5f91d58c78fd47f3b768f7f21b WatchSource:0}: Error finding container 496c3cab090222bebceb03688e3d08b7a7d9bc5f91d58c78fd47f3b768f7f21b: Status 404 returned error can't find the container with id 496c3cab090222bebceb03688e3d08b7a7d9bc5f91d58c78fd47f3b768f7f21b
Sep 11 21:06:56.279895 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:56.279859    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj" event={"ID":"88de7057-f51a-4a6b-b9b9-15203dba6448","Type":"ContainerStarted","Data":"03739f59c86783d940c6406c0792e50a49abc4c092f70f23f7f1db71ca4bd7d4"}
Sep 11 21:06:56.280105 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:56.279908    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj" event={"ID":"88de7057-f51a-4a6b-b9b9-15203dba6448","Type":"ContainerStarted","Data":"47275da3fa016348465a5d99ab4ab10441979ec6234027a6564eaa66e2183e3f"}
Sep 11 21:06:56.280105 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:56.279926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj" event={"ID":"88de7057-f51a-4a6b-b9b9-15203dba6448","Type":"ContainerStarted","Data":"38c4134ee01f3dbdb056894b4c7737d8a53c539b0acfd9c7842105de02f1cb74"}
Sep 11 21:06:56.281380 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:56.281351    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-868576d4cb-jfsgj" event={"ID":"bebc5ca8-5cdf-41db-bda5-ed32fa7fa05d","Type":"ContainerStarted","Data":"23aba66f7118eeb13504ffccc19bdd592d86b45529dcef0c057585f32297bc09"}
Sep 11 21:06:56.281498 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:56.281409    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-868576d4cb-jfsgj"
Sep 11 21:06:56.282981 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:56.282956    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c28km" event={"ID":"e5cb0d0f-a841-4121-b7bf-d6d9628faded","Type":"ContainerStarted","Data":"3e921198b8c3bbfc264b7ccab24ec6596a26aacdccfbd7e09e6ad80010e0fd28"}
Sep 11 21:06:56.283105 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:56.283001    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c28km" event={"ID":"e5cb0d0f-a841-4121-b7bf-d6d9628faded","Type":"ContainerStarted","Data":"496c3cab090222bebceb03688e3d08b7a7d9bc5f91d58c78fd47f3b768f7f21b"}
Sep 11 21:06:56.283448 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:56.283432    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-868576d4cb-jfsgj"
Sep 11 21:06:56.300587 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:56.300545    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-747f98bcc4-wftrj" podStartSLOduration=1.570514013 podStartE2EDuration="7.300531069s" totalImagesPullingTime="5.730017056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:06:56.299472906 +0000 UTC m=+83.003442196" watchObservedRunningTime="2026-09-11 21:06:56.300531069 +0000 UTC m=+83.004500362"
Sep 11 21:06:56.327191 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:56.327143    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-868576d4cb-jfsgj" podStartSLOduration=1.573006173 podStartE2EDuration="7.32712818s" totalImagesPullingTime="5.754122007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:06:56.326722153 +0000 UTC m=+83.030691442" watchObservedRunningTime="2026-09-11 21:06:56.32712818 +0000 UTC m=+83.031097497"
Sep 11 21:06:57.244546 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:57.244518    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hgf4n"
Sep 11 21:06:57.289855 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:57.289805    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c28km" event={"ID":"e5cb0d0f-a841-4121-b7bf-d6d9628faded","Type":"ContainerStarted","Data":"a404bf10c6afb8fcba833a8494ebe49b742c570570eaf972484de9760f715ad7"}
Sep 11 21:06:58.295189 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:58.295155    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c28km" event={"ID":"e5cb0d0f-a841-4121-b7bf-d6d9628faded","Type":"ContainerStarted","Data":"12e77907f0b7a64224c1dc1db2fc43d0d74105280009e0b999d721b230f1b50a"}
Sep 11 21:06:58.313833 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:06:58.313786    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-c28km" podStartSLOduration=35.383305087 podStartE2EDuration="37.313770234s" totalImagesPullingTime="1.930465147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:06:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:06:58.313101892 +0000 UTC m=+85.017071181" watchObservedRunningTime="2026-09-11 21:06:58.313770234 +0000 UTC m=+85.017739521"
Sep 11 21:07:02.880459 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:02.880405    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hm52m"]
Sep 11 21:07:02.883870 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:02.883852    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:02.886737 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:02.886699    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 21:07:02.887083 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:02.887049    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 21:07:02.887083 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:02.887065    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 21:07:02.887494 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:02.887199    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 21:07:02.887494 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:02.887245    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 21:07:02.887494 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:02.887456    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fhhrl\""
Sep 11 21:07:02.887976 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:02.887961    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 21:07:03.010085 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.010058    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-node-exporter-accelerators-collector-config\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.010198 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.010104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7f2n\" (UniqueName: \"kubernetes.io/projected/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-kube-api-access-r7f2n\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.010198 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.010141    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-root\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.010198 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.010168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-node-exporter-wtmp\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.010336 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.010199    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-metrics-client-ca\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.010336 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.010225    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-node-exporter-textfile\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.010336 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.010251    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-sys\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.010336 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.010274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-node-exporter-tls\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.010336 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.010302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.111233 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.111210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-node-exporter-wtmp\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.111363 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.111241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-metrics-client-ca\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.111363 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.111257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-node-exporter-textfile\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.111363 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.111276    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-sys\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.111363 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.111291    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-node-exporter-tls\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.111363 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.111335    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.111601 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.111365    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-sys\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.111601 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.111374    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-node-exporter-wtmp\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.111601 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.111388    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-node-exporter-accelerators-collector-config\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.111601 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.111433    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7f2n\" (UniqueName: \"kubernetes.io/projected/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-kube-api-access-r7f2n\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.111601 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.111473    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-root\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.111601 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.111533    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-root\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.111798 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.111606    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-node-exporter-textfile\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.111919 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.111899    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-metrics-client-ca\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.111982 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.111926    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-node-exporter-accelerators-collector-config\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.113639 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.113616    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.113791 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.113775    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-node-exporter-tls\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.120078 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.120057    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7f2n\" (UniqueName: \"kubernetes.io/projected/1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7-kube-api-access-r7f2n\") pod \"node-exporter-hm52m\" (UID: \"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7\") " pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.193725 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.193702    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hm52m"
Sep 11 21:07:03.204844 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:07:03.204795    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1134f7c2_f8c5_4ab9_b028_3a76c9e0b4a7.slice/crio-d648629369170c4fbd11b14d8b14eaa5ad276fa2e0fc067794095b30d2c721df WatchSource:0}: Error finding container d648629369170c4fbd11b14d8b14eaa5ad276fa2e0fc067794095b30d2c721df: Status 404 returned error can't find the container with id d648629369170c4fbd11b14d8b14eaa5ad276fa2e0fc067794095b30d2c721df
Sep 11 21:07:03.314267 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:03.314241    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hm52m" event={"ID":"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7","Type":"ContainerStarted","Data":"d648629369170c4fbd11b14d8b14eaa5ad276fa2e0fc067794095b30d2c721df"}
Sep 11 21:07:04.319204 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:04.319175    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hm52m" event={"ID":"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7","Type":"ContainerStarted","Data":"97304cfb11541c5de6de51a33025ab36b28a42052751fd10edf8f7a9f3f85064"}
Sep 11 21:07:05.325088 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:05.325028    2578 generic.go:356] "Generic (PLEG): container finished" podID="1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7" containerID="97304cfb11541c5de6de51a33025ab36b28a42052751fd10edf8f7a9f3f85064" exitCode=0
Sep 11 21:07:05.325577 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:05.325120    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hm52m" event={"ID":"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7","Type":"ContainerDied","Data":"97304cfb11541c5de6de51a33025ab36b28a42052751fd10edf8f7a9f3f85064"}
Sep 11 21:07:05.325577 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:05.325181    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hm52m" event={"ID":"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7","Type":"ContainerStarted","Data":"98fb8924f0a7dd3503d6f9ac413992f3b699638fea375d1e3c0831c2bcfaf9dc"}
Sep 11 21:07:05.325577 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:05.325200    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hm52m" event={"ID":"1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7","Type":"ContainerStarted","Data":"964af07f104490d1b0921a3005d0776f0ede2f1aacbb545b17b81d2334fdf4c5"}
Sep 11 21:07:05.346912 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:05.346852    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hm52m" podStartSLOduration=2.301619493 podStartE2EDuration="3.346807462s" totalImagesPullingTime="986.735936ms" totalInitContainerRuntime="58.452033ms" isStatefulPod=true podCreationTimestamp="2026-09-11 21:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:07:05.344579847 +0000 UTC m=+92.048549138" watchObservedRunningTime="2026-09-11 21:07:05.346807462 +0000 UTC m=+92.050776752"
Sep 11 21:07:12.783759 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.783728    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-679b8585d9-xmm2b"]
Sep 11 21:07:12.789998 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.789974    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:12.793065 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.793042    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jgwp6\""
Sep 11 21:07:12.793201 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.793160    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 21:07:12.793201 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.793051    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 21:07:12.793201 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.793053    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 21:07:12.793201 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.793085    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 21:07:12.793426 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.793141    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 21:07:12.793426 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.793260    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 21:07:12.793426 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.793112    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 21:07:12.798357 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.798336    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 21:07:12.799542 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.799524    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-679b8585d9-xmm2b"]
Sep 11 21:07:12.876656 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.876633    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3c174ff-aad4-4543-9c30-0e9284a34da3-oauth-serving-cert\") pod \"console-679b8585d9-xmm2b\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") " pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:12.876752 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.876663    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3c174ff-aad4-4543-9c30-0e9284a34da3-trusted-ca-bundle\") pod \"console-679b8585d9-xmm2b\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") " pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:12.876752 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.876694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3c174ff-aad4-4543-9c30-0e9284a34da3-console-config\") pod \"console-679b8585d9-xmm2b\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") " pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:12.876752 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.876737    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3c174ff-aad4-4543-9c30-0e9284a34da3-console-oauth-config\") pod \"console-679b8585d9-xmm2b\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") " pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:12.876752 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.876752    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c174ff-aad4-4543-9c30-0e9284a34da3-service-ca\") pod \"console-679b8585d9-xmm2b\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") " pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:12.876925 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.876770    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3c174ff-aad4-4543-9c30-0e9284a34da3-console-serving-cert\") pod \"console-679b8585d9-xmm2b\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") " pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:12.876925 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.876794    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47vcr\" (UniqueName: \"kubernetes.io/projected/d3c174ff-aad4-4543-9c30-0e9284a34da3-kube-api-access-47vcr\") pod \"console-679b8585d9-xmm2b\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") " pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:12.977458 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.977436    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3c174ff-aad4-4543-9c30-0e9284a34da3-console-oauth-config\") pod \"console-679b8585d9-xmm2b\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") " pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:12.977537 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.977461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c174ff-aad4-4543-9c30-0e9284a34da3-service-ca\") pod \"console-679b8585d9-xmm2b\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") " pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:12.977537 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.977489    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3c174ff-aad4-4543-9c30-0e9284a34da3-console-serving-cert\") pod \"console-679b8585d9-xmm2b\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") " pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:12.977612 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.977532    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vcr\" (UniqueName: \"kubernetes.io/projected/d3c174ff-aad4-4543-9c30-0e9284a34da3-kube-api-access-47vcr\") pod \"console-679b8585d9-xmm2b\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") " pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:12.977674 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.977610    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3c174ff-aad4-4543-9c30-0e9284a34da3-oauth-serving-cert\") pod \"console-679b8585d9-xmm2b\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") " pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:12.977873 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.977849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3c174ff-aad4-4543-9c30-0e9284a34da3-trusted-ca-bundle\") pod \"console-679b8585d9-xmm2b\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") " pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:12.977996 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.977919    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3c174ff-aad4-4543-9c30-0e9284a34da3-console-config\") pod \"console-679b8585d9-xmm2b\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") " pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:12.978310 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.978286    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c174ff-aad4-4543-9c30-0e9284a34da3-service-ca\") pod \"console-679b8585d9-xmm2b\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") " pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:12.978479 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.978457    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3c174ff-aad4-4543-9c30-0e9284a34da3-oauth-serving-cert\") pod \"console-679b8585d9-xmm2b\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") " pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:12.978607 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.978587    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3c174ff-aad4-4543-9c30-0e9284a34da3-console-config\") pod \"console-679b8585d9-xmm2b\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") " pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:12.978678 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.978666    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3c174ff-aad4-4543-9c30-0e9284a34da3-trusted-ca-bundle\") pod \"console-679b8585d9-xmm2b\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") " pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:12.979895 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.979878    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3c174ff-aad4-4543-9c30-0e9284a34da3-console-serving-cert\") pod \"console-679b8585d9-xmm2b\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") " pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:12.979982 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.979912    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3c174ff-aad4-4543-9c30-0e9284a34da3-console-oauth-config\") pod \"console-679b8585d9-xmm2b\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") " pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:12.986251 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:12.986227    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47vcr\" (UniqueName: \"kubernetes.io/projected/d3c174ff-aad4-4543-9c30-0e9284a34da3-kube-api-access-47vcr\") pod \"console-679b8585d9-xmm2b\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") " pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:13.100171 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:13.100123    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:13.374024 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:13.373945    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-679b8585d9-xmm2b"]
Sep 11 21:07:13.377047 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:07:13.377020    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c174ff_aad4_4543_9c30_0e9284a34da3.slice/crio-5b0a73ea914a0b57c37e494868912be80f7ad909bc611cc6ed9b2ec9ea7e868b WatchSource:0}: Error finding container 5b0a73ea914a0b57c37e494868912be80f7ad909bc611cc6ed9b2ec9ea7e868b: Status 404 returned error can't find the container with id 5b0a73ea914a0b57c37e494868912be80f7ad909bc611cc6ed9b2ec9ea7e868b
Sep 11 21:07:14.355703 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:14.355663    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679b8585d9-xmm2b" event={"ID":"d3c174ff-aad4-4543-9c30-0e9284a34da3","Type":"ContainerStarted","Data":"5b0a73ea914a0b57c37e494868912be80f7ad909bc611cc6ed9b2ec9ea7e868b"}
Sep 11 21:07:15.230950 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:15.230919    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cjtsr"
Sep 11 21:07:17.366410 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:17.366374    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679b8585d9-xmm2b" event={"ID":"d3c174ff-aad4-4543-9c30-0e9284a34da3","Type":"ContainerStarted","Data":"78b3fc16e711177e59f86cd9bc58a5165b3fdeb9b4ddce855aa8c4e739dcb797"}
Sep 11 21:07:17.388763 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:17.388720    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-679b8585d9-xmm2b" podStartSLOduration=2.368855425 podStartE2EDuration="5.388706073s" totalImagesPullingTime="3.019850648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:07:17.388015799 +0000 UTC m=+104.091985087" watchObservedRunningTime="2026-09-11 21:07:17.388706073 +0000 UTC m=+104.092675363"
Sep 11 21:07:23.101160 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:23.101132    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:23.101523 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:23.101406    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:23.105721 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:23.105700    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:07:23.391301 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:07:23.391233    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:08:36.817526 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:08:36.817434    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-679b8585d9-xmm2b"]
Sep 11 21:09:01.836165 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:01.836116    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-679b8585d9-xmm2b" podUID="d3c174ff-aad4-4543-9c30-0e9284a34da3" containerName="console" containerID="cri-o://78b3fc16e711177e59f86cd9bc58a5165b3fdeb9b4ddce855aa8c4e739dcb797" gracePeriod=15
Sep 11 21:09:02.069759 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.069737    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:09:02.117297 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.117227    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3c174ff-aad4-4543-9c30-0e9284a34da3-oauth-serving-cert\") pod \"d3c174ff-aad4-4543-9c30-0e9284a34da3\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") "
Sep 11 21:09:02.117297 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.117267    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3c174ff-aad4-4543-9c30-0e9284a34da3-console-oauth-config\") pod \"d3c174ff-aad4-4543-9c30-0e9284a34da3\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") "
Sep 11 21:09:02.117496 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.117304    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3c174ff-aad4-4543-9c30-0e9284a34da3-console-config\") pod \"d3c174ff-aad4-4543-9c30-0e9284a34da3\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") "
Sep 11 21:09:02.117496 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.117353    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3c174ff-aad4-4543-9c30-0e9284a34da3-trusted-ca-bundle\") pod \"d3c174ff-aad4-4543-9c30-0e9284a34da3\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") "
Sep 11 21:09:02.117496 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.117382    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c174ff-aad4-4543-9c30-0e9284a34da3-service-ca\") pod \"d3c174ff-aad4-4543-9c30-0e9284a34da3\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") "
Sep 11 21:09:02.117650 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.117530    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3c174ff-aad4-4543-9c30-0e9284a34da3-console-serving-cert\") pod \"d3c174ff-aad4-4543-9c30-0e9284a34da3\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") "
Sep 11 21:09:02.117650 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.117588    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47vcr\" (UniqueName: \"kubernetes.io/projected/d3c174ff-aad4-4543-9c30-0e9284a34da3-kube-api-access-47vcr\") pod \"d3c174ff-aad4-4543-9c30-0e9284a34da3\" (UID: \"d3c174ff-aad4-4543-9c30-0e9284a34da3\") "
Sep 11 21:09:02.117650 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.117593    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c174ff-aad4-4543-9c30-0e9284a34da3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d3c174ff-aad4-4543-9c30-0e9284a34da3" (UID: "d3c174ff-aad4-4543-9c30-0e9284a34da3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:09:02.117799 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.117764    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c174ff-aad4-4543-9c30-0e9284a34da3-console-config" (OuterVolumeSpecName: "console-config") pod "d3c174ff-aad4-4543-9c30-0e9284a34da3" (UID: "d3c174ff-aad4-4543-9c30-0e9284a34da3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:09:02.117799 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.117778    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c174ff-aad4-4543-9c30-0e9284a34da3-service-ca" (OuterVolumeSpecName: "service-ca") pod "d3c174ff-aad4-4543-9c30-0e9284a34da3" (UID: "d3c174ff-aad4-4543-9c30-0e9284a34da3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:09:02.117892 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.117808    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3c174ff-aad4-4543-9c30-0e9284a34da3-oauth-serving-cert\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:09:02.117892 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.117803    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c174ff-aad4-4543-9c30-0e9284a34da3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d3c174ff-aad4-4543-9c30-0e9284a34da3" (UID: "d3c174ff-aad4-4543-9c30-0e9284a34da3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:09:02.119650 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.119619    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c174ff-aad4-4543-9c30-0e9284a34da3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d3c174ff-aad4-4543-9c30-0e9284a34da3" (UID: "d3c174ff-aad4-4543-9c30-0e9284a34da3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:09:02.119747 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.119653    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c174ff-aad4-4543-9c30-0e9284a34da3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d3c174ff-aad4-4543-9c30-0e9284a34da3" (UID: "d3c174ff-aad4-4543-9c30-0e9284a34da3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:09:02.119747 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.119690    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c174ff-aad4-4543-9c30-0e9284a34da3-kube-api-access-47vcr" (OuterVolumeSpecName: "kube-api-access-47vcr") pod "d3c174ff-aad4-4543-9c30-0e9284a34da3" (UID: "d3c174ff-aad4-4543-9c30-0e9284a34da3"). InnerVolumeSpecName "kube-api-access-47vcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:09:02.218225 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.218194    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3c174ff-aad4-4543-9c30-0e9284a34da3-console-oauth-config\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:09:02.218355 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.218220    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3c174ff-aad4-4543-9c30-0e9284a34da3-console-config\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:09:02.218355 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.218242    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3c174ff-aad4-4543-9c30-0e9284a34da3-trusted-ca-bundle\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:09:02.218355 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.218251    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c174ff-aad4-4543-9c30-0e9284a34da3-service-ca\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:09:02.218355 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.218259    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3c174ff-aad4-4543-9c30-0e9284a34da3-console-serving-cert\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:09:02.218355 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.218268    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47vcr\" (UniqueName: \"kubernetes.io/projected/d3c174ff-aad4-4543-9c30-0e9284a34da3-kube-api-access-47vcr\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:09:02.723837 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.723782    2578 generic.go:356] "Generic (PLEG): container finished" podID="d3c174ff-aad4-4543-9c30-0e9284a34da3" containerID="78b3fc16e711177e59f86cd9bc58a5165b3fdeb9b4ddce855aa8c4e739dcb797" exitCode=0
Sep 11 21:09:02.723837 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.723840    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679b8585d9-xmm2b"
Sep 11 21:09:02.724090 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.723872    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679b8585d9-xmm2b" event={"ID":"d3c174ff-aad4-4543-9c30-0e9284a34da3","Type":"ContainerDied","Data":"78b3fc16e711177e59f86cd9bc58a5165b3fdeb9b4ddce855aa8c4e739dcb797"}
Sep 11 21:09:02.724090 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.723920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679b8585d9-xmm2b" event={"ID":"d3c174ff-aad4-4543-9c30-0e9284a34da3","Type":"ContainerDied","Data":"5b0a73ea914a0b57c37e494868912be80f7ad909bc611cc6ed9b2ec9ea7e868b"}
Sep 11 21:09:02.724090 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.723944    2578 scope.go:117] "RemoveContainer" containerID="78b3fc16e711177e59f86cd9bc58a5165b3fdeb9b4ddce855aa8c4e739dcb797"
Sep 11 21:09:02.732663 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.732646    2578 scope.go:117] "RemoveContainer" containerID="78b3fc16e711177e59f86cd9bc58a5165b3fdeb9b4ddce855aa8c4e739dcb797"
Sep 11 21:09:02.732962 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:09:02.732940    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78b3fc16e711177e59f86cd9bc58a5165b3fdeb9b4ddce855aa8c4e739dcb797\": container with ID starting with 78b3fc16e711177e59f86cd9bc58a5165b3fdeb9b4ddce855aa8c4e739dcb797 not found: ID does not exist" containerID="78b3fc16e711177e59f86cd9bc58a5165b3fdeb9b4ddce855aa8c4e739dcb797"
Sep 11 21:09:02.733052 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.732968    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78b3fc16e711177e59f86cd9bc58a5165b3fdeb9b4ddce855aa8c4e739dcb797"} err="failed to get container status \"78b3fc16e711177e59f86cd9bc58a5165b3fdeb9b4ddce855aa8c4e739dcb797\": rpc error: code = NotFound desc = could not find container \"78b3fc16e711177e59f86cd9bc58a5165b3fdeb9b4ddce855aa8c4e739dcb797\": container with ID starting with 78b3fc16e711177e59f86cd9bc58a5165b3fdeb9b4ddce855aa8c4e739dcb797 not found: ID does not exist"
Sep 11 21:09:02.748835 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.748789    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-679b8585d9-xmm2b"]
Sep 11 21:09:02.759930 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:02.759903    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-679b8585d9-xmm2b"]
Sep 11 21:09:03.926022 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:09:03.925992    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3c174ff-aad4-4543-9c30-0e9284a34da3" path="/var/lib/kubelet/pods/d3c174ff-aad4-4543-9c30-0e9284a34da3/volumes"
Sep 11 21:10:07.024659 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:07.024626    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hnblw"]
Sep 11 21:10:07.025155 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:07.024895    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c174ff-aad4-4543-9c30-0e9284a34da3" containerName="console"
Sep 11 21:10:07.025155 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:07.024908    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c174ff-aad4-4543-9c30-0e9284a34da3" containerName="console"
Sep 11 21:10:07.025155 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:07.024954    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c174ff-aad4-4543-9c30-0e9284a34da3" containerName="console"
Sep 11 21:10:07.027390 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:07.027375    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hnblw"
Sep 11 21:10:07.029550 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:07.029531    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:10:07.029662 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:07.029533    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-5ckb4\""
Sep 11 21:10:07.030167 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:07.030153    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 21:10:07.036283 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:07.036261    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hnblw"]
Sep 11 21:10:07.149249 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:07.149207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5cxb\" (UniqueName: \"kubernetes.io/projected/d01dd788-065f-4a2d-bf22-bdefe71c6e3d-kube-api-access-p5cxb\") pod \"openshift-lws-operator-fd8ccff4c-hnblw\" (UID: \"d01dd788-065f-4a2d-bf22-bdefe71c6e3d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hnblw"
Sep 11 21:10:07.149420 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:07.149311    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d01dd788-065f-4a2d-bf22-bdefe71c6e3d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hnblw\" (UID: \"d01dd788-065f-4a2d-bf22-bdefe71c6e3d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hnblw"
Sep 11 21:10:07.250120 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:07.250088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d01dd788-065f-4a2d-bf22-bdefe71c6e3d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hnblw\" (UID: \"d01dd788-065f-4a2d-bf22-bdefe71c6e3d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hnblw"
Sep 11 21:10:07.250202 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:07.250130    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5cxb\" (UniqueName: \"kubernetes.io/projected/d01dd788-065f-4a2d-bf22-bdefe71c6e3d-kube-api-access-p5cxb\") pod \"openshift-lws-operator-fd8ccff4c-hnblw\" (UID: \"d01dd788-065f-4a2d-bf22-bdefe71c6e3d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hnblw"
Sep 11 21:10:07.250538 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:07.250517    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d01dd788-065f-4a2d-bf22-bdefe71c6e3d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hnblw\" (UID: \"d01dd788-065f-4a2d-bf22-bdefe71c6e3d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hnblw"
Sep 11 21:10:07.258747 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:07.258724    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5cxb\" (UniqueName: \"kubernetes.io/projected/d01dd788-065f-4a2d-bf22-bdefe71c6e3d-kube-api-access-p5cxb\") pod \"openshift-lws-operator-fd8ccff4c-hnblw\" (UID: \"d01dd788-065f-4a2d-bf22-bdefe71c6e3d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hnblw"
Sep 11 21:10:07.335970 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:07.335891    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hnblw"
Sep 11 21:10:07.606759 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:07.606664    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hnblw"]
Sep 11 21:10:07.610260 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:10:07.610235    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd01dd788_065f_4a2d_bf22_bdefe71c6e3d.slice/crio-6e76e85c0cc7e39f812ce0ff79964c89612b46f775cb2728512ef5586d17122d WatchSource:0}: Error finding container 6e76e85c0cc7e39f812ce0ff79964c89612b46f775cb2728512ef5586d17122d: Status 404 returned error can't find the container with id 6e76e85c0cc7e39f812ce0ff79964c89612b46f775cb2728512ef5586d17122d
Sep 11 21:10:07.938895 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:07.938864    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hnblw" event={"ID":"d01dd788-065f-4a2d-bf22-bdefe71c6e3d","Type":"ContainerStarted","Data":"6e76e85c0cc7e39f812ce0ff79964c89612b46f775cb2728512ef5586d17122d"}
Sep 11 21:10:10.952039 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:10.951999    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hnblw" event={"ID":"d01dd788-065f-4a2d-bf22-bdefe71c6e3d","Type":"ContainerStarted","Data":"2ceb723809d29f609eecf15c7413ac25ae7283aeb32fa6557b1d450fda7705e9"}
Sep 11 21:10:10.970139 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:10.970090    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hnblw" podStartSLOduration=0.848664136 podStartE2EDuration="3.970074367s" totalImagesPullingTime="3.121410231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:10:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:10:10.968911763 +0000 UTC m=+277.672881053" watchObservedRunningTime="2026-09-11 21:10:10.970074367 +0000 UTC m=+277.674043652"
Sep 11 21:10:26.156090 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.156049    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2"]
Sep 11 21:10:26.159096 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.159078    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2"
Sep 11 21:10:26.162242 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.162223    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 21:10:26.162525 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.162506    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 21:10:26.162600 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.162582    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7vdb9\""
Sep 11 21:10:26.163073 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.163059    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 21:10:26.163183 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.163168    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 21:10:26.185566 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.185539    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2"]
Sep 11 21:10:26.285875 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.285841    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnpwj\" (UniqueName: \"kubernetes.io/projected/d6d22dfa-1693-49d0-b4d6-087a181b3b57-kube-api-access-nnpwj\") pod \"opendatahub-operator-controller-manager-8457b95c77-b4xt2\" (UID: \"d6d22dfa-1693-49d0-b4d6-087a181b3b57\") " pod="opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2"
Sep 11 21:10:26.286057 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.285900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d6d22dfa-1693-49d0-b4d6-087a181b3b57-manifests\") pod \"opendatahub-operator-controller-manager-8457b95c77-b4xt2\" (UID: \"d6d22dfa-1693-49d0-b4d6-087a181b3b57\") " pod="opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2"
Sep 11 21:10:26.286057 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.285940    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6d22dfa-1693-49d0-b4d6-087a181b3b57-tmp\") pod \"opendatahub-operator-controller-manager-8457b95c77-b4xt2\" (UID: \"d6d22dfa-1693-49d0-b4d6-087a181b3b57\") " pod="opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2"
Sep 11 21:10:26.286057 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.286010    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d6d22dfa-1693-49d0-b4d6-087a181b3b57-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8457b95c77-b4xt2\" (UID: \"d6d22dfa-1693-49d0-b4d6-087a181b3b57\") " pod="opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2"
Sep 11 21:10:26.286173 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.286106    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6d22dfa-1693-49d0-b4d6-087a181b3b57-webhook-cert\") pod \"opendatahub-operator-controller-manager-8457b95c77-b4xt2\" (UID: \"d6d22dfa-1693-49d0-b4d6-087a181b3b57\") " pod="opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2"
Sep 11 21:10:26.387013 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.386976    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6d22dfa-1693-49d0-b4d6-087a181b3b57-webhook-cert\") pod \"opendatahub-operator-controller-manager-8457b95c77-b4xt2\" (UID: \"d6d22dfa-1693-49d0-b4d6-087a181b3b57\") " pod="opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2"
Sep 11 21:10:26.387013 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.387021    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnpwj\" (UniqueName: \"kubernetes.io/projected/d6d22dfa-1693-49d0-b4d6-087a181b3b57-kube-api-access-nnpwj\") pod \"opendatahub-operator-controller-manager-8457b95c77-b4xt2\" (UID: \"d6d22dfa-1693-49d0-b4d6-087a181b3b57\") " pod="opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2"
Sep 11 21:10:26.387250 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.387115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d6d22dfa-1693-49d0-b4d6-087a181b3b57-manifests\") pod \"opendatahub-operator-controller-manager-8457b95c77-b4xt2\" (UID: \"d6d22dfa-1693-49d0-b4d6-087a181b3b57\") " pod="opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2"
Sep 11 21:10:26.387250 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.387151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6d22dfa-1693-49d0-b4d6-087a181b3b57-tmp\") pod \"opendatahub-operator-controller-manager-8457b95c77-b4xt2\" (UID: \"d6d22dfa-1693-49d0-b4d6-087a181b3b57\") " pod="opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2"
Sep 11 21:10:26.387250 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.387176    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d6d22dfa-1693-49d0-b4d6-087a181b3b57-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8457b95c77-b4xt2\" (UID: \"d6d22dfa-1693-49d0-b4d6-087a181b3b57\") " pod="opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2"
Sep 11 21:10:26.387639 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.387606    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d6d22dfa-1693-49d0-b4d6-087a181b3b57-manifests\") pod \"opendatahub-operator-controller-manager-8457b95c77-b4xt2\" (UID: \"d6d22dfa-1693-49d0-b4d6-087a181b3b57\") " pod="opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2"
Sep 11 21:10:26.389266 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.389240    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6d22dfa-1693-49d0-b4d6-087a181b3b57-tmp\") pod \"opendatahub-operator-controller-manager-8457b95c77-b4xt2\" (UID: \"d6d22dfa-1693-49d0-b4d6-087a181b3b57\") " pod="opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2"
Sep 11 21:10:26.389507 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.389486    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6d22dfa-1693-49d0-b4d6-087a181b3b57-webhook-cert\") pod \"opendatahub-operator-controller-manager-8457b95c77-b4xt2\" (UID: \"d6d22dfa-1693-49d0-b4d6-087a181b3b57\") " pod="opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2"
Sep 11 21:10:26.389575 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.389486    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d6d22dfa-1693-49d0-b4d6-087a181b3b57-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8457b95c77-b4xt2\" (UID: \"d6d22dfa-1693-49d0-b4d6-087a181b3b57\") " pod="opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2"
Sep 11 21:10:26.396756 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.396723    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnpwj\" (UniqueName: \"kubernetes.io/projected/d6d22dfa-1693-49d0-b4d6-087a181b3b57-kube-api-access-nnpwj\") pod \"opendatahub-operator-controller-manager-8457b95c77-b4xt2\" (UID: \"d6d22dfa-1693-49d0-b4d6-087a181b3b57\") " pod="opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2"
Sep 11 21:10:26.468657 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.468617    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2"
Sep 11 21:10:26.750417 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:26.750328    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2"]
Sep 11 21:10:26.754482 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:10:26.754455    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d22dfa_1693_49d0_b4d6_087a181b3b57.slice/crio-5544bf1a650ceda0c19070463e5e4bf6f2428c8049514468ce612b718bcf2330 WatchSource:0}: Error finding container 5544bf1a650ceda0c19070463e5e4bf6f2428c8049514468ce612b718bcf2330: Status 404 returned error can't find the container with id 5544bf1a650ceda0c19070463e5e4bf6f2428c8049514468ce612b718bcf2330
Sep 11 21:10:27.009216 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:27.009131    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2" event={"ID":"d6d22dfa-1693-49d0-b4d6-087a181b3b57","Type":"ContainerStarted","Data":"5544bf1a650ceda0c19070463e5e4bf6f2428c8049514468ce612b718bcf2330"}
Sep 11 21:10:30.023600 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:30.023516    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2" event={"ID":"d6d22dfa-1693-49d0-b4d6-087a181b3b57","Type":"ContainerStarted","Data":"d7ed9915204ea304961019f482663e4c0e0fa27d00cbc7dc6c5bfae5c87febb5"}
Sep 11 21:10:31.028977 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:31.028945    2578 generic.go:356] "Generic (PLEG): container finished" podID="d6d22dfa-1693-49d0-b4d6-087a181b3b57" containerID="d7ed9915204ea304961019f482663e4c0e0fa27d00cbc7dc6c5bfae5c87febb5" exitCode=0
Sep 11 21:10:31.029357 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:31.029009    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2" event={"ID":"d6d22dfa-1693-49d0-b4d6-087a181b3b57","Type":"ContainerDied","Data":"d7ed9915204ea304961019f482663e4c0e0fa27d00cbc7dc6c5bfae5c87febb5"}
Sep 11 21:10:32.041455 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:32.041419    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2" event={"ID":"d6d22dfa-1693-49d0-b4d6-087a181b3b57","Type":"ContainerStarted","Data":"cc8a66412be8202d4f5e71c7da6a61815bae42e8c4a94aeecdae5c1423180061"}
Sep 11 21:10:32.041908 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:32.041579    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2"
Sep 11 21:10:32.066163 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:32.066093    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2" podStartSLOduration=1.665638078 podStartE2EDuration="6.066072415s" totalImagesPullingTime="3.356729365s" totalInitContainerRuntime="1.043704972s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:10:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:10:32.064499778 +0000 UTC m=+298.768469068" watchObservedRunningTime="2026-09-11 21:10:32.066072415 +0000 UTC m=+298.770041706"
Sep 11 21:10:33.871629 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:33.871604    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 21:10:43.047655 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:43.047623    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8457b95c77-b4xt2"
Sep 11 21:10:45.790738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:45.790703    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-677bd9b6b8-p99rw"]
Sep 11 21:10:45.877081 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:45.877038    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-677bd9b6b8-p99rw"]
Sep 11 21:10:45.877268 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:45.877150    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-677bd9b6b8-p99rw"
Sep 11 21:10:45.881831 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:45.881802    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 21:10:45.882002 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:45.881939    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jncg9\""
Sep 11 21:10:45.882561 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:45.882544    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 21:10:45.882788 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:45.882773    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 21:10:45.882886 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:45.882806    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 21:10:45.920077 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:45.920047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tx8x\" (UniqueName: \"kubernetes.io/projected/6c8aa1b3-5d8a-4bce-ad61-b2b79c5c1a88-kube-api-access-6tx8x\") pod \"kube-auth-proxy-677bd9b6b8-p99rw\" (UID: \"6c8aa1b3-5d8a-4bce-ad61-b2b79c5c1a88\") " pod="openshift-ingress/kube-auth-proxy-677bd9b6b8-p99rw"
Sep 11 21:10:45.920242 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:45.920086    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c8aa1b3-5d8a-4bce-ad61-b2b79c5c1a88-tls-certs\") pod \"kube-auth-proxy-677bd9b6b8-p99rw\" (UID: \"6c8aa1b3-5d8a-4bce-ad61-b2b79c5c1a88\") " pod="openshift-ingress/kube-auth-proxy-677bd9b6b8-p99rw"
Sep 11 21:10:45.920242 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:45.920168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c8aa1b3-5d8a-4bce-ad61-b2b79c5c1a88-tmp\") pod \"kube-auth-proxy-677bd9b6b8-p99rw\" (UID: \"6c8aa1b3-5d8a-4bce-ad61-b2b79c5c1a88\") " pod="openshift-ingress/kube-auth-proxy-677bd9b6b8-p99rw"
Sep 11 21:10:46.021235 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:46.021200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c8aa1b3-5d8a-4bce-ad61-b2b79c5c1a88-tls-certs\") pod \"kube-auth-proxy-677bd9b6b8-p99rw\" (UID: \"6c8aa1b3-5d8a-4bce-ad61-b2b79c5c1a88\") " pod="openshift-ingress/kube-auth-proxy-677bd9b6b8-p99rw"
Sep 11 21:10:46.021420 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:46.021353    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c8aa1b3-5d8a-4bce-ad61-b2b79c5c1a88-tmp\") pod \"kube-auth-proxy-677bd9b6b8-p99rw\" (UID: \"6c8aa1b3-5d8a-4bce-ad61-b2b79c5c1a88\") " pod="openshift-ingress/kube-auth-proxy-677bd9b6b8-p99rw"
Sep 11 21:10:46.021493 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:46.021428    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tx8x\" (UniqueName: \"kubernetes.io/projected/6c8aa1b3-5d8a-4bce-ad61-b2b79c5c1a88-kube-api-access-6tx8x\") pod \"kube-auth-proxy-677bd9b6b8-p99rw\" (UID: \"6c8aa1b3-5d8a-4bce-ad61-b2b79c5c1a88\") " pod="openshift-ingress/kube-auth-proxy-677bd9b6b8-p99rw"
Sep 11 21:10:46.023606 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:46.023576    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c8aa1b3-5d8a-4bce-ad61-b2b79c5c1a88-tmp\") pod \"kube-auth-proxy-677bd9b6b8-p99rw\" (UID: \"6c8aa1b3-5d8a-4bce-ad61-b2b79c5c1a88\") " pod="openshift-ingress/kube-auth-proxy-677bd9b6b8-p99rw"
Sep 11 21:10:46.023713 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:46.023701    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c8aa1b3-5d8a-4bce-ad61-b2b79c5c1a88-tls-certs\") pod \"kube-auth-proxy-677bd9b6b8-p99rw\" (UID: \"6c8aa1b3-5d8a-4bce-ad61-b2b79c5c1a88\") " pod="openshift-ingress/kube-auth-proxy-677bd9b6b8-p99rw"
Sep 11 21:10:46.037162 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:46.037139    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tx8x\" (UniqueName: \"kubernetes.io/projected/6c8aa1b3-5d8a-4bce-ad61-b2b79c5c1a88-kube-api-access-6tx8x\") pod \"kube-auth-proxy-677bd9b6b8-p99rw\" (UID: \"6c8aa1b3-5d8a-4bce-ad61-b2b79c5c1a88\") " pod="openshift-ingress/kube-auth-proxy-677bd9b6b8-p99rw"
Sep 11 21:10:46.185920 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:46.185889    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-677bd9b6b8-p99rw"
Sep 11 21:10:46.493964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:46.493880    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-677bd9b6b8-p99rw"]
Sep 11 21:10:46.502345 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:10:46.502312    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8aa1b3_5d8a_4bce_ad61_b2b79c5c1a88.slice/crio-0862d1fbd4f1be493ed3b3f3a6d829e4be9275004f09ee94f9b6f4b467f02a0b WatchSource:0}: Error finding container 0862d1fbd4f1be493ed3b3f3a6d829e4be9275004f09ee94f9b6f4b467f02a0b: Status 404 returned error can't find the container with id 0862d1fbd4f1be493ed3b3f3a6d829e4be9275004f09ee94f9b6f4b467f02a0b
Sep 11 21:10:46.504154 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:46.504140    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:10:46.906242 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:46.906166    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-65cc4f94dc-m467m"]
Sep 11 21:10:46.909484 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:46.909461    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65cc4f94dc-m467m"
Sep 11 21:10:46.914251 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:46.914120    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-9qf8r\""
Sep 11 21:10:46.914251 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:46.914132    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 21:10:46.914251 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:46.914140    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 21:10:46.914251 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:46.914159    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 21:10:46.927975 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:46.927952    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65cc4f94dc-m467m"]
Sep 11 21:10:46.929134 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:46.929112    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/507d53e5-22b0-4a5a-99f4-ad9e39456e29-cert\") pod \"lws-controller-manager-65cc4f94dc-m467m\" (UID: \"507d53e5-22b0-4a5a-99f4-ad9e39456e29\") " pod="openshift-lws-operator/lws-controller-manager-65cc4f94dc-m467m"
Sep 11 21:10:46.929228 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:46.929148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmzxv\" (UniqueName: \"kubernetes.io/projected/507d53e5-22b0-4a5a-99f4-ad9e39456e29-kube-api-access-dmzxv\") pod \"lws-controller-manager-65cc4f94dc-m467m\" (UID: \"507d53e5-22b0-4a5a-99f4-ad9e39456e29\") " pod="openshift-lws-operator/lws-controller-manager-65cc4f94dc-m467m"
Sep 11 21:10:46.929228 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:46.929175    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/507d53e5-22b0-4a5a-99f4-ad9e39456e29-metrics-cert\") pod \"lws-controller-manager-65cc4f94dc-m467m\" (UID: \"507d53e5-22b0-4a5a-99f4-ad9e39456e29\") " pod="openshift-lws-operator/lws-controller-manager-65cc4f94dc-m467m"
Sep 11 21:10:46.929317 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:46.929241    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/507d53e5-22b0-4a5a-99f4-ad9e39456e29-manager-config\") pod \"lws-controller-manager-65cc4f94dc-m467m\" (UID: \"507d53e5-22b0-4a5a-99f4-ad9e39456e29\") " pod="openshift-lws-operator/lws-controller-manager-65cc4f94dc-m467m"
Sep 11 21:10:47.029793 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:47.029756    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/507d53e5-22b0-4a5a-99f4-ad9e39456e29-cert\") pod \"lws-controller-manager-65cc4f94dc-m467m\" (UID: \"507d53e5-22b0-4a5a-99f4-ad9e39456e29\") " pod="openshift-lws-operator/lws-controller-manager-65cc4f94dc-m467m"
Sep 11 21:10:47.029793 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:47.029806    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmzxv\" (UniqueName: \"kubernetes.io/projected/507d53e5-22b0-4a5a-99f4-ad9e39456e29-kube-api-access-dmzxv\") pod \"lws-controller-manager-65cc4f94dc-m467m\" (UID: \"507d53e5-22b0-4a5a-99f4-ad9e39456e29\") " pod="openshift-lws-operator/lws-controller-manager-65cc4f94dc-m467m"
Sep 11 21:10:47.030074 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:47.029854    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/507d53e5-22b0-4a5a-99f4-ad9e39456e29-metrics-cert\") pod \"lws-controller-manager-65cc4f94dc-m467m\" (UID: \"507d53e5-22b0-4a5a-99f4-ad9e39456e29\") " pod="openshift-lws-operator/lws-controller-manager-65cc4f94dc-m467m"
Sep 11 21:10:47.030074 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:47.029907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/507d53e5-22b0-4a5a-99f4-ad9e39456e29-manager-config\") pod \"lws-controller-manager-65cc4f94dc-m467m\" (UID: \"507d53e5-22b0-4a5a-99f4-ad9e39456e29\") " pod="openshift-lws-operator/lws-controller-manager-65cc4f94dc-m467m"
Sep 11 21:10:47.030737 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:47.030709    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/507d53e5-22b0-4a5a-99f4-ad9e39456e29-manager-config\") pod \"lws-controller-manager-65cc4f94dc-m467m\" (UID: \"507d53e5-22b0-4a5a-99f4-ad9e39456e29\") " pod="openshift-lws-operator/lws-controller-manager-65cc4f94dc-m467m"
Sep 11 21:10:47.033278 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:47.033235    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/507d53e5-22b0-4a5a-99f4-ad9e39456e29-cert\") pod \"lws-controller-manager-65cc4f94dc-m467m\" (UID: \"507d53e5-22b0-4a5a-99f4-ad9e39456e29\") " pod="openshift-lws-operator/lws-controller-manager-65cc4f94dc-m467m"
Sep 11 21:10:47.033278 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:47.033235    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/507d53e5-22b0-4a5a-99f4-ad9e39456e29-metrics-cert\") pod \"lws-controller-manager-65cc4f94dc-m467m\" (UID: \"507d53e5-22b0-4a5a-99f4-ad9e39456e29\") " pod="openshift-lws-operator/lws-controller-manager-65cc4f94dc-m467m"
Sep 11 21:10:47.045346 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:47.045320    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmzxv\" (UniqueName: \"kubernetes.io/projected/507d53e5-22b0-4a5a-99f4-ad9e39456e29-kube-api-access-dmzxv\") pod \"lws-controller-manager-65cc4f94dc-m467m\" (UID: \"507d53e5-22b0-4a5a-99f4-ad9e39456e29\") " pod="openshift-lws-operator/lws-controller-manager-65cc4f94dc-m467m"
Sep 11 21:10:47.095405 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:47.095359    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-677bd9b6b8-p99rw" event={"ID":"6c8aa1b3-5d8a-4bce-ad61-b2b79c5c1a88","Type":"ContainerStarted","Data":"0862d1fbd4f1be493ed3b3f3a6d829e4be9275004f09ee94f9b6f4b467f02a0b"}
Sep 11 21:10:47.219340 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:47.219299    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65cc4f94dc-m467m"
Sep 11 21:10:47.527176 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:47.527097    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65cc4f94dc-m467m"]
Sep 11 21:10:47.530469 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:10:47.530441    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507d53e5_22b0_4a5a_99f4_ad9e39456e29.slice/crio-7d446b45f45da8806c5b272dc9ea6092a5da8a7eaab7f0d7cb3c7bd91b1a1c58 WatchSource:0}: Error finding container 7d446b45f45da8806c5b272dc9ea6092a5da8a7eaab7f0d7cb3c7bd91b1a1c58: Status 404 returned error can't find the container with id 7d446b45f45da8806c5b272dc9ea6092a5da8a7eaab7f0d7cb3c7bd91b1a1c58
Sep 11 21:10:48.100565 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:48.100530    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65cc4f94dc-m467m" event={"ID":"507d53e5-22b0-4a5a-99f4-ad9e39456e29","Type":"ContainerStarted","Data":"7d446b45f45da8806c5b272dc9ea6092a5da8a7eaab7f0d7cb3c7bd91b1a1c58"}
Sep 11 21:10:50.110980 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:50.110940    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-677bd9b6b8-p99rw" event={"ID":"6c8aa1b3-5d8a-4bce-ad61-b2b79c5c1a88","Type":"ContainerStarted","Data":"c58915c4e7d9091a89b666f6e50111c91188c2d770eb887f9fbb2673b05fbc3f"}
Sep 11 21:10:50.112526 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:50.112502    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65cc4f94dc-m467m" event={"ID":"507d53e5-22b0-4a5a-99f4-ad9e39456e29","Type":"ContainerStarted","Data":"a020c54cea80978854e5c720932e65ae509bd3a13ec8172fa36b4205f0bae7db"}
Sep 11 21:10:50.112647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:50.112579    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-65cc4f94dc-m467m"
Sep 11 21:10:50.155726 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:50.155670    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-677bd9b6b8-p99rw" podStartSLOduration=2.465649991 podStartE2EDuration="5.155653075s" totalImagesPullingTime="2.690003084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:10:50.134399234 +0000 UTC m=+316.838368525" watchObservedRunningTime="2026-09-11 21:10:50.155653075 +0000 UTC m=+316.859622364"
Sep 11 21:10:50.156041 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:10:50.156007    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-65cc4f94dc-m467m" podStartSLOduration=1.723052032 podStartE2EDuration="4.156000873s" totalImagesPullingTime="2.432948841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:10:50.154404763 +0000 UTC m=+316.858374062" watchObservedRunningTime="2026-09-11 21:10:50.156000873 +0000 UTC m=+316.859970163"
Sep 11 21:11:01.118632 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:11:01.118602    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-65cc4f94dc-m467m"
Sep 11 21:12:24.283716 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:12:24.283680    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-tl8b4"]
Sep 11 21:12:24.286732 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:12:24.286716    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tl8b4"
Sep 11 21:12:24.289005 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:12:24.288983    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 11 21:12:24.289123 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:12:24.289109    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 21:12:24.290031 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:12:24.289991    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 11 21:12:24.290031 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:12:24.289995    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 21:12:24.290249 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:12:24.290034    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hk78l\""
Sep 11 21:12:24.296248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:12:24.296223    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-tl8b4"]
Sep 11 21:12:24.343241 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:12:24.343211    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5cf77d2a-487f-45f5-bf80-ff3768504fa8-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-tl8b4\" (UID: \"5cf77d2a-487f-45f5-bf80-ff3768504fa8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tl8b4"
Sep 11 21:12:24.343241 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:12:24.343247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5cf77d2a-487f-45f5-bf80-ff3768504fa8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-tl8b4\" (UID: \"5cf77d2a-487f-45f5-bf80-ff3768504fa8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tl8b4"
Sep 11 21:12:24.343449 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:12:24.343267    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qph4p\" (UniqueName: \"kubernetes.io/projected/5cf77d2a-487f-45f5-bf80-ff3768504fa8-kube-api-access-qph4p\") pod \"kuadrant-console-plugin-fbdd4f445-tl8b4\" (UID: \"5cf77d2a-487f-45f5-bf80-ff3768504fa8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tl8b4"
Sep 11 21:12:24.444183 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:12:24.444141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5cf77d2a-487f-45f5-bf80-ff3768504fa8-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-tl8b4\" (UID: \"5cf77d2a-487f-45f5-bf80-ff3768504fa8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tl8b4"
Sep 11 21:12:24.444183 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:12:24.444189    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5cf77d2a-487f-45f5-bf80-ff3768504fa8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-tl8b4\" (UID: \"5cf77d2a-487f-45f5-bf80-ff3768504fa8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tl8b4"
Sep 11 21:12:24.444459 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:12:24.444218    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qph4p\" (UniqueName: \"kubernetes.io/projected/5cf77d2a-487f-45f5-bf80-ff3768504fa8-kube-api-access-qph4p\") pod \"kuadrant-console-plugin-fbdd4f445-tl8b4\" (UID: \"5cf77d2a-487f-45f5-bf80-ff3768504fa8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tl8b4"
Sep 11 21:12:24.444459 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:12:24.444350    2578 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 11 21:12:24.444459 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:12:24.444417    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cf77d2a-487f-45f5-bf80-ff3768504fa8-plugin-serving-cert podName:5cf77d2a-487f-45f5-bf80-ff3768504fa8 nodeName:}" failed. No retries permitted until 2026-09-11 21:12:24.944399723 +0000 UTC m=+411.648369005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/5cf77d2a-487f-45f5-bf80-ff3768504fa8-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-tl8b4" (UID: "5cf77d2a-487f-45f5-bf80-ff3768504fa8") : secret "plugin-serving-cert" not found
Sep 11 21:12:24.444920 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:12:24.444899    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5cf77d2a-487f-45f5-bf80-ff3768504fa8-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-tl8b4\" (UID: \"5cf77d2a-487f-45f5-bf80-ff3768504fa8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tl8b4"
Sep 11 21:12:24.459544 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:12:24.459515    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qph4p\" (UniqueName: \"kubernetes.io/projected/5cf77d2a-487f-45f5-bf80-ff3768504fa8-kube-api-access-qph4p\") pod \"kuadrant-console-plugin-fbdd4f445-tl8b4\" (UID: \"5cf77d2a-487f-45f5-bf80-ff3768504fa8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tl8b4"
Sep 11 21:12:24.947941 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:12:24.947898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5cf77d2a-487f-45f5-bf80-ff3768504fa8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-tl8b4\" (UID: \"5cf77d2a-487f-45f5-bf80-ff3768504fa8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tl8b4"
Sep 11 21:12:24.950301 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:12:24.950267    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5cf77d2a-487f-45f5-bf80-ff3768504fa8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-tl8b4\" (UID: \"5cf77d2a-487f-45f5-bf80-ff3768504fa8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tl8b4"
Sep 11 21:12:25.196113 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:12:25.196076    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tl8b4"
Sep 11 21:12:25.468238 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:12:25.468164    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-tl8b4"]
Sep 11 21:12:25.471387 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:12:25.471359    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5cf77d2a_487f_45f5_bf80_ff3768504fa8.slice/crio-35ff9b35d1cd196e4c610aa095aff2a4a8f15fb55a46c85db48865d13012a34b WatchSource:0}: Error finding container 35ff9b35d1cd196e4c610aa095aff2a4a8f15fb55a46c85db48865d13012a34b: Status 404 returned error can't find the container with id 35ff9b35d1cd196e4c610aa095aff2a4a8f15fb55a46c85db48865d13012a34b
Sep 11 21:12:26.451041 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:12:26.450999    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tl8b4" event={"ID":"5cf77d2a-487f-45f5-bf80-ff3768504fa8","Type":"ContainerStarted","Data":"35ff9b35d1cd196e4c610aa095aff2a4a8f15fb55a46c85db48865d13012a34b"}
Sep 11 21:12:30.467797 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:12:30.467757    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tl8b4" event={"ID":"5cf77d2a-487f-45f5-bf80-ff3768504fa8","Type":"ContainerStarted","Data":"ae8aeb47844ea98440a2a703775b0a0c9bc201ded7de78c87968346e0c94755c"}
Sep 11 21:12:30.486805 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:12:30.486748    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tl8b4" podStartSLOduration=2.073803914 podStartE2EDuration="6.486730553s" totalImagesPullingTime="4.412926639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:12:30.48635597 +0000 UTC m=+417.190325263" watchObservedRunningTime="2026-09-11 21:12:30.486730553 +0000 UTC m=+417.190699846"
Sep 11 21:13:09.608295 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:09.608262    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:13:09.613905 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:09.613884    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"
Sep 11 21:13:09.616602 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:09.616574    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 21:13:09.622260 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:09.622235    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:13:09.659313 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:09.659275    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:13:09.696208 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:09.696169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b2f8ff99-9d97-4ed7-9e33-3b186aa765ff-config-file\") pod \"limitador-limitador-84994f7ffc-wq4ps\" (UID: \"b2f8ff99-9d97-4ed7-9e33-3b186aa765ff\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"
Sep 11 21:13:09.696402 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:09.696227    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phxq2\" (UniqueName: \"kubernetes.io/projected/b2f8ff99-9d97-4ed7-9e33-3b186aa765ff-kube-api-access-phxq2\") pod \"limitador-limitador-84994f7ffc-wq4ps\" (UID: \"b2f8ff99-9d97-4ed7-9e33-3b186aa765ff\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"
Sep 11 21:13:09.797145 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:09.797105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phxq2\" (UniqueName: \"kubernetes.io/projected/b2f8ff99-9d97-4ed7-9e33-3b186aa765ff-kube-api-access-phxq2\") pod \"limitador-limitador-84994f7ffc-wq4ps\" (UID: \"b2f8ff99-9d97-4ed7-9e33-3b186aa765ff\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"
Sep 11 21:13:09.797300 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:09.797181    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b2f8ff99-9d97-4ed7-9e33-3b186aa765ff-config-file\") pod \"limitador-limitador-84994f7ffc-wq4ps\" (UID: \"b2f8ff99-9d97-4ed7-9e33-3b186aa765ff\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"
Sep 11 21:13:09.797769 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:09.797752    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b2f8ff99-9d97-4ed7-9e33-3b186aa765ff-config-file\") pod \"limitador-limitador-84994f7ffc-wq4ps\" (UID: \"b2f8ff99-9d97-4ed7-9e33-3b186aa765ff\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"
Sep 11 21:13:09.805982 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:09.805959    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phxq2\" (UniqueName: \"kubernetes.io/projected/b2f8ff99-9d97-4ed7-9e33-3b186aa765ff-kube-api-access-phxq2\") pod \"limitador-limitador-84994f7ffc-wq4ps\" (UID: \"b2f8ff99-9d97-4ed7-9e33-3b186aa765ff\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"
Sep 11 21:13:09.924185 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:09.924161    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"
Sep 11 21:13:10.228667 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:10.228554    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-fwl2l"]
Sep 11 21:13:10.231780 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:13:10.231755    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f8ff99_9d97_4ed7_9e33_3b186aa765ff.slice/crio-9165bf664b4d3c502d3e5caa39cce760fbd7241f28e90a76bf2ccebe203eb986 WatchSource:0}: Error finding container 9165bf664b4d3c502d3e5caa39cce760fbd7241f28e90a76bf2ccebe203eb986: Status 404 returned error can't find the container with id 9165bf664b4d3c502d3e5caa39cce760fbd7241f28e90a76bf2ccebe203eb986
Sep 11 21:13:10.239551 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:10.239526    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:13:10.239663 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:10.239624    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-fwl2l"
Sep 11 21:13:10.242465 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:10.242448    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mxp7h\""
Sep 11 21:13:10.246492 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:10.246468    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-fwl2l"]
Sep 11 21:13:10.301480 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:10.301437    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg6hw\" (UniqueName: \"kubernetes.io/projected/9bdf2ae2-55c6-4660-9fb2-64a9c46efbb2-kube-api-access-cg6hw\") pod \"authorino-7bcfd8f9cd-fwl2l\" (UID: \"9bdf2ae2-55c6-4660-9fb2-64a9c46efbb2\") " pod="kuadrant-system/authorino-7bcfd8f9cd-fwl2l"
Sep 11 21:13:10.402594 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:10.402561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg6hw\" (UniqueName: \"kubernetes.io/projected/9bdf2ae2-55c6-4660-9fb2-64a9c46efbb2-kube-api-access-cg6hw\") pod \"authorino-7bcfd8f9cd-fwl2l\" (UID: \"9bdf2ae2-55c6-4660-9fb2-64a9c46efbb2\") " pod="kuadrant-system/authorino-7bcfd8f9cd-fwl2l"
Sep 11 21:13:10.411123 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:10.411092    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg6hw\" (UniqueName: \"kubernetes.io/projected/9bdf2ae2-55c6-4660-9fb2-64a9c46efbb2-kube-api-access-cg6hw\") pod \"authorino-7bcfd8f9cd-fwl2l\" (UID: \"9bdf2ae2-55c6-4660-9fb2-64a9c46efbb2\") " pod="kuadrant-system/authorino-7bcfd8f9cd-fwl2l"
Sep 11 21:13:10.450666 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:10.450630    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-5flf2"]
Sep 11 21:13:10.453603 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:10.453583    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-5flf2"
Sep 11 21:13:10.467410 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:10.467383    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5flf2"]
Sep 11 21:13:10.550217 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:10.550138    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-fwl2l"
Sep 11 21:13:10.604376 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:10.604344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28nch\" (UniqueName: \"kubernetes.io/projected/b0d93c89-9d05-4aa9-bc56-24d9d5fec7b7-kube-api-access-28nch\") pod \"authorino-59c974bd48-5flf2\" (UID: \"b0d93c89-9d05-4aa9-bc56-24d9d5fec7b7\") " pod="kuadrant-system/authorino-59c974bd48-5flf2"
Sep 11 21:13:10.612252 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:10.612223    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-wq4ps" event={"ID":"b2f8ff99-9d97-4ed7-9e33-3b186aa765ff","Type":"ContainerStarted","Data":"9165bf664b4d3c502d3e5caa39cce760fbd7241f28e90a76bf2ccebe203eb986"}
Sep 11 21:13:10.704866 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:10.704807    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28nch\" (UniqueName: \"kubernetes.io/projected/b0d93c89-9d05-4aa9-bc56-24d9d5fec7b7-kube-api-access-28nch\") pod \"authorino-59c974bd48-5flf2\" (UID: \"b0d93c89-9d05-4aa9-bc56-24d9d5fec7b7\") " pod="kuadrant-system/authorino-59c974bd48-5flf2"
Sep 11 21:13:10.713816 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:10.713783    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28nch\" (UniqueName: \"kubernetes.io/projected/b0d93c89-9d05-4aa9-bc56-24d9d5fec7b7-kube-api-access-28nch\") pod \"authorino-59c974bd48-5flf2\" (UID: \"b0d93c89-9d05-4aa9-bc56-24d9d5fec7b7\") " pod="kuadrant-system/authorino-59c974bd48-5flf2"
Sep 11 21:13:10.768062 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:10.768017    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-5flf2"
Sep 11 21:13:10.819777 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:10.819643    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-fwl2l"]
Sep 11 21:13:10.825447 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:13:10.825379    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bdf2ae2_55c6_4660_9fb2_64a9c46efbb2.slice/crio-2f6d83295ab3a9d8afcf58244742b6dd29996eddf388eec6d221a3eb5aef90f0 WatchSource:0}: Error finding container 2f6d83295ab3a9d8afcf58244742b6dd29996eddf388eec6d221a3eb5aef90f0: Status 404 returned error can't find the container with id 2f6d83295ab3a9d8afcf58244742b6dd29996eddf388eec6d221a3eb5aef90f0
Sep 11 21:13:11.056509 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:11.056476    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5flf2"]
Sep 11 21:13:11.059636 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:13:11.059603    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0d93c89_9d05_4aa9_bc56_24d9d5fec7b7.slice/crio-fe479cec116bc49219a0d611b97a033f90c2a06d894b7b3e4c9f769e69369fb4 WatchSource:0}: Error finding container fe479cec116bc49219a0d611b97a033f90c2a06d894b7b3e4c9f769e69369fb4: Status 404 returned error can't find the container with id fe479cec116bc49219a0d611b97a033f90c2a06d894b7b3e4c9f769e69369fb4
Sep 11 21:13:11.618085 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:11.618040    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-fwl2l" event={"ID":"9bdf2ae2-55c6-4660-9fb2-64a9c46efbb2","Type":"ContainerStarted","Data":"2f6d83295ab3a9d8afcf58244742b6dd29996eddf388eec6d221a3eb5aef90f0"}
Sep 11 21:13:11.619378 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:11.619354    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-5flf2" event={"ID":"b0d93c89-9d05-4aa9-bc56-24d9d5fec7b7","Type":"ContainerStarted","Data":"fe479cec116bc49219a0d611b97a033f90c2a06d894b7b3e4c9f769e69369fb4"}
Sep 11 21:13:12.627522 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:12.627425    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-wq4ps" event={"ID":"b2f8ff99-9d97-4ed7-9e33-3b186aa765ff","Type":"ContainerStarted","Data":"d1552f53e3aee4c28bc6badd4335dbb10925827e42e1bbdb3afd8b1bd01c4b97"}
Sep 11 21:13:12.627995 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:12.627525    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"
Sep 11 21:13:12.647561 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:12.647286    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-wq4ps" podStartSLOduration=1.541849325 podStartE2EDuration="3.647261981s" totalImagesPullingTime="2.105412656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:13:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:13:12.645969987 +0000 UTC m=+459.349939278" watchObservedRunningTime="2026-09-11 21:13:12.647261981 +0000 UTC m=+459.351231273"
Sep 11 21:13:13.632692 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:13.632612    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-fwl2l" event={"ID":"9bdf2ae2-55c6-4660-9fb2-64a9c46efbb2","Type":"ContainerStarted","Data":"0f165fb58ec53283406c7e0c4bef3f01691935d1d87bd6f90900207fdc2df80d"}
Sep 11 21:13:13.647930 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:13.647884    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-fwl2l" podStartSLOduration=1.27706267 podStartE2EDuration="3.64786605s" totalImagesPullingTime="2.37080338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:13:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:13:13.647050915 +0000 UTC m=+460.351020206" watchObservedRunningTime="2026-09-11 21:13:13.64786605 +0000 UTC m=+460.351835337"
Sep 11 21:13:14.637632 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:14.637589    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-5flf2" event={"ID":"b0d93c89-9d05-4aa9-bc56-24d9d5fec7b7","Type":"ContainerStarted","Data":"86edda8da74c53bab7aab339b3dd4979945879543e6e24e6c395715e4595959a"}
Sep 11 21:13:14.655624 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:14.655566    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-5flf2" podStartSLOduration=1.879455017 podStartE2EDuration="4.655548745s" totalImagesPullingTime="2.776093728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:13:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:13:14.654208572 +0000 UTC m=+461.358177863" watchObservedRunningTime="2026-09-11 21:13:14.655548745 +0000 UTC m=+461.359518034"
Sep 11 21:13:14.684699 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:14.684665    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-fwl2l"]
Sep 11 21:13:15.641128 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:15.641083    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-fwl2l" podUID="9bdf2ae2-55c6-4660-9fb2-64a9c46efbb2" containerName="authorino" containerID="cri-o://0f165fb58ec53283406c7e0c4bef3f01691935d1d87bd6f90900207fdc2df80d" gracePeriod=30
Sep 11 21:13:15.881494 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:15.881470    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-fwl2l"
Sep 11 21:13:15.948080 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:15.948048    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cg6hw\" (UniqueName: \"kubernetes.io/projected/9bdf2ae2-55c6-4660-9fb2-64a9c46efbb2-kube-api-access-cg6hw\") pod \"9bdf2ae2-55c6-4660-9fb2-64a9c46efbb2\" (UID: \"9bdf2ae2-55c6-4660-9fb2-64a9c46efbb2\") "
Sep 11 21:13:15.950244 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:15.950216    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bdf2ae2-55c6-4660-9fb2-64a9c46efbb2-kube-api-access-cg6hw" (OuterVolumeSpecName: "kube-api-access-cg6hw") pod "9bdf2ae2-55c6-4660-9fb2-64a9c46efbb2" (UID: "9bdf2ae2-55c6-4660-9fb2-64a9c46efbb2"). InnerVolumeSpecName "kube-api-access-cg6hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:13:16.048816 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:16.048735    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg6hw\" (UniqueName: \"kubernetes.io/projected/9bdf2ae2-55c6-4660-9fb2-64a9c46efbb2-kube-api-access-cg6hw\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:13:16.646061 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:16.646027    2578 generic.go:356] "Generic (PLEG): container finished" podID="9bdf2ae2-55c6-4660-9fb2-64a9c46efbb2" containerID="0f165fb58ec53283406c7e0c4bef3f01691935d1d87bd6f90900207fdc2df80d" exitCode=0
Sep 11 21:13:16.646491 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:16.646070    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-fwl2l"
Sep 11 21:13:16.646491 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:16.646113    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-fwl2l" event={"ID":"9bdf2ae2-55c6-4660-9fb2-64a9c46efbb2","Type":"ContainerDied","Data":"0f165fb58ec53283406c7e0c4bef3f01691935d1d87bd6f90900207fdc2df80d"}
Sep 11 21:13:16.646491 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:16.646158    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-fwl2l" event={"ID":"9bdf2ae2-55c6-4660-9fb2-64a9c46efbb2","Type":"ContainerDied","Data":"2f6d83295ab3a9d8afcf58244742b6dd29996eddf388eec6d221a3eb5aef90f0"}
Sep 11 21:13:16.646491 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:16.646176    2578 scope.go:117] "RemoveContainer" containerID="0f165fb58ec53283406c7e0c4bef3f01691935d1d87bd6f90900207fdc2df80d"
Sep 11 21:13:16.654375 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:16.654354    2578 scope.go:117] "RemoveContainer" containerID="0f165fb58ec53283406c7e0c4bef3f01691935d1d87bd6f90900207fdc2df80d"
Sep 11 21:13:16.654693 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:13:16.654673    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f165fb58ec53283406c7e0c4bef3f01691935d1d87bd6f90900207fdc2df80d\": container with ID starting with 0f165fb58ec53283406c7e0c4bef3f01691935d1d87bd6f90900207fdc2df80d not found: ID does not exist" containerID="0f165fb58ec53283406c7e0c4bef3f01691935d1d87bd6f90900207fdc2df80d"
Sep 11 21:13:16.654776 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:16.654702    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f165fb58ec53283406c7e0c4bef3f01691935d1d87bd6f90900207fdc2df80d"} err="failed to get container status \"0f165fb58ec53283406c7e0c4bef3f01691935d1d87bd6f90900207fdc2df80d\": rpc error: code = NotFound desc = could not find container \"0f165fb58ec53283406c7e0c4bef3f01691935d1d87bd6f90900207fdc2df80d\": container with ID starting with 0f165fb58ec53283406c7e0c4bef3f01691935d1d87bd6f90900207fdc2df80d not found: ID does not exist"
Sep 11 21:13:16.671574 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:16.671543    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-fwl2l"]
Sep 11 21:13:16.679209 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:16.679182    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-fwl2l"]
Sep 11 21:13:17.926990 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:17.926959    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bdf2ae2-55c6-4660-9fb2-64a9c46efbb2" path="/var/lib/kubelet/pods/9bdf2ae2-55c6-4660-9fb2-64a9c46efbb2/volumes"
Sep 11 21:13:23.633302 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:23.633268    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"
Sep 11 21:13:38.891266 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:38.891228    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-dkg48"]
Sep 11 21:13:38.891667 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:38.891518    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bdf2ae2-55c6-4660-9fb2-64a9c46efbb2" containerName="authorino"
Sep 11 21:13:38.891667 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:38.891530    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bdf2ae2-55c6-4660-9fb2-64a9c46efbb2" containerName="authorino"
Sep 11 21:13:38.891667 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:38.891586    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bdf2ae2-55c6-4660-9fb2-64a9c46efbb2" containerName="authorino"
Sep 11 21:13:38.897353 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:38.897332    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dkg48"
Sep 11 21:13:38.902058 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:38.902034    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-dkg48"]
Sep 11 21:13:38.927519 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:38.927486    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlft4\" (UniqueName: \"kubernetes.io/projected/b4ebf558-38d4-4584-a324-c18dc4139ff4-kube-api-access-dlft4\") pod \"authorino-557f876f69-dkg48\" (UID: \"b4ebf558-38d4-4584-a324-c18dc4139ff4\") " pod="kuadrant-system/authorino-557f876f69-dkg48"
Sep 11 21:13:38.947164 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:38.947131    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-dkg48"]
Sep 11 21:13:38.947379 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:13:38.947358    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dlft4], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-dkg48" podUID="b4ebf558-38d4-4584-a324-c18dc4139ff4"
Sep 11 21:13:38.973897 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:38.973864    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-rlws5"]
Sep 11 21:13:38.976796 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:38.976779    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-rlws5"
Sep 11 21:13:38.979259 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:38.979239    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 21:13:38.987780 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:38.987757    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-rlws5"]
Sep 11 21:13:39.028412 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.028370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c30434b8-e63b-4920-a846-9d94b7c22e99-tls-cert\") pod \"authorino-6d4dfc649f-rlws5\" (UID: \"c30434b8-e63b-4920-a846-9d94b7c22e99\") " pod="kuadrant-system/authorino-6d4dfc649f-rlws5"
Sep 11 21:13:39.028580 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.028453    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlft4\" (UniqueName: \"kubernetes.io/projected/b4ebf558-38d4-4584-a324-c18dc4139ff4-kube-api-access-dlft4\") pod \"authorino-557f876f69-dkg48\" (UID: \"b4ebf558-38d4-4584-a324-c18dc4139ff4\") " pod="kuadrant-system/authorino-557f876f69-dkg48"
Sep 11 21:13:39.028580 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.028473    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6sbl\" (UniqueName: \"kubernetes.io/projected/c30434b8-e63b-4920-a846-9d94b7c22e99-kube-api-access-f6sbl\") pod \"authorino-6d4dfc649f-rlws5\" (UID: \"c30434b8-e63b-4920-a846-9d94b7c22e99\") " pod="kuadrant-system/authorino-6d4dfc649f-rlws5"
Sep 11 21:13:39.036503 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.036477    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlft4\" (UniqueName: \"kubernetes.io/projected/b4ebf558-38d4-4584-a324-c18dc4139ff4-kube-api-access-dlft4\") pod \"authorino-557f876f69-dkg48\" (UID: \"b4ebf558-38d4-4584-a324-c18dc4139ff4\") " pod="kuadrant-system/authorino-557f876f69-dkg48"
Sep 11 21:13:39.129618 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.129583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c30434b8-e63b-4920-a846-9d94b7c22e99-tls-cert\") pod \"authorino-6d4dfc649f-rlws5\" (UID: \"c30434b8-e63b-4920-a846-9d94b7c22e99\") " pod="kuadrant-system/authorino-6d4dfc649f-rlws5"
Sep 11 21:13:39.129800 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.129654    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6sbl\" (UniqueName: \"kubernetes.io/projected/c30434b8-e63b-4920-a846-9d94b7c22e99-kube-api-access-f6sbl\") pod \"authorino-6d4dfc649f-rlws5\" (UID: \"c30434b8-e63b-4920-a846-9d94b7c22e99\") " pod="kuadrant-system/authorino-6d4dfc649f-rlws5"
Sep 11 21:13:39.132055 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.132032    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c30434b8-e63b-4920-a846-9d94b7c22e99-tls-cert\") pod \"authorino-6d4dfc649f-rlws5\" (UID: \"c30434b8-e63b-4920-a846-9d94b7c22e99\") " pod="kuadrant-system/authorino-6d4dfc649f-rlws5"
Sep 11 21:13:39.140123 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.140101    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6sbl\" (UniqueName: \"kubernetes.io/projected/c30434b8-e63b-4920-a846-9d94b7c22e99-kube-api-access-f6sbl\") pod \"authorino-6d4dfc649f-rlws5\" (UID: \"c30434b8-e63b-4920-a846-9d94b7c22e99\") " pod="kuadrant-system/authorino-6d4dfc649f-rlws5"
Sep 11 21:13:39.162593 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.162522    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-rlws5"]
Sep 11 21:13:39.162765 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.162753    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-rlws5"
Sep 11 21:13:39.194382 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.194348    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-ccd5578fd-sxbg8"]
Sep 11 21:13:39.197573 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.197549    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ccd5578fd-sxbg8"
Sep 11 21:13:39.211663 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.211617    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-ccd5578fd-sxbg8"]
Sep 11 21:13:39.230519 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.230492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/61ce80c5-faed-4b9e-9407-27a7f2472e97-tls-cert\") pod \"authorino-ccd5578fd-sxbg8\" (UID: \"61ce80c5-faed-4b9e-9407-27a7f2472e97\") " pod="kuadrant-system/authorino-ccd5578fd-sxbg8"
Sep 11 21:13:39.230670 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.230525    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl679\" (UniqueName: \"kubernetes.io/projected/61ce80c5-faed-4b9e-9407-27a7f2472e97-kube-api-access-bl679\") pod \"authorino-ccd5578fd-sxbg8\" (UID: \"61ce80c5-faed-4b9e-9407-27a7f2472e97\") " pod="kuadrant-system/authorino-ccd5578fd-sxbg8"
Sep 11 21:13:39.331331 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.331288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/61ce80c5-faed-4b9e-9407-27a7f2472e97-tls-cert\") pod \"authorino-ccd5578fd-sxbg8\" (UID: \"61ce80c5-faed-4b9e-9407-27a7f2472e97\") " pod="kuadrant-system/authorino-ccd5578fd-sxbg8"
Sep 11 21:13:39.331331 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.331335    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl679\" (UniqueName: \"kubernetes.io/projected/61ce80c5-faed-4b9e-9407-27a7f2472e97-kube-api-access-bl679\") pod \"authorino-ccd5578fd-sxbg8\" (UID: \"61ce80c5-faed-4b9e-9407-27a7f2472e97\") " pod="kuadrant-system/authorino-ccd5578fd-sxbg8"
Sep 11 21:13:39.333751 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.333730    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/61ce80c5-faed-4b9e-9407-27a7f2472e97-tls-cert\") pod \"authorino-ccd5578fd-sxbg8\" (UID: \"61ce80c5-faed-4b9e-9407-27a7f2472e97\") " pod="kuadrant-system/authorino-ccd5578fd-sxbg8"
Sep 11 21:13:39.339241 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.339217    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl679\" (UniqueName: \"kubernetes.io/projected/61ce80c5-faed-4b9e-9407-27a7f2472e97-kube-api-access-bl679\") pod \"authorino-ccd5578fd-sxbg8\" (UID: \"61ce80c5-faed-4b9e-9407-27a7f2472e97\") " pod="kuadrant-system/authorino-ccd5578fd-sxbg8"
Sep 11 21:13:39.435008 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.434982    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-rlws5"]
Sep 11 21:13:39.437416 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:13:39.437390    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc30434b8_e63b_4920_a846_9d94b7c22e99.slice/crio-91ccf0d3ea2247a20b64419d663e64dd28e4df57048f93f7f77c39eeaacb3dec WatchSource:0}: Error finding container 91ccf0d3ea2247a20b64419d663e64dd28e4df57048f93f7f77c39eeaacb3dec: Status 404 returned error can't find the container with id 91ccf0d3ea2247a20b64419d663e64dd28e4df57048f93f7f77c39eeaacb3dec
Sep 11 21:13:39.506885 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.506845    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ccd5578fd-sxbg8"
Sep 11 21:13:39.739377 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.739301    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dkg48"
Sep 11 21:13:39.739377 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.739311    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-rlws5" event={"ID":"c30434b8-e63b-4920-a846-9d94b7c22e99","Type":"ContainerStarted","Data":"91ccf0d3ea2247a20b64419d663e64dd28e4df57048f93f7f77c39eeaacb3dec"}
Sep 11 21:13:39.745682 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.745662    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dkg48"
Sep 11 21:13:39.786213 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.786181    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-ccd5578fd-sxbg8"]
Sep 11 21:13:39.788795 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:13:39.788770    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61ce80c5_faed_4b9e_9407_27a7f2472e97.slice/crio-1d3c0f34b8aee502d2d6579b3b392dd9c367517c60699f2afe2c64deb0aea696 WatchSource:0}: Error finding container 1d3c0f34b8aee502d2d6579b3b392dd9c367517c60699f2afe2c64deb0aea696: Status 404 returned error can't find the container with id 1d3c0f34b8aee502d2d6579b3b392dd9c367517c60699f2afe2c64deb0aea696
Sep 11 21:13:39.834643 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.834612    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlft4\" (UniqueName: \"kubernetes.io/projected/b4ebf558-38d4-4584-a324-c18dc4139ff4-kube-api-access-dlft4\") pod \"b4ebf558-38d4-4584-a324-c18dc4139ff4\" (UID: \"b4ebf558-38d4-4584-a324-c18dc4139ff4\") "
Sep 11 21:13:39.836660 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.836636    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ebf558-38d4-4584-a324-c18dc4139ff4-kube-api-access-dlft4" (OuterVolumeSpecName: "kube-api-access-dlft4") pod "b4ebf558-38d4-4584-a324-c18dc4139ff4" (UID: "b4ebf558-38d4-4584-a324-c18dc4139ff4"). InnerVolumeSpecName "kube-api-access-dlft4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:13:39.935883 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:39.935786    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlft4\" (UniqueName: \"kubernetes.io/projected/b4ebf558-38d4-4584-a324-c18dc4139ff4-kube-api-access-dlft4\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:13:40.745106 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:40.745060    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-rlws5" event={"ID":"c30434b8-e63b-4920-a846-9d94b7c22e99","Type":"ContainerStarted","Data":"f57a15d6eb412abc7c980b9aef061c5167aa81b204f9c983501fc31cca754921"}
Sep 11 21:13:40.745295 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:40.745141    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6d4dfc649f-rlws5" podUID="c30434b8-e63b-4920-a846-9d94b7c22e99" containerName="authorino" containerID="cri-o://f57a15d6eb412abc7c980b9aef061c5167aa81b204f9c983501fc31cca754921" gracePeriod=30
Sep 11 21:13:40.746586 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:40.746565    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dkg48"
Sep 11 21:13:40.746586 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:40.746569    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-ccd5578fd-sxbg8" event={"ID":"61ce80c5-faed-4b9e-9407-27a7f2472e97","Type":"ContainerStarted","Data":"5fabfaa6dfa4f3c88edb3f764a522948629af74fdb310580206e470cefbfaf89"}
Sep 11 21:13:40.746722 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:40.746609    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-ccd5578fd-sxbg8" event={"ID":"61ce80c5-faed-4b9e-9407-27a7f2472e97","Type":"ContainerStarted","Data":"1d3c0f34b8aee502d2d6579b3b392dd9c367517c60699f2afe2c64deb0aea696"}
Sep 11 21:13:40.765029 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:40.764985    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d4dfc649f-rlws5" podStartSLOduration=2.3336524 podStartE2EDuration="2.764968949s" totalImagesPullingTime="431.316549ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:13:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:13:40.763517784 +0000 UTC m=+487.467487073" watchObservedRunningTime="2026-09-11 21:13:40.764968949 +0000 UTC m=+487.468938240"
Sep 11 21:13:40.779194 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:40.779033    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-ccd5578fd-sxbg8" podStartSLOduration=1.323749491 podStartE2EDuration="1.77901391s" totalImagesPullingTime="455.264419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:13:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:13:40.778459573 +0000 UTC m=+487.482428863" watchObservedRunningTime="2026-09-11 21:13:40.77901391 +0000 UTC m=+487.482983201"
Sep 11 21:13:40.808043 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:40.808010    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-dkg48"]
Sep 11 21:13:40.813449 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:40.813420    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5flf2"]
Sep 11 21:13:40.813659 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:40.813634    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-5flf2" podUID="b0d93c89-9d05-4aa9-bc56-24d9d5fec7b7" containerName="authorino" containerID="cri-o://86edda8da74c53bab7aab339b3dd4979945879543e6e24e6c395715e4595959a" gracePeriod=30
Sep 11 21:13:40.814979 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:40.814943    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-dkg48"]
Sep 11 21:13:41.063219 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.063191    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-rlws5"
Sep 11 21:13:41.088950 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.088920    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-5flf2"
Sep 11 21:13:41.142918 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.142883    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6sbl\" (UniqueName: \"kubernetes.io/projected/c30434b8-e63b-4920-a846-9d94b7c22e99-kube-api-access-f6sbl\") pod \"c30434b8-e63b-4920-a846-9d94b7c22e99\" (UID: \"c30434b8-e63b-4920-a846-9d94b7c22e99\") "
Sep 11 21:13:41.143104 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.142935    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c30434b8-e63b-4920-a846-9d94b7c22e99-tls-cert\") pod \"c30434b8-e63b-4920-a846-9d94b7c22e99\" (UID: \"c30434b8-e63b-4920-a846-9d94b7c22e99\") "
Sep 11 21:13:41.143104 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.142955    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28nch\" (UniqueName: \"kubernetes.io/projected/b0d93c89-9d05-4aa9-bc56-24d9d5fec7b7-kube-api-access-28nch\") pod \"b0d93c89-9d05-4aa9-bc56-24d9d5fec7b7\" (UID: \"b0d93c89-9d05-4aa9-bc56-24d9d5fec7b7\") "
Sep 11 21:13:41.145148 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.145108    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d93c89-9d05-4aa9-bc56-24d9d5fec7b7-kube-api-access-28nch" (OuterVolumeSpecName: "kube-api-access-28nch") pod "b0d93c89-9d05-4aa9-bc56-24d9d5fec7b7" (UID: "b0d93c89-9d05-4aa9-bc56-24d9d5fec7b7"). InnerVolumeSpecName "kube-api-access-28nch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:13:41.145278 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.145193    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c30434b8-e63b-4920-a846-9d94b7c22e99-kube-api-access-f6sbl" (OuterVolumeSpecName: "kube-api-access-f6sbl") pod "c30434b8-e63b-4920-a846-9d94b7c22e99" (UID: "c30434b8-e63b-4920-a846-9d94b7c22e99"). InnerVolumeSpecName "kube-api-access-f6sbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:13:41.152229 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.152204    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c30434b8-e63b-4920-a846-9d94b7c22e99-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "c30434b8-e63b-4920-a846-9d94b7c22e99" (UID: "c30434b8-e63b-4920-a846-9d94b7c22e99"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:13:41.243433 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.243394    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6sbl\" (UniqueName: \"kubernetes.io/projected/c30434b8-e63b-4920-a846-9d94b7c22e99-kube-api-access-f6sbl\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:13:41.243433 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.243426    2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c30434b8-e63b-4920-a846-9d94b7c22e99-tls-cert\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:13:41.243433 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.243437    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28nch\" (UniqueName: \"kubernetes.io/projected/b0d93c89-9d05-4aa9-bc56-24d9d5fec7b7-kube-api-access-28nch\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:13:41.751252 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.751221    2578 generic.go:356] "Generic (PLEG): container finished" podID="b0d93c89-9d05-4aa9-bc56-24d9d5fec7b7" containerID="86edda8da74c53bab7aab339b3dd4979945879543e6e24e6c395715e4595959a" exitCode=0
Sep 11 21:13:41.751427 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.751263    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-5flf2"
Sep 11 21:13:41.751427 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.751290    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-5flf2" event={"ID":"b0d93c89-9d05-4aa9-bc56-24d9d5fec7b7","Type":"ContainerDied","Data":"86edda8da74c53bab7aab339b3dd4979945879543e6e24e6c395715e4595959a"}
Sep 11 21:13:41.751427 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.751328    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-5flf2" event={"ID":"b0d93c89-9d05-4aa9-bc56-24d9d5fec7b7","Type":"ContainerDied","Data":"fe479cec116bc49219a0d611b97a033f90c2a06d894b7b3e4c9f769e69369fb4"}
Sep 11 21:13:41.751427 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.751349    2578 scope.go:117] "RemoveContainer" containerID="86edda8da74c53bab7aab339b3dd4979945879543e6e24e6c395715e4595959a"
Sep 11 21:13:41.752652 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.752608    2578 generic.go:356] "Generic (PLEG): container finished" podID="c30434b8-e63b-4920-a846-9d94b7c22e99" containerID="f57a15d6eb412abc7c980b9aef061c5167aa81b204f9c983501fc31cca754921" exitCode=0
Sep 11 21:13:41.752652 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.752635    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-rlws5" event={"ID":"c30434b8-e63b-4920-a846-9d94b7c22e99","Type":"ContainerDied","Data":"f57a15d6eb412abc7c980b9aef061c5167aa81b204f9c983501fc31cca754921"}
Sep 11 21:13:41.752786 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.752662    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-rlws5"
Sep 11 21:13:41.752786 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.752671    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4dfc649f-rlws5" event={"ID":"c30434b8-e63b-4920-a846-9d94b7c22e99","Type":"ContainerDied","Data":"91ccf0d3ea2247a20b64419d663e64dd28e4df57048f93f7f77c39eeaacb3dec"}
Sep 11 21:13:41.760292 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.760274    2578 scope.go:117] "RemoveContainer" containerID="86edda8da74c53bab7aab339b3dd4979945879543e6e24e6c395715e4595959a"
Sep 11 21:13:41.760607 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:13:41.760565    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86edda8da74c53bab7aab339b3dd4979945879543e6e24e6c395715e4595959a\": container with ID starting with 86edda8da74c53bab7aab339b3dd4979945879543e6e24e6c395715e4595959a not found: ID does not exist" containerID="86edda8da74c53bab7aab339b3dd4979945879543e6e24e6c395715e4595959a"
Sep 11 21:13:41.760720 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.760616    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86edda8da74c53bab7aab339b3dd4979945879543e6e24e6c395715e4595959a"} err="failed to get container status \"86edda8da74c53bab7aab339b3dd4979945879543e6e24e6c395715e4595959a\": rpc error: code = NotFound desc = could not find container \"86edda8da74c53bab7aab339b3dd4979945879543e6e24e6c395715e4595959a\": container with ID starting with 86edda8da74c53bab7aab339b3dd4979945879543e6e24e6c395715e4595959a not found: ID does not exist"
Sep 11 21:13:41.760720 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.760641    2578 scope.go:117] "RemoveContainer" containerID="f57a15d6eb412abc7c980b9aef061c5167aa81b204f9c983501fc31cca754921"
Sep 11 21:13:41.768004 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.767986    2578 scope.go:117] "RemoveContainer" containerID="f57a15d6eb412abc7c980b9aef061c5167aa81b204f9c983501fc31cca754921"
Sep 11 21:13:41.768273 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:13:41.768255    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f57a15d6eb412abc7c980b9aef061c5167aa81b204f9c983501fc31cca754921\": container with ID starting with f57a15d6eb412abc7c980b9aef061c5167aa81b204f9c983501fc31cca754921 not found: ID does not exist" containerID="f57a15d6eb412abc7c980b9aef061c5167aa81b204f9c983501fc31cca754921"
Sep 11 21:13:41.768322 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.768281    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f57a15d6eb412abc7c980b9aef061c5167aa81b204f9c983501fc31cca754921"} err="failed to get container status \"f57a15d6eb412abc7c980b9aef061c5167aa81b204f9c983501fc31cca754921\": rpc error: code = NotFound desc = could not find container \"f57a15d6eb412abc7c980b9aef061c5167aa81b204f9c983501fc31cca754921\": container with ID starting with f57a15d6eb412abc7c980b9aef061c5167aa81b204f9c983501fc31cca754921 not found: ID does not exist"
Sep 11 21:13:41.774433 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.774412    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5flf2"]
Sep 11 21:13:41.779017 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.778993    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5flf2"]
Sep 11 21:13:41.790833 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.790787    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-rlws5"]
Sep 11 21:13:41.793156 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.793133    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-rlws5"]
Sep 11 21:13:41.927769 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.927737    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0d93c89-9d05-4aa9-bc56-24d9d5fec7b7" path="/var/lib/kubelet/pods/b0d93c89-9d05-4aa9-bc56-24d9d5fec7b7/volumes"
Sep 11 21:13:41.928078 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.928064    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4ebf558-38d4-4584-a324-c18dc4139ff4" path="/var/lib/kubelet/pods/b4ebf558-38d4-4584-a324-c18dc4139ff4/volumes"
Sep 11 21:13:41.928276 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:13:41.928266    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c30434b8-e63b-4920-a846-9d94b7c22e99" path="/var/lib/kubelet/pods/c30434b8-e63b-4920-a846-9d94b7c22e99/volumes"
Sep 11 21:14:24.016190 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:14:24.016157    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:14:48.519629 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:14:48.519591    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:14:57.415622 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:14:57.415581    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:15:04.832318 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:04.832280    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:15:12.625075 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:12.622312    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:15:24.015770 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.015740    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"]
Sep 11 21:15:24.016160 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.016028    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0d93c89-9d05-4aa9-bc56-24d9d5fec7b7" containerName="authorino"
Sep 11 21:15:24.016160 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.016042    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0d93c89-9d05-4aa9-bc56-24d9d5fec7b7" containerName="authorino"
Sep 11 21:15:24.016160 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.016049    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c30434b8-e63b-4920-a846-9d94b7c22e99" containerName="authorino"
Sep 11 21:15:24.016160 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.016054    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c30434b8-e63b-4920-a846-9d94b7c22e99" containerName="authorino"
Sep 11 21:15:24.016160 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.016112    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0d93c89-9d05-4aa9-bc56-24d9d5fec7b7" containerName="authorino"
Sep 11 21:15:24.016160 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.016119    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c30434b8-e63b-4920-a846-9d94b7c22e99" containerName="authorino"
Sep 11 21:15:24.019111 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.019096    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"
Sep 11 21:15:24.021617 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.021594    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 21:15:24.021729 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.021616    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 21:15:24.021729 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.021676    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-m6p5f\""
Sep 11 21:15:24.022411 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.022394    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 11 21:15:24.029697 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.029674    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"]
Sep 11 21:15:24.192296 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.192255    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e59a89-cdba-4e67-971d-351cabc2607c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r\" (UID: \"e9e59a89-cdba-4e67-971d-351cabc2607c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"
Sep 11 21:15:24.192296 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.192300    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e59a89-cdba-4e67-971d-351cabc2607c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r\" (UID: \"e9e59a89-cdba-4e67-971d-351cabc2607c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"
Sep 11 21:15:24.192531 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.192326    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhwb5\" (UniqueName: \"kubernetes.io/projected/e9e59a89-cdba-4e67-971d-351cabc2607c-kube-api-access-dhwb5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r\" (UID: \"e9e59a89-cdba-4e67-971d-351cabc2607c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"
Sep 11 21:15:24.192531 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.192364    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e59a89-cdba-4e67-971d-351cabc2607c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r\" (UID: \"e9e59a89-cdba-4e67-971d-351cabc2607c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"
Sep 11 21:15:24.192531 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.192382    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e59a89-cdba-4e67-971d-351cabc2607c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r\" (UID: \"e9e59a89-cdba-4e67-971d-351cabc2607c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"
Sep 11 21:15:24.192531 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.192402    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e59a89-cdba-4e67-971d-351cabc2607c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r\" (UID: \"e9e59a89-cdba-4e67-971d-351cabc2607c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"
Sep 11 21:15:24.293590 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.293502    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e59a89-cdba-4e67-971d-351cabc2607c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r\" (UID: \"e9e59a89-cdba-4e67-971d-351cabc2607c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"
Sep 11 21:15:24.293590 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.293553    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhwb5\" (UniqueName: \"kubernetes.io/projected/e9e59a89-cdba-4e67-971d-351cabc2607c-kube-api-access-dhwb5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r\" (UID: \"e9e59a89-cdba-4e67-971d-351cabc2607c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"
Sep 11 21:15:24.293590 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.293582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e59a89-cdba-4e67-971d-351cabc2607c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r\" (UID: \"e9e59a89-cdba-4e67-971d-351cabc2607c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"
Sep 11 21:15:24.293876 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.293615    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e59a89-cdba-4e67-971d-351cabc2607c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r\" (UID: \"e9e59a89-cdba-4e67-971d-351cabc2607c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"
Sep 11 21:15:24.293876 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.293635    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e59a89-cdba-4e67-971d-351cabc2607c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r\" (UID: \"e9e59a89-cdba-4e67-971d-351cabc2607c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"
Sep 11 21:15:24.293876 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.293691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e59a89-cdba-4e67-971d-351cabc2607c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r\" (UID: \"e9e59a89-cdba-4e67-971d-351cabc2607c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"
Sep 11 21:15:24.294044 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.294003    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e59a89-cdba-4e67-971d-351cabc2607c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r\" (UID: \"e9e59a89-cdba-4e67-971d-351cabc2607c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"
Sep 11 21:15:24.294044 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.294020    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e59a89-cdba-4e67-971d-351cabc2607c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r\" (UID: \"e9e59a89-cdba-4e67-971d-351cabc2607c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"
Sep 11 21:15:24.294133 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.294073    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e59a89-cdba-4e67-971d-351cabc2607c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r\" (UID: \"e9e59a89-cdba-4e67-971d-351cabc2607c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"
Sep 11 21:15:24.295899 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.295874    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e59a89-cdba-4e67-971d-351cabc2607c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r\" (UID: \"e9e59a89-cdba-4e67-971d-351cabc2607c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"
Sep 11 21:15:24.296018 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.295959    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e59a89-cdba-4e67-971d-351cabc2607c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r\" (UID: \"e9e59a89-cdba-4e67-971d-351cabc2607c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"
Sep 11 21:15:24.302081 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.302058    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhwb5\" (UniqueName: \"kubernetes.io/projected/e9e59a89-cdba-4e67-971d-351cabc2607c-kube-api-access-dhwb5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r\" (UID: \"e9e59a89-cdba-4e67-971d-351cabc2607c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"
Sep 11 21:15:24.330787 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.330760    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"
Sep 11 21:15:24.599578 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:24.599555    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"]
Sep 11 21:15:24.602176 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:15:24.602149    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e59a89_cdba_4e67_971d_351cabc2607c.slice/crio-5bd83309d2ebbacc69dfc0a4a4635ed092f8e8f0dd6b19992badef9c7c252b4c WatchSource:0}: Error finding container 5bd83309d2ebbacc69dfc0a4a4635ed092f8e8f0dd6b19992badef9c7c252b4c: Status 404 returned error can't find the container with id 5bd83309d2ebbacc69dfc0a4a4635ed092f8e8f0dd6b19992badef9c7c252b4c
Sep 11 21:15:25.159623 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:25.159582    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r" event={"ID":"e9e59a89-cdba-4e67-971d-351cabc2607c","Type":"ContainerStarted","Data":"5bd83309d2ebbacc69dfc0a4a4635ed092f8e8f0dd6b19992badef9c7c252b4c"}
Sep 11 21:15:25.618506 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:25.618473    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:15:26.164924 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:26.164890    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r" event={"ID":"e9e59a89-cdba-4e67-971d-351cabc2607c","Type":"ContainerStarted","Data":"f6ff0106e6d71e56ae30022bc7a758010001f9e58c276c69a5258dbbad493541"}
Sep 11 21:15:26.184702 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:26.184646    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r" podStartSLOduration=2.194197074 podStartE2EDuration="3.184629022s" totalImagesPullingTime="990.431948ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:15:26.183685955 +0000 UTC m=+592.887655247" watchObservedRunningTime="2026-09-11 21:15:26.184629022 +0000 UTC m=+592.888598334"
Sep 11 21:15:32.032514 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:32.032481    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:15:34.331356 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:34.331320    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"
Sep 11 21:15:34.331752 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:34.331414    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"
Sep 11 21:15:34.344017 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:34.343992    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"
Sep 11 21:15:35.210533 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:35.210508    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r"
Sep 11 21:15:43.131877 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:43.131753    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"]
Sep 11 21:15:43.147018 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:43.137387    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"
Sep 11 21:15:43.147018 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:43.140114    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 21:15:43.147467 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:43.147385    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"]
Sep 11 21:15:43.235178 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:43.235151    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcrdw\" (UniqueName: \"kubernetes.io/projected/5b863e7e-d89a-4f7f-a062-0e7be1c656b3-kube-api-access-pcrdw\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf\" (UID: \"5b863e7e-d89a-4f7f-a062-0e7be1c656b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"
Sep 11 21:15:43.235178 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:43.235180    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b863e7e-d89a-4f7f-a062-0e7be1c656b3-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf\" (UID: \"5b863e7e-d89a-4f7f-a062-0e7be1c656b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"
Sep 11 21:15:43.235376 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:43.235216    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b863e7e-d89a-4f7f-a062-0e7be1c656b3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf\" (UID: \"5b863e7e-d89a-4f7f-a062-0e7be1c656b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"
Sep 11 21:15:43.235376 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:43.235274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b863e7e-d89a-4f7f-a062-0e7be1c656b3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf\" (UID: \"5b863e7e-d89a-4f7f-a062-0e7be1c656b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"
Sep 11 21:15:43.235376 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:43.235315    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b863e7e-d89a-4f7f-a062-0e7be1c656b3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf\" (UID: \"5b863e7e-d89a-4f7f-a062-0e7be1c656b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"
Sep 11 21:15:43.235376 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:43.235340    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b863e7e-d89a-4f7f-a062-0e7be1c656b3-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf\" (UID: \"5b863e7e-d89a-4f7f-a062-0e7be1c656b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"
Sep 11 21:15:43.336441 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:43.336404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b863e7e-d89a-4f7f-a062-0e7be1c656b3-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf\" (UID: \"5b863e7e-d89a-4f7f-a062-0e7be1c656b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"
Sep 11 21:15:43.336441 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:43.336451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcrdw\" (UniqueName: \"kubernetes.io/projected/5b863e7e-d89a-4f7f-a062-0e7be1c656b3-kube-api-access-pcrdw\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf\" (UID: \"5b863e7e-d89a-4f7f-a062-0e7be1c656b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"
Sep 11 21:15:43.336679 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:43.336468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b863e7e-d89a-4f7f-a062-0e7be1c656b3-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf\" (UID: \"5b863e7e-d89a-4f7f-a062-0e7be1c656b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"
Sep 11 21:15:43.336679 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:43.336504    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b863e7e-d89a-4f7f-a062-0e7be1c656b3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf\" (UID: \"5b863e7e-d89a-4f7f-a062-0e7be1c656b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"
Sep 11 21:15:43.336679 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:43.336543    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b863e7e-d89a-4f7f-a062-0e7be1c656b3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf\" (UID: \"5b863e7e-d89a-4f7f-a062-0e7be1c656b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"
Sep 11 21:15:43.336679 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:43.336594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b863e7e-d89a-4f7f-a062-0e7be1c656b3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf\" (UID: \"5b863e7e-d89a-4f7f-a062-0e7be1c656b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"
Sep 11 21:15:43.336965 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:43.336905    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b863e7e-d89a-4f7f-a062-0e7be1c656b3-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf\" (UID: \"5b863e7e-d89a-4f7f-a062-0e7be1c656b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"
Sep 11 21:15:43.337016 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:43.336967    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b863e7e-d89a-4f7f-a062-0e7be1c656b3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf\" (UID: \"5b863e7e-d89a-4f7f-a062-0e7be1c656b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"
Sep 11 21:15:43.337054 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:43.337025    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b863e7e-d89a-4f7f-a062-0e7be1c656b3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf\" (UID: \"5b863e7e-d89a-4f7f-a062-0e7be1c656b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"
Sep 11 21:15:43.338647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:43.338623    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b863e7e-d89a-4f7f-a062-0e7be1c656b3-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf\" (UID: \"5b863e7e-d89a-4f7f-a062-0e7be1c656b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"
Sep 11 21:15:43.339057 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:43.339036    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b863e7e-d89a-4f7f-a062-0e7be1c656b3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf\" (UID: \"5b863e7e-d89a-4f7f-a062-0e7be1c656b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"
Sep 11 21:15:43.345174 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:43.345153    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcrdw\" (UniqueName: \"kubernetes.io/projected/5b863e7e-d89a-4f7f-a062-0e7be1c656b3-kube-api-access-pcrdw\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf\" (UID: \"5b863e7e-d89a-4f7f-a062-0e7be1c656b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"
Sep 11 21:15:43.445323 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:43.445291    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"
Sep 11 21:15:43.713557 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:43.713532    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"]
Sep 11 21:15:43.715777 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:15:43.715744    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b863e7e_d89a_4f7f_a062_0e7be1c656b3.slice/crio-d9be2041cdcfc4f79913cbb875b1f8c3440642014d162d6fdb8b5803b9b65b88 WatchSource:0}: Error finding container d9be2041cdcfc4f79913cbb875b1f8c3440642014d162d6fdb8b5803b9b65b88: Status 404 returned error can't find the container with id d9be2041cdcfc4f79913cbb875b1f8c3440642014d162d6fdb8b5803b9b65b88
Sep 11 21:15:44.237104 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:44.237069    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf" event={"ID":"5b863e7e-d89a-4f7f-a062-0e7be1c656b3","Type":"ContainerStarted","Data":"3b49125fc64dedca5bf03f1f8ce2464feed74d5133d23310b0b9e33e454e10a6"}
Sep 11 21:15:44.237104 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:44.237109    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf" event={"ID":"5b863e7e-d89a-4f7f-a062-0e7be1c656b3","Type":"ContainerStarted","Data":"d9be2041cdcfc4f79913cbb875b1f8c3440642014d162d6fdb8b5803b9b65b88"}
Sep 11 21:15:44.256280 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:44.256178    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf" podStartSLOduration=1.066906355 podStartE2EDuration="1.256158945s" totalImagesPullingTime="189.25259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:15:44.255446901 +0000 UTC m=+610.959416190" watchObservedRunningTime="2026-09-11 21:15:44.256158945 +0000 UTC m=+610.960128236"
Sep 11 21:15:45.421883 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:45.421847    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:15:53.445901 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:53.445866    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"
Sep 11 21:15:53.446318 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:53.446000    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"
Sep 11 21:15:53.458368 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:53.458345    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"
Sep 11 21:15:54.292041 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:15:54.292009    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf"
Sep 11 21:16:28.820191 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:28.820157    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-bbdbcc98c-pn522"]
Sep 11 21:16:28.823137 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:28.823122    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bbdbcc98c-pn522"
Sep 11 21:16:28.832629 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:28.832603    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bbdbcc98c-pn522"]
Sep 11 21:16:28.879240 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:28.879205    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwqnt\" (UniqueName: \"kubernetes.io/projected/e1542ea2-457c-41d8-9a59-aec6f3e2240d-kube-api-access-hwqnt\") pod \"authorino-bbdbcc98c-pn522\" (UID: \"e1542ea2-457c-41d8-9a59-aec6f3e2240d\") " pod="kuadrant-system/authorino-bbdbcc98c-pn522"
Sep 11 21:16:28.879418 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:28.879262    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e1542ea2-457c-41d8-9a59-aec6f3e2240d-tls-cert\") pod \"authorino-bbdbcc98c-pn522\" (UID: \"e1542ea2-457c-41d8-9a59-aec6f3e2240d\") " pod="kuadrant-system/authorino-bbdbcc98c-pn522"
Sep 11 21:16:28.980307 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:28.980268    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e1542ea2-457c-41d8-9a59-aec6f3e2240d-tls-cert\") pod \"authorino-bbdbcc98c-pn522\" (UID: \"e1542ea2-457c-41d8-9a59-aec6f3e2240d\") " pod="kuadrant-system/authorino-bbdbcc98c-pn522"
Sep 11 21:16:28.980476 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:28.980336    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwqnt\" (UniqueName: \"kubernetes.io/projected/e1542ea2-457c-41d8-9a59-aec6f3e2240d-kube-api-access-hwqnt\") pod \"authorino-bbdbcc98c-pn522\" (UID: \"e1542ea2-457c-41d8-9a59-aec6f3e2240d\") " pod="kuadrant-system/authorino-bbdbcc98c-pn522"
Sep 11 21:16:28.982873 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:28.982842    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e1542ea2-457c-41d8-9a59-aec6f3e2240d-tls-cert\") pod \"authorino-bbdbcc98c-pn522\" (UID: \"e1542ea2-457c-41d8-9a59-aec6f3e2240d\") " pod="kuadrant-system/authorino-bbdbcc98c-pn522"
Sep 11 21:16:28.989116 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:28.989094    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwqnt\" (UniqueName: \"kubernetes.io/projected/e1542ea2-457c-41d8-9a59-aec6f3e2240d-kube-api-access-hwqnt\") pod \"authorino-bbdbcc98c-pn522\" (UID: \"e1542ea2-457c-41d8-9a59-aec6f3e2240d\") " pod="kuadrant-system/authorino-bbdbcc98c-pn522"
Sep 11 21:16:29.132034 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:29.131945    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bbdbcc98c-pn522"
Sep 11 21:16:29.403181 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:29.403108    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bbdbcc98c-pn522"]
Sep 11 21:16:29.405947 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:16:29.405922    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1542ea2_457c_41d8_9a59_aec6f3e2240d.slice/crio-4fe49cffc98df7899a318f8f13f64eeb2e7147594e940c6424b56b544af80f91 WatchSource:0}: Error finding container 4fe49cffc98df7899a318f8f13f64eeb2e7147594e940c6424b56b544af80f91: Status 404 returned error can't find the container with id 4fe49cffc98df7899a318f8f13f64eeb2e7147594e940c6424b56b544af80f91
Sep 11 21:16:29.407201 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:29.407181    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:16:29.423579 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:29.423557    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bbdbcc98c-pn522" event={"ID":"e1542ea2-457c-41d8-9a59-aec6f3e2240d","Type":"ContainerStarted","Data":"4fe49cffc98df7899a318f8f13f64eeb2e7147594e940c6424b56b544af80f91"}
Sep 11 21:16:30.428883 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:30.428840    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bbdbcc98c-pn522" event={"ID":"e1542ea2-457c-41d8-9a59-aec6f3e2240d","Type":"ContainerStarted","Data":"7910c939b2ecbf85763867b17c576c3b9fd75e56fb8e1205056954f3c97cf8e2"}
Sep 11 21:16:30.445491 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:30.445436    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-bbdbcc98c-pn522" podStartSLOduration=1.608485647 podStartE2EDuration="2.445420154s" totalImagesPullingTime="836.934507ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:16:30.444752915 +0000 UTC m=+657.148722200" watchObservedRunningTime="2026-09-11 21:16:30.445420154 +0000 UTC m=+657.149389442"
Sep 11 21:16:30.476458 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:30.476425    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-ccd5578fd-sxbg8"]
Sep 11 21:16:30.476633 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:30.476613    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-ccd5578fd-sxbg8" podUID="61ce80c5-faed-4b9e-9407-27a7f2472e97" containerName="authorino" containerID="cri-o://5fabfaa6dfa4f3c88edb3f764a522948629af74fdb310580206e470cefbfaf89" gracePeriod=30
Sep 11 21:16:30.718073 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:30.718040    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ccd5578fd-sxbg8"
Sep 11 21:16:30.798654 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:30.798558    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/61ce80c5-faed-4b9e-9407-27a7f2472e97-tls-cert\") pod \"61ce80c5-faed-4b9e-9407-27a7f2472e97\" (UID: \"61ce80c5-faed-4b9e-9407-27a7f2472e97\") "
Sep 11 21:16:30.798654 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:30.798639    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bl679\" (UniqueName: \"kubernetes.io/projected/61ce80c5-faed-4b9e-9407-27a7f2472e97-kube-api-access-bl679\") pod \"61ce80c5-faed-4b9e-9407-27a7f2472e97\" (UID: \"61ce80c5-faed-4b9e-9407-27a7f2472e97\") "
Sep 11 21:16:30.800899 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:30.800866    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ce80c5-faed-4b9e-9407-27a7f2472e97-kube-api-access-bl679" (OuterVolumeSpecName: "kube-api-access-bl679") pod "61ce80c5-faed-4b9e-9407-27a7f2472e97" (UID: "61ce80c5-faed-4b9e-9407-27a7f2472e97"). InnerVolumeSpecName "kube-api-access-bl679". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:16:30.808118 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:30.808089    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61ce80c5-faed-4b9e-9407-27a7f2472e97-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "61ce80c5-faed-4b9e-9407-27a7f2472e97" (UID: "61ce80c5-faed-4b9e-9407-27a7f2472e97"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:16:30.899789 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:30.899752    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl679\" (UniqueName: \"kubernetes.io/projected/61ce80c5-faed-4b9e-9407-27a7f2472e97-kube-api-access-bl679\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:16:30.899789 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:30.899782    2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/61ce80c5-faed-4b9e-9407-27a7f2472e97-tls-cert\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:16:31.435393 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:31.435356    2578 generic.go:356] "Generic (PLEG): container finished" podID="61ce80c5-faed-4b9e-9407-27a7f2472e97" containerID="5fabfaa6dfa4f3c88edb3f764a522948629af74fdb310580206e470cefbfaf89" exitCode=0
Sep 11 21:16:31.435809 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:31.435449    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ccd5578fd-sxbg8"
Sep 11 21:16:31.435809 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:31.435486    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-ccd5578fd-sxbg8" event={"ID":"61ce80c5-faed-4b9e-9407-27a7f2472e97","Type":"ContainerDied","Data":"5fabfaa6dfa4f3c88edb3f764a522948629af74fdb310580206e470cefbfaf89"}
Sep 11 21:16:31.435809 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:31.435525    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-ccd5578fd-sxbg8" event={"ID":"61ce80c5-faed-4b9e-9407-27a7f2472e97","Type":"ContainerDied","Data":"1d3c0f34b8aee502d2d6579b3b392dd9c367517c60699f2afe2c64deb0aea696"}
Sep 11 21:16:31.435809 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:31.435547    2578 scope.go:117] "RemoveContainer" containerID="5fabfaa6dfa4f3c88edb3f764a522948629af74fdb310580206e470cefbfaf89"
Sep 11 21:16:31.444652 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:31.444635    2578 scope.go:117] "RemoveContainer" containerID="5fabfaa6dfa4f3c88edb3f764a522948629af74fdb310580206e470cefbfaf89"
Sep 11 21:16:31.444937 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:16:31.444909    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fabfaa6dfa4f3c88edb3f764a522948629af74fdb310580206e470cefbfaf89\": container with ID starting with 5fabfaa6dfa4f3c88edb3f764a522948629af74fdb310580206e470cefbfaf89 not found: ID does not exist" containerID="5fabfaa6dfa4f3c88edb3f764a522948629af74fdb310580206e470cefbfaf89"
Sep 11 21:16:31.445009 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:31.444943    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fabfaa6dfa4f3c88edb3f764a522948629af74fdb310580206e470cefbfaf89"} err="failed to get container status \"5fabfaa6dfa4f3c88edb3f764a522948629af74fdb310580206e470cefbfaf89\": rpc error: code = NotFound desc = could not find container \"5fabfaa6dfa4f3c88edb3f764a522948629af74fdb310580206e470cefbfaf89\": container with ID starting with 5fabfaa6dfa4f3c88edb3f764a522948629af74fdb310580206e470cefbfaf89 not found: ID does not exist"
Sep 11 21:16:31.459252 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:31.459230    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-ccd5578fd-sxbg8"]
Sep 11 21:16:31.465573 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:31.465549    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-ccd5578fd-sxbg8"]
Sep 11 21:16:31.928438 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:31.928406    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61ce80c5-faed-4b9e-9407-27a7f2472e97" path="/var/lib/kubelet/pods/61ce80c5-faed-4b9e-9407-27a7f2472e97/volumes"
Sep 11 21:16:49.927037 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:49.927008    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7711-ad1f22"]
Sep 11 21:16:49.927467 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:49.927245    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61ce80c5-faed-4b9e-9407-27a7f2472e97" containerName="authorino"
Sep 11 21:16:49.927467 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:49.927254    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="61ce80c5-faed-4b9e-9407-27a7f2472e97" containerName="authorino"
Sep 11 21:16:49.927467 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:49.927315    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="61ce80c5-faed-4b9e-9407-27a7f2472e97" containerName="authorino"
Sep 11 21:16:49.931580 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:49.931562    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-ad1f22"
Sep 11 21:16:49.933729 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:49.933705    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6dwfw\""
Sep 11 21:16:49.934567 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:49.934541    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7711-ad1f22"]
Sep 11 21:16:50.060166 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:50.060131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btfx8\" (UniqueName: \"kubernetes.io/projected/fd404c9a-a075-4692-89e5-30f3fc322d4b-kube-api-access-btfx8\") pod \"test-curl-7711-ad1f22\" (UID: \"fd404c9a-a075-4692-89e5-30f3fc322d4b\") " pod="opendatahub/test-curl-7711-ad1f22"
Sep 11 21:16:50.160807 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:50.160771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btfx8\" (UniqueName: \"kubernetes.io/projected/fd404c9a-a075-4692-89e5-30f3fc322d4b-kube-api-access-btfx8\") pod \"test-curl-7711-ad1f22\" (UID: \"fd404c9a-a075-4692-89e5-30f3fc322d4b\") " pod="opendatahub/test-curl-7711-ad1f22"
Sep 11 21:16:50.183220 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:50.183152    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btfx8\" (UniqueName: \"kubernetes.io/projected/fd404c9a-a075-4692-89e5-30f3fc322d4b-kube-api-access-btfx8\") pod \"test-curl-7711-ad1f22\" (UID: \"fd404c9a-a075-4692-89e5-30f3fc322d4b\") " pod="opendatahub/test-curl-7711-ad1f22"
Sep 11 21:16:50.241074 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:50.241038    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-ad1f22"
Sep 11 21:16:50.515665 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:50.515585    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7711-ad1f22"]
Sep 11 21:16:50.518976 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:16:50.518952    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd404c9a_a075_4692_89e5_30f3fc322d4b.slice/crio-fd4a87f9dfe8348342426384db245f00795b27095872c7d9fcc91708049d738d WatchSource:0}: Error finding container fd4a87f9dfe8348342426384db245f00795b27095872c7d9fcc91708049d738d: Status 404 returned error can't find the container with id fd4a87f9dfe8348342426384db245f00795b27095872c7d9fcc91708049d738d
Sep 11 21:16:51.522831 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:51.522790    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7711-ad1f22" event={"ID":"fd404c9a-a075-4692-89e5-30f3fc322d4b","Type":"ContainerStarted","Data":"fd4a87f9dfe8348342426384db245f00795b27095872c7d9fcc91708049d738d"}
Sep 11 21:16:52.135971 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:52.135936    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:16:53.533995 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:53.533951    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7711-ad1f22" event={"ID":"fd404c9a-a075-4692-89e5-30f3fc322d4b","Type":"ContainerStarted","Data":"bc4d488af4c67a341c269d61f2260fd9830debe611b3c6eb7cffb883d4c33380"}
Sep 11 21:16:53.551331 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:53.551242    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7711-ad1f22" podStartSLOduration=2.565751261 podStartE2EDuration="4.551220219s" totalImagesPullingTime="1.985468958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:16:53.549449131 +0000 UTC m=+680.253418421" watchObservedRunningTime="2026-09-11 21:16:53.551220219 +0000 UTC m=+680.255189512"
Sep 11 21:16:54.182988 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:54.182957    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7711-ad1f22"]
Sep 11 21:16:54.203978 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:54.203900    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7711-ad1f22"]
Sep 11 21:16:54.326691 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:54.324521    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7711-604c53"]
Sep 11 21:16:54.326691 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:54.325138    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd404c9a-a075-4692-89e5-30f3fc322d4b" containerName="test-curl-7711-ad1f22"
Sep 11 21:16:54.326691 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:54.325172    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd404c9a-a075-4692-89e5-30f3fc322d4b" containerName="test-curl-7711-ad1f22"
Sep 11 21:16:54.326691 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:54.325313    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd404c9a-a075-4692-89e5-30f3fc322d4b" containerName="test-curl-7711-ad1f22"
Sep 11 21:16:54.329053 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:54.329030    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-604c53"
Sep 11 21:16:54.332246 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:54.332220    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7711-604c53"]
Sep 11 21:16:54.391331 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:54.391292    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6w87\" (UniqueName: \"kubernetes.io/projected/d34e3b08-b395-4a82-80c7-3293ab09fcd2-kube-api-access-w6w87\") pod \"test-curl-7711-604c53\" (UID: \"d34e3b08-b395-4a82-80c7-3293ab09fcd2\") " pod="opendatahub/test-curl-7711-604c53"
Sep 11 21:16:54.491669 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:54.491576    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6w87\" (UniqueName: \"kubernetes.io/projected/d34e3b08-b395-4a82-80c7-3293ab09fcd2-kube-api-access-w6w87\") pod \"test-curl-7711-604c53\" (UID: \"d34e3b08-b395-4a82-80c7-3293ab09fcd2\") " pod="opendatahub/test-curl-7711-604c53"
Sep 11 21:16:54.499344 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:54.499318    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6w87\" (UniqueName: \"kubernetes.io/projected/d34e3b08-b395-4a82-80c7-3293ab09fcd2-kube-api-access-w6w87\") pod \"test-curl-7711-604c53\" (UID: \"d34e3b08-b395-4a82-80c7-3293ab09fcd2\") " pod="opendatahub/test-curl-7711-604c53"
Sep 11 21:16:54.538422 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:16:54.538393    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7711-ad1f22" container="test-curl-7711-ad1f22"
Sep 11 21:16:54.640704 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:54.640670    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-604c53"
Sep 11 21:16:54.911864 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:54.911761    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7711-604c53"]
Sep 11 21:16:54.915146 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:16:54.915111    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd34e3b08_b395_4a82_80c7_3293ab09fcd2.slice/crio-ffbcd3e88b86887bb998b6aeadf7c747a538f764b03b3168daaaf4bc82de38d9 WatchSource:0}: Error finding container ffbcd3e88b86887bb998b6aeadf7c747a538f764b03b3168daaaf4bc82de38d9: Status 404 returned error can't find the container with id ffbcd3e88b86887bb998b6aeadf7c747a538f764b03b3168daaaf4bc82de38d9
Sep 11 21:16:55.543705 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:55.543666    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7711-604c53" event={"ID":"d34e3b08-b395-4a82-80c7-3293ab09fcd2","Type":"ContainerStarted","Data":"7f5929a0964346fd84defd198c359d0eefa460c36d5ae36c49daa93d6465b53d"}
Sep 11 21:16:55.544180 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:55.543719    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7711-604c53" event={"ID":"d34e3b08-b395-4a82-80c7-3293ab09fcd2","Type":"ContainerStarted","Data":"ffbcd3e88b86887bb998b6aeadf7c747a538f764b03b3168daaaf4bc82de38d9"}
Sep 11 21:16:55.544180 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:55.543766    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7711-ad1f22" podUID="fd404c9a-a075-4692-89e5-30f3fc322d4b" containerName="test-curl-7711-ad1f22" containerID="cri-o://bc4d488af4c67a341c269d61f2260fd9830debe611b3c6eb7cffb883d4c33380" gracePeriod=2
Sep 11 21:16:55.558080 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:55.558019    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7711-604c53" podStartSLOduration=1.557996768 podStartE2EDuration="1.557996768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:16:55.557221336 +0000 UTC m=+682.261190625" watchObservedRunningTime="2026-09-11 21:16:55.557996768 +0000 UTC m=+682.261966058"
Sep 11 21:16:55.559384 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:16:55.559346    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7711-ad1f22\" is forbidden: User \"system:node:ip-10-0-142-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-224.ec2.internal' and this object" podUID="fd404c9a-a075-4692-89e5-30f3fc322d4b" pod="opendatahub/test-curl-7711-ad1f22"
Sep 11 21:16:55.560998 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:16:55.560976    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7711-ad1f22\" is forbidden: User \"system:node:ip-10-0-142-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-224.ec2.internal' and this object" podUID="fd404c9a-a075-4692-89e5-30f3fc322d4b" pod="opendatahub/test-curl-7711-ad1f22"
Sep 11 21:16:56.171357 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.171321    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn"]
Sep 11 21:16:56.174639 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.174622    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn"
Sep 11 21:16:56.177672 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.177640    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-ea5e33\""
Sep 11 21:16:56.177838 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.177728    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-ea5e33-dockercfg-l4ht6\""
Sep 11 21:16:56.188889 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.188861    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn"]
Sep 11 21:16:56.201510 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.201420    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcn8x\" (UniqueName: \"kubernetes.io/projected/f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9-kube-api-access-qcn8x\") pod \"payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn\" (UID: \"f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn"
Sep 11 21:16:56.201653 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.201573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn\" (UID: \"f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn"
Sep 11 21:16:56.212605 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.212577    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx"]
Sep 11 21:16:56.215968 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.215920    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7711-604c53"]
Sep 11 21:16:56.216114 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.216021    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx"
Sep 11 21:16:56.218116 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.218095    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7711-604c53"]
Sep 11 21:16:56.227935 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.227908    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx"]
Sep 11 21:16:56.302151 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.302100    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn\" (UID: \"f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn"
Sep 11 21:16:56.302151 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.302152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb307345-d97a-470b-8d73-1406c761d5c4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx\" (UID: \"fb307345-d97a-470b-8d73-1406c761d5c4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx"
Sep 11 21:16:56.302403 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.302262    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkkft\" (UniqueName: \"kubernetes.io/projected/fb307345-d97a-470b-8d73-1406c761d5c4-kube-api-access-dkkft\") pod \"payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx\" (UID: \"fb307345-d97a-470b-8d73-1406c761d5c4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx"
Sep 11 21:16:56.302403 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.302316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcn8x\" (UniqueName: \"kubernetes.io/projected/f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9-kube-api-access-qcn8x\") pod \"payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn\" (UID: \"f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn"
Sep 11 21:16:56.302915 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.302895    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn\" (UID: \"f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn"
Sep 11 21:16:56.310883 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.310855    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcn8x\" (UniqueName: \"kubernetes.io/projected/f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9-kube-api-access-qcn8x\") pod \"payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn\" (UID: \"f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn"
Sep 11 21:16:56.402795 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.402751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb307345-d97a-470b-8d73-1406c761d5c4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx\" (UID: \"fb307345-d97a-470b-8d73-1406c761d5c4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx"
Sep 11 21:16:56.402795 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.402795    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkkft\" (UniqueName: \"kubernetes.io/projected/fb307345-d97a-470b-8d73-1406c761d5c4-kube-api-access-dkkft\") pod \"payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx\" (UID: \"fb307345-d97a-470b-8d73-1406c761d5c4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx"
Sep 11 21:16:56.403516 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.403492    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb307345-d97a-470b-8d73-1406c761d5c4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx\" (UID: \"fb307345-d97a-470b-8d73-1406c761d5c4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx"
Sep 11 21:16:56.414435 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.414409    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkkft\" (UniqueName: \"kubernetes.io/projected/fb307345-d97a-470b-8d73-1406c761d5c4-kube-api-access-dkkft\") pod \"payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx\" (UID: \"fb307345-d97a-470b-8d73-1406c761d5c4\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx"
Sep 11 21:16:56.484411 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.484305    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn"
Sep 11 21:16:56.528618 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.528576    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx"
Sep 11 21:16:56.549520 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:16:56.549488    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7711-604c53" container="test-curl-7711-604c53"
Sep 11 21:16:56.759169 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.759006    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn"]
Sep 11 21:16:56.761735 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:16:56.761711    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e9c119_d654_4e0e_9b2a_215f5c0ebfd9.slice/crio-fc5046f323855f508c305cbca6943d35916e4f4e32d20459d036fbcba66609fa WatchSource:0}: Error finding container fc5046f323855f508c305cbca6943d35916e4f4e32d20459d036fbcba66609fa: Status 404 returned error can't find the container with id fc5046f323855f508c305cbca6943d35916e4f4e32d20459d036fbcba66609fa
Sep 11 21:16:56.812335 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:56.812306    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx"]
Sep 11 21:16:56.815272 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:16:56.815242    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb307345_d97a_470b_8d73_1406c761d5c4.slice/crio-0959f16e7ac31341a336775c907a5360eff1729f875d4bb0faf7c9535761aa14 WatchSource:0}: Error finding container 0959f16e7ac31341a336775c907a5360eff1729f875d4bb0faf7c9535761aa14: Status 404 returned error can't find the container with id 0959f16e7ac31341a336775c907a5360eff1729f875d4bb0faf7c9535761aa14
Sep 11 21:16:57.558232 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:57.558188    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn" event={"ID":"f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9","Type":"ContainerStarted","Data":"fc5046f323855f508c305cbca6943d35916e4f4e32d20459d036fbcba66609fa"}
Sep 11 21:16:57.560461 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:57.560424    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7711-604c53" podUID="d34e3b08-b395-4a82-80c7-3293ab09fcd2" containerName="test-curl-7711-604c53" containerID="cri-o://7f5929a0964346fd84defd198c359d0eefa460c36d5ae36c49daa93d6465b53d" gracePeriod=2
Sep 11 21:16:57.560774 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:57.560751    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx" event={"ID":"fb307345-d97a-470b-8d73-1406c761d5c4","Type":"ContainerStarted","Data":"0959f16e7ac31341a336775c907a5360eff1729f875d4bb0faf7c9535761aa14"}
Sep 11 21:16:57.562960 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:16:57.562928    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7711-604c53\" is forbidden: User \"system:node:ip-10-0-142-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-224.ec2.internal' and this object" podUID="d34e3b08-b395-4a82-80c7-3293ab09fcd2" pod="opendatahub/test-curl-7711-604c53"
Sep 11 21:16:57.794676 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:57.794228    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-ad1f22"
Sep 11 21:16:57.796139 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:16:57.796102    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7711-604c53\" is forbidden: User \"system:node:ip-10-0-142-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-224.ec2.internal' and this object" podUID="d34e3b08-b395-4a82-80c7-3293ab09fcd2" pod="opendatahub/test-curl-7711-604c53"
Sep 11 21:16:57.910572 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:57.910491    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"]
Sep 11 21:16:57.911133 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:57.911113    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d34e3b08-b395-4a82-80c7-3293ab09fcd2" containerName="test-curl-7711-604c53"
Sep 11 21:16:57.911133 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:57.911136    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d34e3b08-b395-4a82-80c7-3293ab09fcd2" containerName="test-curl-7711-604c53"
Sep 11 21:16:57.911322 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:57.911288    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d34e3b08-b395-4a82-80c7-3293ab09fcd2" containerName="test-curl-7711-604c53"
Sep 11 21:16:57.915316 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:57.915246    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:57.915796 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:57.915642    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btfx8\" (UniqueName: \"kubernetes.io/projected/fd404c9a-a075-4692-89e5-30f3fc322d4b-kube-api-access-btfx8\") pod \"fd404c9a-a075-4692-89e5-30f3fc322d4b\" (UID: \"fd404c9a-a075-4692-89e5-30f3fc322d4b\") "
Sep 11 21:16:57.915796 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:57.915769    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-workload-socket\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:57.916294 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:57.915798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istio-envoy\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:57.916294 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:57.915849    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-credential-socket\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:57.916294 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:57.915905    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvrpn\" (UniqueName: \"kubernetes.io/projected/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-kube-api-access-xvrpn\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:57.916294 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:57.915954    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istio-token\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:57.916294 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:57.915983    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-workload-certs\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:57.916294 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:57.916022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istio-data\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:57.916294 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:57.916048    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istiod-ca-cert\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:57.916294 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:57.916113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istio-podinfo\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:57.920120 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:16:57.919294    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7711-604c53\" is forbidden: User \"system:node:ip-10-0-142-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-224.ec2.internal' and this object" podUID="d34e3b08-b395-4a82-80c7-3293ab09fcd2" pod="opendatahub/test-curl-7711-604c53"
Sep 11 21:16:57.920120 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:57.919323    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8dbad4ec-openshift-default-dockercfg-csm5p\""
Sep 11 21:16:57.920120 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:57.919328    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 21:16:57.920337 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:57.919765    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd404c9a-a075-4692-89e5-30f3fc322d4b-kube-api-access-btfx8" (OuterVolumeSpecName: "kube-api-access-btfx8") pod "fd404c9a-a075-4692-89e5-30f3fc322d4b" (UID: "fd404c9a-a075-4692-89e5-30f3fc322d4b"). InnerVolumeSpecName "kube-api-access-btfx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:16:57.932137 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:57.932061    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd404c9a-a075-4692-89e5-30f3fc322d4b" path="/var/lib/kubelet/pods/fd404c9a-a075-4692-89e5-30f3fc322d4b/volumes"
Sep 11 21:16:57.932925 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:57.932498    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"]
Sep 11 21:16:58.017260 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.017166    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-workload-socket\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:58.017260 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.017209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istio-envoy\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:58.017260 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.017238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-credential-socket\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:58.017577 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.017273    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvrpn\" (UniqueName: \"kubernetes.io/projected/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-kube-api-access-xvrpn\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:58.017577 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.017315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istio-token\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:58.017577 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.017342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-workload-certs\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:58.017577 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.017380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istio-data\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:58.017577 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.017406    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istiod-ca-cert\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:58.017577 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.017444    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istio-podinfo\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:58.017577 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.017505    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btfx8\" (UniqueName: \"kubernetes.io/projected/fd404c9a-a075-4692-89e5-30f3fc322d4b-kube-api-access-btfx8\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:16:58.018078 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.017805    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-workload-certs\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:58.018078 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.017850    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-workload-socket\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:58.018241 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.018120    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istio-data\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:58.018241 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.018132    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-credential-socket\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:58.018568 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.018470    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istiod-ca-cert\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:58.020766 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.020707    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istio-envoy\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:58.022294 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.022270    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istio-podinfo\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:58.026929 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.026889    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istio-token\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:58.027449 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.027426    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvrpn\" (UniqueName: \"kubernetes.io/projected/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-kube-api-access-xvrpn\") pod \"e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") " pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:58.241185 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.241151    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:16:58.566574 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.566495    2578 generic.go:356] "Generic (PLEG): container finished" podID="fd404c9a-a075-4692-89e5-30f3fc322d4b" containerID="bc4d488af4c67a341c269d61f2260fd9830debe611b3c6eb7cffb883d4c33380" exitCode=137
Sep 11 21:16:58.566574 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.566542    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-ad1f22"
Sep 11 21:16:58.566574 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.566565    2578 scope.go:117] "RemoveContainer" containerID="bc4d488af4c67a341c269d61f2260fd9830debe611b3c6eb7cffb883d4c33380"
Sep 11 21:16:58.916961 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.916932    2578 scope.go:117] "RemoveContainer" containerID="bc4d488af4c67a341c269d61f2260fd9830debe611b3c6eb7cffb883d4c33380"
Sep 11 21:16:58.917325 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:16:58.917305    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc4d488af4c67a341c269d61f2260fd9830debe611b3c6eb7cffb883d4c33380\": container with ID starting with bc4d488af4c67a341c269d61f2260fd9830debe611b3c6eb7cffb883d4c33380 not found: ID does not exist" containerID="bc4d488af4c67a341c269d61f2260fd9830debe611b3c6eb7cffb883d4c33380"
Sep 11 21:16:58.917389 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:58.917339    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc4d488af4c67a341c269d61f2260fd9830debe611b3c6eb7cffb883d4c33380"} err="failed to get container status \"bc4d488af4c67a341c269d61f2260fd9830debe611b3c6eb7cffb883d4c33380\": rpc error: code = NotFound desc = could not find container \"bc4d488af4c67a341c269d61f2260fd9830debe611b3c6eb7cffb883d4c33380\": container with ID starting with bc4d488af4c67a341c269d61f2260fd9830debe611b3c6eb7cffb883d4c33380 not found: ID does not exist"
Sep 11 21:16:59.488346 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:59.488322    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"]
Sep 11 21:16:59.491009 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:16:59.490980    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ba4b30_5dcb_4bbd_833e_c8bfb3b58eed.slice/crio-1aec0e9d1709e44f3078cfa184c25c25d0434c82149508015791c086a2f5e634 WatchSource:0}: Error finding container 1aec0e9d1709e44f3078cfa184c25c25d0434c82149508015791c086a2f5e634: Status 404 returned error can't find the container with id 1aec0e9d1709e44f3078cfa184c25c25d0434c82149508015791c086a2f5e634
Sep 11 21:16:59.572964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:59.572933    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm" event={"ID":"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed","Type":"ContainerStarted","Data":"1aec0e9d1709e44f3078cfa184c25c25d0434c82149508015791c086a2f5e634"}
Sep 11 21:16:59.574594 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:59.574553    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn" event={"ID":"f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9","Type":"ContainerStarted","Data":"22ccde18d7b5851c7bc9e7e996fb835290567331228bafa5a317fd73aa17572a"}
Sep 11 21:16:59.574811 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:59.574767    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn"
Sep 11 21:16:59.576486 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:59.576460    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx" event={"ID":"fb307345-d97a-470b-8d73-1406c761d5c4","Type":"ContainerStarted","Data":"cecedc96b71117cd24eb623b236632a19d0dc6927118014a0d4cf31ac1214689"}
Sep 11 21:16:59.576599 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:59.576514    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx"
Sep 11 21:16:59.590175 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:59.590123    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn" podStartSLOduration=1.070637695 podStartE2EDuration="3.590107806s" totalImagesPullingTime="2.519470111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:16:59.589204507 +0000 UTC m=+686.293173797" watchObservedRunningTime="2026-09-11 21:16:59.590107806 +0000 UTC m=+686.294077097"
Sep 11 21:16:59.608244 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:59.608082    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx" podStartSLOduration=1.134649996 podStartE2EDuration="3.608062433s" totalImagesPullingTime="2.473412437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:16:59.606987423 +0000 UTC m=+686.310956724" watchObservedRunningTime="2026-09-11 21:16:59.608062433 +0000 UTC m=+686.312031724"
Sep 11 21:16:59.754685 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:59.754658    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-604c53"
Sep 11 21:16:59.833733 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:59.833699    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6w87\" (UniqueName: \"kubernetes.io/projected/d34e3b08-b395-4a82-80c7-3293ab09fcd2-kube-api-access-w6w87\") pod \"d34e3b08-b395-4a82-80c7-3293ab09fcd2\" (UID: \"d34e3b08-b395-4a82-80c7-3293ab09fcd2\") "
Sep 11 21:16:59.836192 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:59.836159    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d34e3b08-b395-4a82-80c7-3293ab09fcd2-kube-api-access-w6w87" (OuterVolumeSpecName: "kube-api-access-w6w87") pod "d34e3b08-b395-4a82-80c7-3293ab09fcd2" (UID: "d34e3b08-b395-4a82-80c7-3293ab09fcd2"). InnerVolumeSpecName "kube-api-access-w6w87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:16:59.927228 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:59.927195    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d34e3b08-b395-4a82-80c7-3293ab09fcd2" path="/var/lib/kubelet/pods/d34e3b08-b395-4a82-80c7-3293ab09fcd2/volumes"
Sep 11 21:16:59.934950 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:16:59.934751    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6w87\" (UniqueName: \"kubernetes.io/projected/d34e3b08-b395-4a82-80c7-3293ab09fcd2-kube-api-access-w6w87\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:17:00.585242 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:00.585066    2578 generic.go:356] "Generic (PLEG): container finished" podID="d34e3b08-b395-4a82-80c7-3293ab09fcd2" containerID="7f5929a0964346fd84defd198c359d0eefa460c36d5ae36c49daa93d6465b53d" exitCode=137
Sep 11 21:17:00.585242 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:00.585123    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7711-604c53"
Sep 11 21:17:00.585242 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:00.585170    2578 scope.go:117] "RemoveContainer" containerID="7f5929a0964346fd84defd198c359d0eefa460c36d5ae36c49daa93d6465b53d"
Sep 11 21:17:00.595582 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:00.595482    2578 scope.go:117] "RemoveContainer" containerID="7f5929a0964346fd84defd198c359d0eefa460c36d5ae36c49daa93d6465b53d"
Sep 11 21:17:00.595910 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:17:00.595880    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5929a0964346fd84defd198c359d0eefa460c36d5ae36c49daa93d6465b53d\": container with ID starting with 7f5929a0964346fd84defd198c359d0eefa460c36d5ae36c49daa93d6465b53d not found: ID does not exist" containerID="7f5929a0964346fd84defd198c359d0eefa460c36d5ae36c49daa93d6465b53d"
Sep 11 21:17:00.596026 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:00.595923    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5929a0964346fd84defd198c359d0eefa460c36d5ae36c49daa93d6465b53d"} err="failed to get container status \"7f5929a0964346fd84defd198c359d0eefa460c36d5ae36c49daa93d6465b53d\": rpc error: code = NotFound desc = could not find container \"7f5929a0964346fd84defd198c359d0eefa460c36d5ae36c49daa93d6465b53d\": container with ID starting with 7f5929a0964346fd84defd198c359d0eefa460c36d5ae36c49daa93d6465b53d not found: ID does not exist"
Sep 11 21:17:02.318607 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:02.318558    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:17:02.318917 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:02.318657    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:17:02.318917 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:02.318733    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:17:02.598912 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:02.598801    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm" event={"ID":"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed","Type":"ContainerStarted","Data":"782463b20a7e4fd210042a0618ed56d5eb979d4d8e2dbc6024fd5f54c49a1181"}
Sep 11 21:17:02.620922 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:02.620810    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm" podStartSLOduration=2.796006352 podStartE2EDuration="5.620793899s" totalImagesPullingTime="2.824787547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:17:02.618590481 +0000 UTC m=+689.322559772" watchObservedRunningTime="2026-09-11 21:17:02.620793899 +0000 UTC m=+689.324763189"
Sep 11 21:17:03.242227 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:03.242185    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:17:03.247351 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:03.247327    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:17:03.603676 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:03.603598    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:17:03.604728 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:03.604709    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:17:05.826718 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:05.826683    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-765a18-8576855845-4qwxt"]
Sep 11 21:17:05.830087 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:05.830070    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-765a18-8576855845-4qwxt"
Sep 11 21:17:05.833348 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:05.833326    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-765a18-dockercfg-s9pkp\""
Sep 11 21:17:05.833973 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:05.833958    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-765a18\""
Sep 11 21:17:05.842643 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:05.842612    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-765a18-8576855845-4qwxt"]
Sep 11 21:17:05.887317 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:05.887277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7388aa75-9611-4fab-a5b0-3b914a083584-plugins-config-volume\") pod \"payload-processing-e2e-autores-765a18-8576855845-4qwxt\" (UID: \"7388aa75-9611-4fab-a5b0-3b914a083584\") " pod="openshift-ingress/payload-processing-e2e-autores-765a18-8576855845-4qwxt"
Sep 11 21:17:05.887504 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:05.887368    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4hjv\" (UniqueName: \"kubernetes.io/projected/7388aa75-9611-4fab-a5b0-3b914a083584-kube-api-access-l4hjv\") pod \"payload-processing-e2e-autores-765a18-8576855845-4qwxt\" (UID: \"7388aa75-9611-4fab-a5b0-3b914a083584\") " pod="openshift-ingress/payload-processing-e2e-autores-765a18-8576855845-4qwxt"
Sep 11 21:17:05.988669 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:05.988629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4hjv\" (UniqueName: \"kubernetes.io/projected/7388aa75-9611-4fab-a5b0-3b914a083584-kube-api-access-l4hjv\") pod \"payload-processing-e2e-autores-765a18-8576855845-4qwxt\" (UID: \"7388aa75-9611-4fab-a5b0-3b914a083584\") " pod="openshift-ingress/payload-processing-e2e-autores-765a18-8576855845-4qwxt"
Sep 11 21:17:05.988864 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:05.988706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7388aa75-9611-4fab-a5b0-3b914a083584-plugins-config-volume\") pod \"payload-processing-e2e-autores-765a18-8576855845-4qwxt\" (UID: \"7388aa75-9611-4fab-a5b0-3b914a083584\") " pod="openshift-ingress/payload-processing-e2e-autores-765a18-8576855845-4qwxt"
Sep 11 21:17:05.989451 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:05.989432    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7388aa75-9611-4fab-a5b0-3b914a083584-plugins-config-volume\") pod \"payload-processing-e2e-autores-765a18-8576855845-4qwxt\" (UID: \"7388aa75-9611-4fab-a5b0-3b914a083584\") " pod="openshift-ingress/payload-processing-e2e-autores-765a18-8576855845-4qwxt"
Sep 11 21:17:06.002185 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:06.002153    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4hjv\" (UniqueName: \"kubernetes.io/projected/7388aa75-9611-4fab-a5b0-3b914a083584-kube-api-access-l4hjv\") pod \"payload-processing-e2e-autores-765a18-8576855845-4qwxt\" (UID: \"7388aa75-9611-4fab-a5b0-3b914a083584\") " pod="openshift-ingress/payload-processing-e2e-autores-765a18-8576855845-4qwxt"
Sep 11 21:17:06.139950 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:06.139865    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-765a18-8576855845-4qwxt"
Sep 11 21:17:06.415417 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:06.415333    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-765a18-8576855845-4qwxt"]
Sep 11 21:17:06.417999 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:17:06.417961    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7388aa75_9611_4fab_a5b0_3b914a083584.slice/crio-52365cc7e25cb58f4be243b94bf68b0f1df62568baefd373c3df7c5b2f83bde3 WatchSource:0}: Error finding container 52365cc7e25cb58f4be243b94bf68b0f1df62568baefd373c3df7c5b2f83bde3: Status 404 returned error can't find the container with id 52365cc7e25cb58f4be243b94bf68b0f1df62568baefd373c3df7c5b2f83bde3
Sep 11 21:17:06.618268 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:06.618236    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-765a18-8576855845-4qwxt" event={"ID":"7388aa75-9611-4fab-a5b0-3b914a083584","Type":"ContainerStarted","Data":"706cd964c45fcf393d7a07e0817d2d5673da83b0ece5f40386c98ea72d4253ee"}
Sep 11 21:17:06.618473 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:06.618297    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-765a18-8576855845-4qwxt" event={"ID":"7388aa75-9611-4fab-a5b0-3b914a083584","Type":"ContainerStarted","Data":"52365cc7e25cb58f4be243b94bf68b0f1df62568baefd373c3df7c5b2f83bde3"}
Sep 11 21:17:06.618473 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:06.618338    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-765a18-8576855845-4qwxt"
Sep 11 21:17:06.638283 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:06.638223    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-765a18-8576855845-4qwxt" podStartSLOduration=1.6382060680000001 podStartE2EDuration="1.638206068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:17:06.637402779 +0000 UTC m=+693.341372059" watchObservedRunningTime="2026-09-11 21:17:06.638206068 +0000 UTC m=+693.342175371"
Sep 11 21:17:08.818226 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:08.818188    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m"]
Sep 11 21:17:08.821463 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:08.821447    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m"
Sep 11 21:17:08.823857 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:08.823837    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8dbad4ec\""
Sep 11 21:17:08.837204 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:08.837178    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m"]
Sep 11 21:17:08.913250 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:08.913210    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac75cbb-3a45-437e-94c7-8651b6bb39e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m\" (UID: \"fac75cbb-3a45-437e-94c7-8651b6bb39e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m"
Sep 11 21:17:08.913250 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:08.913254    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85ct9\" (UniqueName: \"kubernetes.io/projected/fac75cbb-3a45-437e-94c7-8651b6bb39e9-kube-api-access-85ct9\") pod \"payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m\" (UID: \"fac75cbb-3a45-437e-94c7-8651b6bb39e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m"
Sep 11 21:17:09.014186 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:09.014139    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac75cbb-3a45-437e-94c7-8651b6bb39e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m\" (UID: \"fac75cbb-3a45-437e-94c7-8651b6bb39e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m"
Sep 11 21:17:09.014439 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:09.014407    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85ct9\" (UniqueName: \"kubernetes.io/projected/fac75cbb-3a45-437e-94c7-8651b6bb39e9-kube-api-access-85ct9\") pod \"payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m\" (UID: \"fac75cbb-3a45-437e-94c7-8651b6bb39e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m"
Sep 11 21:17:09.014806 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:09.014786    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac75cbb-3a45-437e-94c7-8651b6bb39e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m\" (UID: \"fac75cbb-3a45-437e-94c7-8651b6bb39e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m"
Sep 11 21:17:09.022349 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:09.022326    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85ct9\" (UniqueName: \"kubernetes.io/projected/fac75cbb-3a45-437e-94c7-8651b6bb39e9-kube-api-access-85ct9\") pod \"payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m\" (UID: \"fac75cbb-3a45-437e-94c7-8651b6bb39e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m"
Sep 11 21:17:09.131273 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:09.131181    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m"
Sep 11 21:17:09.404707 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:09.404683    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m"]
Sep 11 21:17:09.406765 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:17:09.406736    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac75cbb_3a45_437e_94c7_8651b6bb39e9.slice/crio-9c3014ce7822ff7f6c727c0ffb224fd19d33a80470e098eaebb6558006273209 WatchSource:0}: Error finding container 9c3014ce7822ff7f6c727c0ffb224fd19d33a80470e098eaebb6558006273209: Status 404 returned error can't find the container with id 9c3014ce7822ff7f6c727c0ffb224fd19d33a80470e098eaebb6558006273209
Sep 11 21:17:09.633494 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:09.633460    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m" event={"ID":"fac75cbb-3a45-437e-94c7-8651b6bb39e9","Type":"ContainerStarted","Data":"88dab615518964a01db1d037a050c964e79f16226318b84e355ffae97f072151"}
Sep 11 21:17:09.633690 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:09.633519    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m" event={"ID":"fac75cbb-3a45-437e-94c7-8651b6bb39e9","Type":"ContainerStarted","Data":"9c3014ce7822ff7f6c727c0ffb224fd19d33a80470e098eaebb6558006273209"}
Sep 11 21:17:09.633690 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:09.633589    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m"
Sep 11 21:17:09.650608 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:09.650387    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m" podStartSLOduration=1.650366962 podStartE2EDuration="1.650366962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:17:09.650157221 +0000 UTC m=+696.354126515" watchObservedRunningTime="2026-09-11 21:17:09.650366962 +0000 UTC m=+696.354336253"
Sep 11 21:17:10.587542 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:10.587512    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn"
Sep 11 21:17:10.588031 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:10.587585    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx"
Sep 11 21:17:15.153908 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.153870    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982"]
Sep 11 21:17:15.157180 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.157164    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982"
Sep 11 21:17:15.159401 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.159373    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2b95f8-dockercfg-ztjcm\""
Sep 11 21:17:15.159544 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.159420    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2b95f8\""
Sep 11 21:17:15.168197 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.168174    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982"]
Sep 11 21:17:15.191070 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.191037    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp"]
Sep 11 21:17:15.194097 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.194080    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp"
Sep 11 21:17:15.206170 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.206146    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp"]
Sep 11 21:17:15.271146 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.271111    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx6fm\" (UniqueName: \"kubernetes.io/projected/55889c70-06c0-4fc3-adc1-c122113be4ae-kube-api-access-gx6fm\") pod \"payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp\" (UID: \"55889c70-06c0-4fc3-adc1-c122113be4ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp"
Sep 11 21:17:15.271146 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.271148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rd79\" (UniqueName: \"kubernetes.io/projected/a9ddb125-b7ae-4f13-8e77-f8a18107c88d-kube-api-access-6rd79\") pod \"payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982\" (UID: \"a9ddb125-b7ae-4f13-8e77-f8a18107c88d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982"
Sep 11 21:17:15.271330 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.271171    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55889c70-06c0-4fc3-adc1-c122113be4ae-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp\" (UID: \"55889c70-06c0-4fc3-adc1-c122113be4ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp"
Sep 11 21:17:15.271330 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.271257    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ddb125-b7ae-4f13-8e77-f8a18107c88d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982\" (UID: \"a9ddb125-b7ae-4f13-8e77-f8a18107c88d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982"
Sep 11 21:17:15.352084 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.352048    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:17:15.371807 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.371774    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx6fm\" (UniqueName: \"kubernetes.io/projected/55889c70-06c0-4fc3-adc1-c122113be4ae-kube-api-access-gx6fm\") pod \"payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp\" (UID: \"55889c70-06c0-4fc3-adc1-c122113be4ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp"
Sep 11 21:17:15.371943 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.371837    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rd79\" (UniqueName: \"kubernetes.io/projected/a9ddb125-b7ae-4f13-8e77-f8a18107c88d-kube-api-access-6rd79\") pod \"payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982\" (UID: \"a9ddb125-b7ae-4f13-8e77-f8a18107c88d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982"
Sep 11 21:17:15.371943 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.371875    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55889c70-06c0-4fc3-adc1-c122113be4ae-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp\" (UID: \"55889c70-06c0-4fc3-adc1-c122113be4ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp"
Sep 11 21:17:15.371943 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.371905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ddb125-b7ae-4f13-8e77-f8a18107c88d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982\" (UID: \"a9ddb125-b7ae-4f13-8e77-f8a18107c88d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982"
Sep 11 21:17:15.372629 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.372608    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55889c70-06c0-4fc3-adc1-c122113be4ae-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp\" (UID: \"55889c70-06c0-4fc3-adc1-c122113be4ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp"
Sep 11 21:17:15.372673 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.372654    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ddb125-b7ae-4f13-8e77-f8a18107c88d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982\" (UID: \"a9ddb125-b7ae-4f13-8e77-f8a18107c88d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982"
Sep 11 21:17:15.379961 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.379939    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rd79\" (UniqueName: \"kubernetes.io/projected/a9ddb125-b7ae-4f13-8e77-f8a18107c88d-kube-api-access-6rd79\") pod \"payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982\" (UID: \"a9ddb125-b7ae-4f13-8e77-f8a18107c88d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982"
Sep 11 21:17:15.380071 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.380001    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx6fm\" (UniqueName: \"kubernetes.io/projected/55889c70-06c0-4fc3-adc1-c122113be4ae-kube-api-access-gx6fm\") pod \"payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp\" (UID: \"55889c70-06c0-4fc3-adc1-c122113be4ae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp"
Sep 11 21:17:15.467486 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.467451    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982"
Sep 11 21:17:15.504440 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.504396    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp"
Sep 11 21:17:15.746499 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.746474    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982"]
Sep 11 21:17:15.748805 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:17:15.748775    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ddb125_b7ae_4f13_8e77_f8a18107c88d.slice/crio-7e2bbbd602d09da001702e437409010b1b97a137422955b7277c135b2d3d8893 WatchSource:0}: Error finding container 7e2bbbd602d09da001702e437409010b1b97a137422955b7277c135b2d3d8893: Status 404 returned error can't find the container with id 7e2bbbd602d09da001702e437409010b1b97a137422955b7277c135b2d3d8893
Sep 11 21:17:15.800704 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:15.800660    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp"]
Sep 11 21:17:15.804514 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:17:15.804487    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55889c70_06c0_4fc3_adc1_c122113be4ae.slice/crio-ae5f84d7e8bcfcd09c7def7e58c785c4b855e9541d2092a06a755b2d6b35694c WatchSource:0}: Error finding container ae5f84d7e8bcfcd09c7def7e58c785c4b855e9541d2092a06a755b2d6b35694c: Status 404 returned error can't find the container with id ae5f84d7e8bcfcd09c7def7e58c785c4b855e9541d2092a06a755b2d6b35694c
Sep 11 21:17:16.667855 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:16.667789    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982" event={"ID":"a9ddb125-b7ae-4f13-8e77-f8a18107c88d","Type":"ContainerStarted","Data":"9853d7df85e338e7c901ed4f1f7bb800367060a2577e66e4102729e942ae2a52"}
Sep 11 21:17:16.668450 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:16.668413    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982" event={"ID":"a9ddb125-b7ae-4f13-8e77-f8a18107c88d","Type":"ContainerStarted","Data":"7e2bbbd602d09da001702e437409010b1b97a137422955b7277c135b2d3d8893"}
Sep 11 21:17:16.668579 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:16.668459    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982"
Sep 11 21:17:16.669937 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:16.669903    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp" event={"ID":"55889c70-06c0-4fc3-adc1-c122113be4ae","Type":"ContainerStarted","Data":"9c033aee13982f20d72e194645b4078f5b7578e02ac395c3c8c857943bd38b26"}
Sep 11 21:17:16.670057 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:16.669962    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp"
Sep 11 21:17:16.670057 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:16.669986    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp" event={"ID":"55889c70-06c0-4fc3-adc1-c122113be4ae","Type":"ContainerStarted","Data":"ae5f84d7e8bcfcd09c7def7e58c785c4b855e9541d2092a06a755b2d6b35694c"}
Sep 11 21:17:16.686567 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:16.686496    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982" podStartSLOduration=1.686480129 podStartE2EDuration="1.686480129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:17:16.684179052 +0000 UTC m=+703.388148341" watchObservedRunningTime="2026-09-11 21:17:16.686480129 +0000 UTC m=+703.390449419"
Sep 11 21:17:16.701592 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:16.701530    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp" podStartSLOduration=1.701507508 podStartE2EDuration="1.701507508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:17:16.700717168 +0000 UTC m=+703.404686458" watchObservedRunningTime="2026-09-11 21:17:16.701507508 +0000 UTC m=+703.405476805"
Sep 11 21:17:17.625291 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:17.625247    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-765a18-8576855845-4qwxt"
Sep 11 21:17:20.642198 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:20.642166    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m"
Sep 11 21:17:25.710660 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:25.710624    2578 generic.go:356] "Generic (PLEG): container finished" podID="e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" containerID="782463b20a7e4fd210042a0618ed56d5eb979d4d8e2dbc6024fd5f54c49a1181" exitCode=0
Sep 11 21:17:25.711087 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:25.710694    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm" event={"ID":"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed","Type":"ContainerDied","Data":"782463b20a7e4fd210042a0618ed56d5eb979d4d8e2dbc6024fd5f54c49a1181"}
Sep 11 21:17:25.711127 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:25.711114    2578 scope.go:117] "RemoveContainer" containerID="782463b20a7e4fd210042a0618ed56d5eb979d4d8e2dbc6024fd5f54c49a1181"
Sep 11 21:17:25.712219 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:25.712188    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:17:25.712311 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:25.712255    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:17:25.712311 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:25.712288    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:17:26.717176 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:26.717144    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm" event={"ID":"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed","Type":"ContainerStarted","Data":"3b0582bb94dfeaef17e689a4564eccaa9b16248257f45a7ca46444ee84095ef6"}
Sep 11 21:17:27.241690 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:27.241647    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:17:27.242003 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:27.241812    2578 patch_prober.go:28] interesting pod/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 11 21:17:27.242167 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:27.242136    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm" podUID="e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 11 21:17:27.676952 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:27.676925    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp"
Sep 11 21:17:28.242301 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:28.242261    2578 patch_prober.go:28] interesting pod/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 11 21:17:28.242864 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:28.242802    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm" podUID="e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 11 21:17:28.681953 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:28.681922    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982"
Sep 11 21:17:29.241602 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:29.241558    2578 patch_prober.go:28] interesting pod/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 11 21:17:29.241866 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:29.241624    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm" podUID="e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 11 21:17:30.242217 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:30.242166    2578 patch_prober.go:28] interesting pod/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 11 21:17:30.242703 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:30.242262    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm" podUID="e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 11 21:17:31.242578 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:31.242531    2578 patch_prober.go:28] interesting pod/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 11 21:17:31.243079 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:31.242604    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm" podUID="e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 11 21:17:32.245846 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:32.245792    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:17:32.246306 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:32.246205    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:17:32.246964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:32.246946    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:17:34.800049 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:34.800013    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8dbad4ec-457bz"]
Sep 11 21:17:34.809222 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:34.809196    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8dbad4ec-457bz"
Sep 11 21:17:34.812881 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:34.812849    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:17:34.813055 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:34.812851    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfm9g\""
Sep 11 21:17:34.813201 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:34.813182    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8dbad4ec-457bz"]
Sep 11 21:17:34.949173 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:34.949130    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613db705-650c-499d-94a7-19c709fb1c64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8dbad4ec-457bz\" (UID: \"613db705-650c-499d-94a7-19c709fb1c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8dbad4ec-457bz"
Sep 11 21:17:35.050127 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:35.050038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613db705-650c-499d-94a7-19c709fb1c64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8dbad4ec-457bz\" (UID: \"613db705-650c-499d-94a7-19c709fb1c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8dbad4ec-457bz"
Sep 11 21:17:35.050674 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:35.050654    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613db705-650c-499d-94a7-19c709fb1c64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8dbad4ec-457bz\" (UID: \"613db705-650c-499d-94a7-19c709fb1c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8dbad4ec-457bz"
Sep 11 21:17:35.121659 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:35.121625    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8dbad4ec-457bz"
Sep 11 21:17:35.397076 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:35.396903    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8dbad4ec-457bz"]
Sep 11 21:17:35.399698 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:17:35.399671    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613db705_650c_499d_94a7_19c709fb1c64.slice/crio-874aca4afb77c55ec3a3432b454bc36552d5daaf98978cd4a4fc2ea4bff8f392 WatchSource:0}: Error finding container 874aca4afb77c55ec3a3432b454bc36552d5daaf98978cd4a4fc2ea4bff8f392: Status 404 returned error can't find the container with id 874aca4afb77c55ec3a3432b454bc36552d5daaf98978cd4a4fc2ea4bff8f392
Sep 11 21:17:35.769010 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:35.768977    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8dbad4ec-457bz" event={"ID":"613db705-650c-499d-94a7-19c709fb1c64","Type":"ContainerStarted","Data":"874aca4afb77c55ec3a3432b454bc36552d5daaf98978cd4a4fc2ea4bff8f392"}
Sep 11 21:17:38.786327 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:38.786296    2578 generic.go:356] "Generic (PLEG): container finished" podID="613db705-650c-499d-94a7-19c709fb1c64" containerID="4614e5f9c432316ae8271c6803a0f256dc0be148feb05927e23eb36a84f79210" exitCode=0
Sep 11 21:17:38.786717 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:38.786389    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8dbad4ec-457bz" event={"ID":"613db705-650c-499d-94a7-19c709fb1c64","Type":"ContainerDied","Data":"4614e5f9c432316ae8271c6803a0f256dc0be148feb05927e23eb36a84f79210"}
Sep 11 21:17:39.019126 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:39.019094    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:17:39.919425 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:39.919401    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8dbad4ec-457bz"
Sep 11 21:17:40.091569 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:40.091490    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613db705-650c-499d-94a7-19c709fb1c64-maas-api-service-ca\") pod \"613db705-650c-499d-94a7-19c709fb1c64\" (UID: \"613db705-650c-499d-94a7-19c709fb1c64\") "
Sep 11 21:17:40.091903 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:40.091810    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613db705-650c-499d-94a7-19c709fb1c64-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "613db705-650c-499d-94a7-19c709fb1c64" (UID: "613db705-650c-499d-94a7-19c709fb1c64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:17:40.192663 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:40.192614    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613db705-650c-499d-94a7-19c709fb1c64-maas-api-service-ca\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:17:40.796904 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:40.796873    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8dbad4ec-457bz"
Sep 11 21:17:40.797100 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:40.796865    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8dbad4ec-457bz" event={"ID":"613db705-650c-499d-94a7-19c709fb1c64","Type":"ContainerDied","Data":"874aca4afb77c55ec3a3432b454bc36552d5daaf98978cd4a4fc2ea4bff8f392"}
Sep 11 21:17:40.797100 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:40.797019    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="874aca4afb77c55ec3a3432b454bc36552d5daaf98978cd4a4fc2ea4bff8f392"
Sep 11 21:17:46.054067 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:46.054021    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m"]
Sep 11 21:17:46.054691 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:46.054299    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m" podUID="fac75cbb-3a45-437e-94c7-8651b6bb39e9" containerName="payload-pre-processing" containerID="cri-o://88dab615518964a01db1d037a050c964e79f16226318b84e355ffae97f072151" gracePeriod=30
Sep 11 21:17:46.240620 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:46.240582    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8dbad4ec-457bz"]
Sep 11 21:17:46.247372 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:46.247347    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8dbad4ec-457bz"]
Sep 11 21:17:46.302482 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:46.302457    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m"
Sep 11 21:17:46.372847 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:46.372742    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"]
Sep 11 21:17:46.372991 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:46.372964    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm" podUID="e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" containerName="istio-proxy" containerID="cri-o://3b0582bb94dfeaef17e689a4564eccaa9b16248257f45a7ca46444ee84095ef6" gracePeriod=30
Sep 11 21:17:46.447811 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:46.447780    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac75cbb-3a45-437e-94c7-8651b6bb39e9-plugins-config-volume\") pod \"fac75cbb-3a45-437e-94c7-8651b6bb39e9\" (UID: \"fac75cbb-3a45-437e-94c7-8651b6bb39e9\") "
Sep 11 21:17:46.448009 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:46.447861    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85ct9\" (UniqueName: \"kubernetes.io/projected/fac75cbb-3a45-437e-94c7-8651b6bb39e9-kube-api-access-85ct9\") pod \"fac75cbb-3a45-437e-94c7-8651b6bb39e9\" (UID: \"fac75cbb-3a45-437e-94c7-8651b6bb39e9\") "
Sep 11 21:17:46.448192 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:46.448165    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fac75cbb-3a45-437e-94c7-8651b6bb39e9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fac75cbb-3a45-437e-94c7-8651b6bb39e9" (UID: "fac75cbb-3a45-437e-94c7-8651b6bb39e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:17:46.450042 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:46.450021    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac75cbb-3a45-437e-94c7-8651b6bb39e9-kube-api-access-85ct9" (OuterVolumeSpecName: "kube-api-access-85ct9") pod "fac75cbb-3a45-437e-94c7-8651b6bb39e9" (UID: "fac75cbb-3a45-437e-94c7-8651b6bb39e9"). InnerVolumeSpecName "kube-api-access-85ct9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:17:46.549295 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:46.549259    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac75cbb-3a45-437e-94c7-8651b6bb39e9-plugins-config-volume\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:17:46.549295 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:46.549287    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85ct9\" (UniqueName: \"kubernetes.io/projected/fac75cbb-3a45-437e-94c7-8651b6bb39e9-kube-api-access-85ct9\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:17:46.824842 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:46.824789    2578 generic.go:356] "Generic (PLEG): container finished" podID="fac75cbb-3a45-437e-94c7-8651b6bb39e9" containerID="88dab615518964a01db1d037a050c964e79f16226318b84e355ffae97f072151" exitCode=0
Sep 11 21:17:46.825019 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:46.824863    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m"
Sep 11 21:17:46.825019 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:46.824878    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m" event={"ID":"fac75cbb-3a45-437e-94c7-8651b6bb39e9","Type":"ContainerDied","Data":"88dab615518964a01db1d037a050c964e79f16226318b84e355ffae97f072151"}
Sep 11 21:17:46.825019 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:46.824933    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m" event={"ID":"fac75cbb-3a45-437e-94c7-8651b6bb39e9","Type":"ContainerDied","Data":"9c3014ce7822ff7f6c727c0ffb224fd19d33a80470e098eaebb6558006273209"}
Sep 11 21:17:46.825019 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:46.824956    2578 scope.go:117] "RemoveContainer" containerID="88dab615518964a01db1d037a050c964e79f16226318b84e355ffae97f072151"
Sep 11 21:17:46.834001 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:46.833981    2578 scope.go:117] "RemoveContainer" containerID="88dab615518964a01db1d037a050c964e79f16226318b84e355ffae97f072151"
Sep 11 21:17:46.834293 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:17:46.834274    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88dab615518964a01db1d037a050c964e79f16226318b84e355ffae97f072151\": container with ID starting with 88dab615518964a01db1d037a050c964e79f16226318b84e355ffae97f072151 not found: ID does not exist" containerID="88dab615518964a01db1d037a050c964e79f16226318b84e355ffae97f072151"
Sep 11 21:17:46.834360 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:46.834307    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88dab615518964a01db1d037a050c964e79f16226318b84e355ffae97f072151"} err="failed to get container status \"88dab615518964a01db1d037a050c964e79f16226318b84e355ffae97f072151\": rpc error: code = NotFound desc = could not find container \"88dab615518964a01db1d037a050c964e79f16226318b84e355ffae97f072151\": container with ID starting with 88dab615518964a01db1d037a050c964e79f16226318b84e355ffae97f072151 not found: ID does not exist"
Sep 11 21:17:46.852879 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:46.852853    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m"]
Sep 11 21:17:46.861869 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:46.861842    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8dbad4ec-5f6b69699d-fkk6m"]
Sep 11 21:17:47.248332 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.248285    2578 patch_prober.go:28] interesting pod/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 21:17:47.248717 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.248347    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm" podUID="e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 21:17:47.605812 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.605729    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"]
Sep 11 21:17:47.606101 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.606088    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fac75cbb-3a45-437e-94c7-8651b6bb39e9" containerName="payload-pre-processing"
Sep 11 21:17:47.606139 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.606103    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fac75cbb-3a45-437e-94c7-8651b6bb39e9" containerName="payload-pre-processing"
Sep 11 21:17:47.606139 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.606114    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="613db705-650c-499d-94a7-19c709fb1c64" containerName="revoke-keys"
Sep 11 21:17:47.606139 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.606121    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="613db705-650c-499d-94a7-19c709fb1c64" containerName="revoke-keys"
Sep 11 21:17:47.606244 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.606185    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="613db705-650c-499d-94a7-19c709fb1c64" containerName="revoke-keys"
Sep 11 21:17:47.606244 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.606196    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fac75cbb-3a45-437e-94c7-8651b6bb39e9" containerName="payload-pre-processing"
Sep 11 21:17:47.610253 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.610237    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.617284 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.617269    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-2d3d3219-gw-openshift-default-dockercfg-ggmql\""
Sep 11 21:17:47.636683 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.636658    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"]
Sep 11 21:17:47.759407 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.759362    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-istio-envoy\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.759612 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.759435    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/644cf0ad-7860-4448-937c-d6e195947628-istiod-ca-cert\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.759612 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.759470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-istio-data\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.759612 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.759499    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/644cf0ad-7860-4448-937c-d6e195947628-istio-podinfo\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.759612 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.759583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhdc2\" (UniqueName: \"kubernetes.io/projected/644cf0ad-7860-4448-937c-d6e195947628-kube-api-access-fhdc2\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.759812 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.759634    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-workload-socket\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.759812 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.759663    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-credential-socket\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.759812 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.759709    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/644cf0ad-7860-4448-937c-d6e195947628-istio-token\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.759812 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.759737    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-workload-certs\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.860240 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.860150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/644cf0ad-7860-4448-937c-d6e195947628-istiod-ca-cert\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.860240 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.860196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-istio-data\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.860450 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.860321    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/644cf0ad-7860-4448-937c-d6e195947628-istio-podinfo\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.860450 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.860364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhdc2\" (UniqueName: \"kubernetes.io/projected/644cf0ad-7860-4448-937c-d6e195947628-kube-api-access-fhdc2\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.860450 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.860384    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-workload-socket\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.860577 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.860536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-credential-socket\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.860626 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.860607    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/644cf0ad-7860-4448-937c-d6e195947628-istio-token\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.860683 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.860640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-workload-certs\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.860738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.860695    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-istio-envoy\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.860796 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.860637    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-istio-data\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.860997 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.860979    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/644cf0ad-7860-4448-937c-d6e195947628-istiod-ca-cert\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.861076 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.861026    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-credential-socket\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.861076 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.861057    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-workload-socket\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.861155 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.861075    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-workload-certs\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.862836 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.862798    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/644cf0ad-7860-4448-937c-d6e195947628-istio-podinfo\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.862971 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.862956    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-istio-envoy\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.881285 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.881247    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/644cf0ad-7860-4448-937c-d6e195947628-istio-token\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.881285 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.881280    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhdc2\" (UniqueName: \"kubernetes.io/projected/644cf0ad-7860-4448-937c-d6e195947628-kube-api-access-fhdc2\") pod \"e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") " pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.919839 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.919797    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:47.927761 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.927723    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="613db705-650c-499d-94a7-19c709fb1c64" path="/var/lib/kubelet/pods/613db705-650c-499d-94a7-19c709fb1c64/volumes"
Sep 11 21:17:47.928128 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:47.928109    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fac75cbb-3a45-437e-94c7-8651b6bb39e9" path="/var/lib/kubelet/pods/fac75cbb-3a45-437e-94c7-8651b6bb39e9/volumes"
Sep 11 21:17:48.198847 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:48.198801    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"]
Sep 11 21:17:48.201093 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:17:48.201071    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644cf0ad_7860_4448_937c_d6e195947628.slice/crio-c1945820570445d6c3f03ba6cd4e843c6593d8554095680e50f19fa4cc335473 WatchSource:0}: Error finding container c1945820570445d6c3f03ba6cd4e843c6593d8554095680e50f19fa4cc335473: Status 404 returned error can't find the container with id c1945820570445d6c3f03ba6cd4e843c6593d8554095680e50f19fa4cc335473
Sep 11 21:17:48.203400 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:48.203366    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:17:48.203478 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:48.203429    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:17:48.203478 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:48.203455    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:17:48.841619 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:48.841559    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z" event={"ID":"644cf0ad-7860-4448-937c-d6e195947628","Type":"ContainerStarted","Data":"a026221b0b7828ca9fcc4b4685da2777176b7a0702770a86f0ace20c191f42fe"}
Sep 11 21:17:48.842079 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:48.841743    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z" event={"ID":"644cf0ad-7860-4448-937c-d6e195947628","Type":"ContainerStarted","Data":"c1945820570445d6c3f03ba6cd4e843c6593d8554095680e50f19fa4cc335473"}
Sep 11 21:17:48.866382 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:48.866334    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z" podStartSLOduration=1.866318336 podStartE2EDuration="1.866318336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:17:48.86478507 +0000 UTC m=+735.568754359" watchObservedRunningTime="2026-09-11 21:17:48.866318336 +0000 UTC m=+735.570287626"
Sep 11 21:17:48.920339 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:48.920308    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:49.927625 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:49.927597    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:50.850614 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:50.850579    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:50.851575 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:50.851555    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:17:51.616883 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.616862    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:17:51.694564 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.694533    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-workload-certs\") pod \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") "
Sep 11 21:17:51.694722 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.694572    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvrpn\" (UniqueName: \"kubernetes.io/projected/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-kube-api-access-xvrpn\") pod \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") "
Sep 11 21:17:51.694722 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.694596    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istiod-ca-cert\") pod \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") "
Sep 11 21:17:51.694722 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.694620    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istio-token\") pod \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") "
Sep 11 21:17:51.694916 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.694736    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istio-podinfo\") pod \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") "
Sep 11 21:17:51.694916 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.694778    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-credential-socket\") pod \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") "
Sep 11 21:17:51.694916 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.694841    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istio-data\") pod \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") "
Sep 11 21:17:51.694916 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.694879    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istio-envoy\") pod \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") "
Sep 11 21:17:51.695112 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.694928    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-workload-socket\") pod \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\" (UID: \"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed\") "
Sep 11 21:17:51.695112 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.694930    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" (UID: "e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:17:51.695112 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.695057    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" (UID: "e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:17:51.695112 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.695073    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" (UID: "e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:17:51.695296 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.695150    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istio-data" (OuterVolumeSpecName: "istio-data") pod "e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" (UID: "e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:17:51.695296 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.695244    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-credential-socket\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:17:51.695296 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.695248    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" (UID: "e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:17:51.695296 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.695263    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istio-data\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:17:51.695296 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.695278    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-workload-certs\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:17:51.695296 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.695290    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istiod-ca-cert\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:17:51.697008 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.696979    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" (UID: "e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:17:51.697117 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.696992    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" (UID: "e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:17:51.697220 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.697204    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istio-token" (OuterVolumeSpecName: "istio-token") pod "e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" (UID: "e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:17:51.697279 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.697228    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-kube-api-access-xvrpn" (OuterVolumeSpecName: "kube-api-access-xvrpn") pod "e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" (UID: "e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed"). InnerVolumeSpecName "kube-api-access-xvrpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:17:51.795636 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.795601    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istio-envoy\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:17:51.795636 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.795629    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-workload-socket\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:17:51.795636 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.795640    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvrpn\" (UniqueName: \"kubernetes.io/projected/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-kube-api-access-xvrpn\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:17:51.795901 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.795649    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istio-token\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:17:51.795901 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.795657    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed-istio-podinfo\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:17:51.856682 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.856646    2578 generic.go:356] "Generic (PLEG): container finished" podID="e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" containerID="3b0582bb94dfeaef17e689a4564eccaa9b16248257f45a7ca46444ee84095ef6" exitCode=0
Sep 11 21:17:51.856816 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.856704    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"
Sep 11 21:17:51.856816 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.856735    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm" event={"ID":"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed","Type":"ContainerDied","Data":"3b0582bb94dfeaef17e689a4564eccaa9b16248257f45a7ca46444ee84095ef6"}
Sep 11 21:17:51.856816 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.856789    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm" event={"ID":"e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed","Type":"ContainerDied","Data":"1aec0e9d1709e44f3078cfa184c25c25d0434c82149508015791c086a2f5e634"}
Sep 11 21:17:51.856816 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.856811    2578 scope.go:117] "RemoveContainer" containerID="3b0582bb94dfeaef17e689a4564eccaa9b16248257f45a7ca46444ee84095ef6"
Sep 11 21:17:51.867599 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.867576    2578 scope.go:117] "RemoveContainer" containerID="782463b20a7e4fd210042a0618ed56d5eb979d4d8e2dbc6024fd5f54c49a1181"
Sep 11 21:17:51.876743 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.876722    2578 scope.go:117] "RemoveContainer" containerID="3b0582bb94dfeaef17e689a4564eccaa9b16248257f45a7ca46444ee84095ef6"
Sep 11 21:17:51.877073 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:17:51.877056    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0582bb94dfeaef17e689a4564eccaa9b16248257f45a7ca46444ee84095ef6\": container with ID starting with 3b0582bb94dfeaef17e689a4564eccaa9b16248257f45a7ca46444ee84095ef6 not found: ID does not exist" containerID="3b0582bb94dfeaef17e689a4564eccaa9b16248257f45a7ca46444ee84095ef6"
Sep 11 21:17:51.877132 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.877082    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0582bb94dfeaef17e689a4564eccaa9b16248257f45a7ca46444ee84095ef6"} err="failed to get container status \"3b0582bb94dfeaef17e689a4564eccaa9b16248257f45a7ca46444ee84095ef6\": rpc error: code = NotFound desc = could not find container \"3b0582bb94dfeaef17e689a4564eccaa9b16248257f45a7ca46444ee84095ef6\": container with ID starting with 3b0582bb94dfeaef17e689a4564eccaa9b16248257f45a7ca46444ee84095ef6 not found: ID does not exist"
Sep 11 21:17:51.877132 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.877099    2578 scope.go:117] "RemoveContainer" containerID="782463b20a7e4fd210042a0618ed56d5eb979d4d8e2dbc6024fd5f54c49a1181"
Sep 11 21:17:51.877377 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:17:51.877358    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"782463b20a7e4fd210042a0618ed56d5eb979d4d8e2dbc6024fd5f54c49a1181\": container with ID starting with 782463b20a7e4fd210042a0618ed56d5eb979d4d8e2dbc6024fd5f54c49a1181 not found: ID does not exist" containerID="782463b20a7e4fd210042a0618ed56d5eb979d4d8e2dbc6024fd5f54c49a1181"
Sep 11 21:17:51.877430 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.877383    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"782463b20a7e4fd210042a0618ed56d5eb979d4d8e2dbc6024fd5f54c49a1181"} err="failed to get container status \"782463b20a7e4fd210042a0618ed56d5eb979d4d8e2dbc6024fd5f54c49a1181\": rpc error: code = NotFound desc = could not find container \"782463b20a7e4fd210042a0618ed56d5eb979d4d8e2dbc6024fd5f54c49a1181\": container with ID starting with 782463b20a7e4fd210042a0618ed56d5eb979d4d8e2dbc6024fd5f54c49a1181 not found: ID does not exist"
Sep 11 21:17:51.895331 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.895299    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"]
Sep 11 21:17:51.902247 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.902220    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8dbad4ec-openshift-default-68c79d4446-tbdlm"]
Sep 11 21:17:51.927471 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:51.927444    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" path="/var/lib/kubelet/pods/e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed/volumes"
Sep 11 21:17:52.608904 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.608868    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"]
Sep 11 21:17:52.609198 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.609185    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" containerName="istio-proxy"
Sep 11 21:17:52.609240 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.609200    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" containerName="istio-proxy"
Sep 11 21:17:52.609274 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.609260    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" containerName="istio-proxy"
Sep 11 21:17:52.613500 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.613482    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:17:52.616447 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.616410    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ea5e33\"/\"openshift-service-ca.crt\""
Sep 11 21:17:52.616611 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.616483    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ea5e33\"/\"kube-root-ca.crt\""
Sep 11 21:17:52.616611 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.616491    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ea5e33\"/\"simulator-w0-ea5e33-kserve-self-signed-certs\""
Sep 11 21:17:52.616611 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.616517    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ea5e33\"/\"default-dockercfg-bvdgp\""
Sep 11 21:17:52.623760 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.623733    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"]
Sep 11 21:17:52.702043 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.702006    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-tmp-dir\") pod \"simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7\" (UID: \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\") " pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:17:52.702185 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.702086    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-dshm\") pod \"simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7\" (UID: \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\") " pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:17:52.702185 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.702160    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-home\") pod \"simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7\" (UID: \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\") " pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:17:52.702281 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.702186    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qclcl\" (UniqueName: \"kubernetes.io/projected/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-kube-api-access-qclcl\") pod \"simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7\" (UID: \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\") " pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:17:52.702281 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.702215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-tls-certs\") pod \"simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7\" (UID: \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\") " pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:17:52.702378 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.702323    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-model-cache\") pod \"simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7\" (UID: \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\") " pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:17:52.802775 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.802737    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-model-cache\") pod \"simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7\" (UID: \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\") " pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:17:52.802985 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.802786    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-tmp-dir\") pod \"simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7\" (UID: \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\") " pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:17:52.802985 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.802851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-dshm\") pod \"simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7\" (UID: \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\") " pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:17:52.802985 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.802900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-home\") pod \"simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7\" (UID: \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\") " pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:17:52.802985 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.802917    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qclcl\" (UniqueName: \"kubernetes.io/projected/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-kube-api-access-qclcl\") pod \"simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7\" (UID: \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\") " pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:17:52.802985 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.802937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-tls-certs\") pod \"simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7\" (UID: \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\") " pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:17:52.803278 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.803249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-model-cache\") pod \"simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7\" (UID: \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\") " pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:17:52.803412 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.803298    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-tmp-dir\") pod \"simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7\" (UID: \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\") " pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:17:52.803412 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.803349    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-home\") pod \"simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7\" (UID: \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\") " pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:17:52.805284 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.805264    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-dshm\") pod \"simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7\" (UID: \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\") " pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:17:52.805627 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.805608    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-tls-certs\") pod \"simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7\" (UID: \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\") " pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:17:52.813431 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.813406    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qclcl\" (UniqueName: \"kubernetes.io/projected/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-kube-api-access-qclcl\") pod \"simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7\" (UID: \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\") " pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:17:52.922756 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:52.922717    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:17:53.199049 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:53.198965    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"]
Sep 11 21:17:53.201737 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:17:53.201708    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e6496ab_6ff6_476a_9a3b_f3b316c7099e.slice/crio-c264a2dfb73ec31d2b3f41687fdf44ef8df5c281fdf47861d42c83283ccfb1c3 WatchSource:0}: Error finding container c264a2dfb73ec31d2b3f41687fdf44ef8df5c281fdf47861d42c83283ccfb1c3: Status 404 returned error can't find the container with id c264a2dfb73ec31d2b3f41687fdf44ef8df5c281fdf47861d42c83283ccfb1c3
Sep 11 21:17:53.453659 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:53.453571    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs"]
Sep 11 21:17:53.453997 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:53.453981    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" containerName="istio-proxy"
Sep 11 21:17:53.453997 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:53.453997    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" containerName="istio-proxy"
Sep 11 21:17:53.454155 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:53.454056    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3ba4b30-5dcb-4bbd-833e-c8bfb3b58eed" containerName="istio-proxy"
Sep 11 21:17:53.457081 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:53.457055    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs"
Sep 11 21:17:53.459326 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:53.459302    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-2d3d3219-dockercfg-dmbcj\""
Sep 11 21:17:53.459889 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:53.459867    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-2d3d3219\""
Sep 11 21:17:53.473223 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:53.473193    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs"]
Sep 11 21:17:53.509096 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:53.509063    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba43dd5-45a9-4b06-98a3-22105a01de56-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs\" (UID: \"cba43dd5-45a9-4b06-98a3-22105a01de56\") " pod="openshift-ingress/payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs"
Sep 11 21:17:53.509283 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:53.509127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgsr2\" (UniqueName: \"kubernetes.io/projected/cba43dd5-45a9-4b06-98a3-22105a01de56-kube-api-access-tgsr2\") pod \"payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs\" (UID: \"cba43dd5-45a9-4b06-98a3-22105a01de56\") " pod="openshift-ingress/payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs"
Sep 11 21:17:53.609518 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:53.609478    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgsr2\" (UniqueName: \"kubernetes.io/projected/cba43dd5-45a9-4b06-98a3-22105a01de56-kube-api-access-tgsr2\") pod \"payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs\" (UID: \"cba43dd5-45a9-4b06-98a3-22105a01de56\") " pod="openshift-ingress/payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs"
Sep 11 21:17:53.609696 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:53.609546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba43dd5-45a9-4b06-98a3-22105a01de56-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs\" (UID: \"cba43dd5-45a9-4b06-98a3-22105a01de56\") " pod="openshift-ingress/payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs"
Sep 11 21:17:53.610307 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:53.610288    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba43dd5-45a9-4b06-98a3-22105a01de56-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs\" (UID: \"cba43dd5-45a9-4b06-98a3-22105a01de56\") " pod="openshift-ingress/payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs"
Sep 11 21:17:53.618088 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:53.618065    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgsr2\" (UniqueName: \"kubernetes.io/projected/cba43dd5-45a9-4b06-98a3-22105a01de56-kube-api-access-tgsr2\") pod \"payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs\" (UID: \"cba43dd5-45a9-4b06-98a3-22105a01de56\") " pod="openshift-ingress/payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs"
Sep 11 21:17:53.766938 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:53.766804    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs"
Sep 11 21:17:53.870211 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:53.870176    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7" event={"ID":"7e6496ab-6ff6-476a-9a3b-f3b316c7099e","Type":"ContainerStarted","Data":"84c99a94b449e7432646b2b959110905cbcfe4f58d68bd0b8cb8b16b8f19b034"}
Sep 11 21:17:53.870360 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:53.870222    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7" event={"ID":"7e6496ab-6ff6-476a-9a3b-f3b316c7099e","Type":"ContainerStarted","Data":"c264a2dfb73ec31d2b3f41687fdf44ef8df5c281fdf47861d42c83283ccfb1c3"}
Sep 11 21:17:53.893294 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:53.893236    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7" podStartSLOduration=1.89321961 podStartE2EDuration="1.89321961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:17:53.891439119 +0000 UTC m=+740.595408412" watchObservedRunningTime="2026-09-11 21:17:53.89321961 +0000 UTC m=+740.597188923"
Sep 11 21:17:54.047664 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:54.047574    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs"]
Sep 11 21:17:54.050611 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:17:54.050583    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba43dd5_45a9_4b06_98a3_22105a01de56.slice/crio-adbbfb41a512f28c708502883458dd7dfe9b4dc7e62652763282c77f09609059 WatchSource:0}: Error finding container adbbfb41a512f28c708502883458dd7dfe9b4dc7e62652763282c77f09609059: Status 404 returned error can't find the container with id adbbfb41a512f28c708502883458dd7dfe9b4dc7e62652763282c77f09609059
Sep 11 21:17:54.220631 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:54.220598    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:17:54.881082 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:54.881044    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs" event={"ID":"cba43dd5-45a9-4b06-98a3-22105a01de56","Type":"ContainerStarted","Data":"84a198f44d9e60ca8a2a354abe12d2b9f54b59556696f0833e9e80ddc5541441"}
Sep 11 21:17:54.881462 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:54.881106    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs" event={"ID":"cba43dd5-45a9-4b06-98a3-22105a01de56","Type":"ContainerStarted","Data":"adbbfb41a512f28c708502883458dd7dfe9b4dc7e62652763282c77f09609059"}
Sep 11 21:17:54.901338 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:54.901280    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs" podStartSLOduration=1.9012624470000001 podStartE2EDuration="1.901262447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:17:54.900477505 +0000 UTC m=+741.604446796" watchObservedRunningTime="2026-09-11 21:17:54.901262447 +0000 UTC m=+741.605231736"
Sep 11 21:17:55.885919 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:17:55.885882    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs"
Sep 11 21:18:02.923075 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:02.923036    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:18:02.923513 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:02.923175    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:18:02.936040 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:02.936012    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:18:03.936271 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:03.936241    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:18:06.895565 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:06.895536    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs"
Sep 11 21:18:10.920867 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:10.920810    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:18:13.993548 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:13.993513    2578 generic.go:356] "Generic (PLEG): container finished" podID="644cf0ad-7860-4448-937c-d6e195947628" containerID="a026221b0b7828ca9fcc4b4685da2777176b7a0702770a86f0ace20c191f42fe" exitCode=0
Sep 11 21:18:13.994003 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:13.993583    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z" event={"ID":"644cf0ad-7860-4448-937c-d6e195947628","Type":"ContainerDied","Data":"a026221b0b7828ca9fcc4b4685da2777176b7a0702770a86f0ace20c191f42fe"}
Sep 11 21:18:13.994003 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:13.993997    2578 scope.go:117] "RemoveContainer" containerID="a026221b0b7828ca9fcc4b4685da2777176b7a0702770a86f0ace20c191f42fe"
Sep 11 21:18:13.995187 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:13.995159    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:18:13.995261 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:13.995231    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:18:13.995297 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:13.995264    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:18:15.001161 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:15.001130    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z" event={"ID":"644cf0ad-7860-4448-937c-d6e195947628","Type":"ContainerStarted","Data":"0d45b7192d053da4093aa0adf607a36ffaa79c52b31a9286ae1ef17f1392a674"}
Sep 11 21:18:15.296807 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:15.296722    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d3d3219-tk85l"]
Sep 11 21:18:15.300166 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:15.300146    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d3d3219-tk85l"
Sep 11 21:18:15.302308 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:15.302290    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:18:15.302450 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:15.302400    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfm9g\""
Sep 11 21:18:15.310019 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:15.309981    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d3d3219-tk85l"]
Sep 11 21:18:15.399899 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:15.399862    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f2e15df-7190-4d10-ad03-6cfb8148770c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2d3d3219-tk85l\" (UID: \"9f2e15df-7190-4d10-ad03-6cfb8148770c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d3d3219-tk85l"
Sep 11 21:18:15.501219 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:15.501173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f2e15df-7190-4d10-ad03-6cfb8148770c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2d3d3219-tk85l\" (UID: \"9f2e15df-7190-4d10-ad03-6cfb8148770c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d3d3219-tk85l"
Sep 11 21:18:15.501863 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:15.501811    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f2e15df-7190-4d10-ad03-6cfb8148770c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2d3d3219-tk85l\" (UID: \"9f2e15df-7190-4d10-ad03-6cfb8148770c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d3d3219-tk85l"
Sep 11 21:18:15.618253 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:15.618142    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d3d3219-tk85l"
Sep 11 21:18:15.919273 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:15.919192    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d3d3219-tk85l"]
Sep 11 21:18:15.920422 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:15.920396    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:18:15.920669 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:15.920646    2578 patch_prober.go:28] interesting pod/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 11 21:18:15.920722 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:15.920702    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z" podUID="644cf0ad-7860-4448-937c-d6e195947628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 11 21:18:15.922426 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:18:15.922402    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2e15df_7190_4d10_ad03_6cfb8148770c.slice/crio-0db31bb9952e0e8f937b763b4b751e92250aafa221f7b8c824eceab26d0be1bf WatchSource:0}: Error finding container 0db31bb9952e0e8f937b763b4b751e92250aafa221f7b8c824eceab26d0be1bf: Status 404 returned error can't find the container with id 0db31bb9952e0e8f937b763b4b751e92250aafa221f7b8c824eceab26d0be1bf
Sep 11 21:18:16.007887 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:16.007852    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d3d3219-tk85l" event={"ID":"9f2e15df-7190-4d10-ad03-6cfb8148770c","Type":"ContainerStarted","Data":"0db31bb9952e0e8f937b763b4b751e92250aafa221f7b8c824eceab26d0be1bf"}
Sep 11 21:18:16.920244 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:16.920200    2578 patch_prober.go:28] interesting pod/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 11 21:18:16.920516 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:16.920261    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z" podUID="644cf0ad-7860-4448-937c-d6e195947628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 11 21:18:17.015155 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:17.015118    2578 generic.go:356] "Generic (PLEG): container finished" podID="9f2e15df-7190-4d10-ad03-6cfb8148770c" containerID="57a149534803a00b53dd2d0bc300762d6da9c915ead78f32ac45b5600c816fca" exitCode=0
Sep 11 21:18:17.015552 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:17.015201    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d3d3219-tk85l" event={"ID":"9f2e15df-7190-4d10-ad03-6cfb8148770c","Type":"ContainerDied","Data":"57a149534803a00b53dd2d0bc300762d6da9c915ead78f32ac45b5600c816fca"}
Sep 11 21:18:17.920489 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:17.920444    2578 patch_prober.go:28] interesting pod/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 11 21:18:17.920683 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:17.920511    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z" podUID="644cf0ad-7860-4448-937c-d6e195947628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 11 21:18:18.198882 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:18.198854    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d3d3219-tk85l"
Sep 11 21:18:18.343148 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:18.343107    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f2e15df-7190-4d10-ad03-6cfb8148770c-maas-api-service-ca\") pod \"9f2e15df-7190-4d10-ad03-6cfb8148770c\" (UID: \"9f2e15df-7190-4d10-ad03-6cfb8148770c\") "
Sep 11 21:18:18.344096 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:18.343724    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f2e15df-7190-4d10-ad03-6cfb8148770c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9f2e15df-7190-4d10-ad03-6cfb8148770c" (UID: "9f2e15df-7190-4d10-ad03-6cfb8148770c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:18:18.445163 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:18.445053    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f2e15df-7190-4d10-ad03-6cfb8148770c-maas-api-service-ca\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:18:18.920305 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:18.920210    2578 patch_prober.go:28] interesting pod/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 11 21:18:18.920305 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:18.920276    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z" podUID="644cf0ad-7860-4448-937c-d6e195947628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 11 21:18:19.026562 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:19.026516    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d3d3219-tk85l" event={"ID":"9f2e15df-7190-4d10-ad03-6cfb8148770c","Type":"ContainerDied","Data":"0db31bb9952e0e8f937b763b4b751e92250aafa221f7b8c824eceab26d0be1bf"}
Sep 11 21:18:19.026775 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:19.026592    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0db31bb9952e0e8f937b763b4b751e92250aafa221f7b8c824eceab26d0be1bf"
Sep 11 21:18:19.026775 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:19.026648    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d3d3219-tk85l"
Sep 11 21:18:19.920384 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:19.920349    2578 patch_prober.go:28] interesting pod/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 11 21:18:19.920767 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:19.920408    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z" podUID="644cf0ad-7860-4448-937c-d6e195947628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 11 21:18:20.851272 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:20.851236    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:18:20.924813 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:20.924785    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:18:21.038354 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:21.038317    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:18:21.293622 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:21.293585    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs"]
Sep 11 21:18:21.293860 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:21.293832    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs" podUID="cba43dd5-45a9-4b06-98a3-22105a01de56" containerName="payload-processing" containerID="cri-o://84a198f44d9e60ca8a2a354abe12d2b9f54b59556696f0833e9e80ddc5541441" gracePeriod=30
Sep 11 21:18:21.482207 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:21.482170    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d3d3219-tk85l"]
Sep 11 21:18:21.484661 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:21.484633    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2d3d3219-tk85l"]
Sep 11 21:18:21.556110 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:21.556043    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs"
Sep 11 21:18:21.621301 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:21.621265    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"]
Sep 11 21:18:21.675538 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:21.675500    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba43dd5-45a9-4b06-98a3-22105a01de56-plugins-config-volume\") pod \"cba43dd5-45a9-4b06-98a3-22105a01de56\" (UID: \"cba43dd5-45a9-4b06-98a3-22105a01de56\") "
Sep 11 21:18:21.675538 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:21.675543    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgsr2\" (UniqueName: \"kubernetes.io/projected/cba43dd5-45a9-4b06-98a3-22105a01de56-kube-api-access-tgsr2\") pod \"cba43dd5-45a9-4b06-98a3-22105a01de56\" (UID: \"cba43dd5-45a9-4b06-98a3-22105a01de56\") "
Sep 11 21:18:21.676005 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:21.675981    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cba43dd5-45a9-4b06-98a3-22105a01de56-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cba43dd5-45a9-4b06-98a3-22105a01de56" (UID: "cba43dd5-45a9-4b06-98a3-22105a01de56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:18:21.677733 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:21.677703    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba43dd5-45a9-4b06-98a3-22105a01de56-kube-api-access-tgsr2" (OuterVolumeSpecName: "kube-api-access-tgsr2") pod "cba43dd5-45a9-4b06-98a3-22105a01de56" (UID: "cba43dd5-45a9-4b06-98a3-22105a01de56"). InnerVolumeSpecName "kube-api-access-tgsr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:18:21.776983 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:21.776942    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cba43dd5-45a9-4b06-98a3-22105a01de56-plugins-config-volume\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:18:21.776983 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:21.776978    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgsr2\" (UniqueName: \"kubernetes.io/projected/cba43dd5-45a9-4b06-98a3-22105a01de56-kube-api-access-tgsr2\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:18:21.927105 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:21.927076    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f2e15df-7190-4d10-ad03-6cfb8148770c" path="/var/lib/kubelet/pods/9f2e15df-7190-4d10-ad03-6cfb8148770c/volumes"
Sep 11 21:18:22.043072 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.043040    2578 generic.go:356] "Generic (PLEG): container finished" podID="cba43dd5-45a9-4b06-98a3-22105a01de56" containerID="84a198f44d9e60ca8a2a354abe12d2b9f54b59556696f0833e9e80ddc5541441" exitCode=0
Sep 11 21:18:22.043266 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.043082    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs"
Sep 11 21:18:22.043266 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.043132    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs" event={"ID":"cba43dd5-45a9-4b06-98a3-22105a01de56","Type":"ContainerDied","Data":"84a198f44d9e60ca8a2a354abe12d2b9f54b59556696f0833e9e80ddc5541441"}
Sep 11 21:18:22.043266 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.043200    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs" event={"ID":"cba43dd5-45a9-4b06-98a3-22105a01de56","Type":"ContainerDied","Data":"adbbfb41a512f28c708502883458dd7dfe9b4dc7e62652763282c77f09609059"}
Sep 11 21:18:22.043266 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.043230    2578 scope.go:117] "RemoveContainer" containerID="84a198f44d9e60ca8a2a354abe12d2b9f54b59556696f0833e9e80ddc5541441"
Sep 11 21:18:22.043486 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.043441    2578 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z" secret="" err="secret \"e2e-migrate-2d3d3219-gw-openshift-default-dockercfg-ggmql\" not found"
Sep 11 21:18:22.052138 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.052117    2578 scope.go:117] "RemoveContainer" containerID="84a198f44d9e60ca8a2a354abe12d2b9f54b59556696f0833e9e80ddc5541441"
Sep 11 21:18:22.052465 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:18:22.052438    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84a198f44d9e60ca8a2a354abe12d2b9f54b59556696f0833e9e80ddc5541441\": container with ID starting with 84a198f44d9e60ca8a2a354abe12d2b9f54b59556696f0833e9e80ddc5541441 not found: ID does not exist" containerID="84a198f44d9e60ca8a2a354abe12d2b9f54b59556696f0833e9e80ddc5541441"
Sep 11 21:18:22.052525 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.052475    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84a198f44d9e60ca8a2a354abe12d2b9f54b59556696f0833e9e80ddc5541441"} err="failed to get container status \"84a198f44d9e60ca8a2a354abe12d2b9f54b59556696f0833e9e80ddc5541441\": rpc error: code = NotFound desc = could not find container \"84a198f44d9e60ca8a2a354abe12d2b9f54b59556696f0833e9e80ddc5541441\": container with ID starting with 84a198f44d9e60ca8a2a354abe12d2b9f54b59556696f0833e9e80ddc5541441 not found: ID does not exist"
Sep 11 21:18:22.058713 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.058687    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs"]
Sep 11 21:18:22.060859 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.060832    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2d3d3219-5557684d66-q92bs"]
Sep 11 21:18:22.370149 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.370066    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"]
Sep 11 21:18:22.370407 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.370395    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f2e15df-7190-4d10-ad03-6cfb8148770c" containerName="revoke-keys"
Sep 11 21:18:22.370461 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.370409    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f2e15df-7190-4d10-ad03-6cfb8148770c" containerName="revoke-keys"
Sep 11 21:18:22.370461 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.370418    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cba43dd5-45a9-4b06-98a3-22105a01de56" containerName="payload-processing"
Sep 11 21:18:22.370461 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.370424    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cba43dd5-45a9-4b06-98a3-22105a01de56" containerName="payload-processing"
Sep 11 21:18:22.370568 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.370491    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f2e15df-7190-4d10-ad03-6cfb8148770c" containerName="revoke-keys"
Sep 11 21:18:22.370568 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.370502    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cba43dd5-45a9-4b06-98a3-22105a01de56" containerName="payload-processing"
Sep 11 21:18:22.374800 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.374767    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.382335 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.382315    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-257d9182-openshift-default-dockercfg-b78w8\""
Sep 11 21:18:22.394123 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.394088    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"]
Sep 11 21:18:22.483959 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.483910    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2kl8\" (UniqueName: \"kubernetes.io/projected/b4cfe8fc-7176-42d0-a27e-a3d05efca597-kube-api-access-m2kl8\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.484163 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.483980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-workload-certs\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.484163 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.484001    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-credential-socket\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.484163 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.484025    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istio-podinfo\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.484163 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.484046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-workload-socket\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.484383 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.484152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istio-envoy\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.484383 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.484199    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istiod-ca-cert\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.484383 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.484242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istio-token\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.484383 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.484274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istio-data\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.585199 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.585158    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istio-podinfo\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.585199 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.585199    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-workload-socket\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.585448 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.585225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istio-envoy\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.585448 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.585255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istiod-ca-cert\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.585448 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.585288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istio-token\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.585448 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.585326    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istio-data\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.585448 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.585374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2kl8\" (UniqueName: \"kubernetes.io/projected/b4cfe8fc-7176-42d0-a27e-a3d05efca597-kube-api-access-m2kl8\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.585448 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.585418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-workload-certs\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.585448 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.585447    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-credential-socket\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.585763 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.585643    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-workload-socket\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.585849 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.585797    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istio-data\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.585922 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.585905    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-credential-socket\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.586016 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.585992    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-workload-certs\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.586179 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.586155    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istiod-ca-cert\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.587744 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.587715    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istio-envoy\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.587892 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.587791    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istio-podinfo\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.596801 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.596768    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istio-token\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.597039 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.597016    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2kl8\" (UniqueName: \"kubernetes.io/projected/b4cfe8fc-7176-42d0-a27e-a3d05efca597-kube-api-access-m2kl8\") pod \"e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") " pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.685199 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.685163    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:22.979345 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.979242    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"]
Sep 11 21:18:22.983757 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:18:22.983731    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4cfe8fc_7176_42d0_a27e_a3d05efca597.slice/crio-2862b70c926a859e85e065a91a2a2bd30e441517b98dd0f7be7fa78604eecd04 WatchSource:0}: Error finding container 2862b70c926a859e85e065a91a2a2bd30e441517b98dd0f7be7fa78604eecd04: Status 404 returned error can't find the container with id 2862b70c926a859e85e065a91a2a2bd30e441517b98dd0f7be7fa78604eecd04
Sep 11 21:18:22.986002 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.985969    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:18:22.986068 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.986043    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:18:22.986107 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:22.986072    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:18:23.050662 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:23.050622    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh" event={"ID":"b4cfe8fc-7176-42d0-a27e-a3d05efca597","Type":"ContainerStarted","Data":"76b290fb3d56ead04ad6cb38535ecc04150d2ec89ec28ab4e36bd9ebc65634ea"}
Sep 11 21:18:23.050841 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:23.050691    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh" event={"ID":"b4cfe8fc-7176-42d0-a27e-a3d05efca597","Type":"ContainerStarted","Data":"2862b70c926a859e85e065a91a2a2bd30e441517b98dd0f7be7fa78604eecd04"}
Sep 11 21:18:23.051955 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:23.051920    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z" podUID="644cf0ad-7860-4448-937c-d6e195947628" containerName="istio-proxy" containerID="cri-o://0d45b7192d053da4093aa0adf607a36ffaa79c52b31a9286ae1ef17f1392a674" gracePeriod=30
Sep 11 21:18:23.084835 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:23.084763    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh" podStartSLOduration=1.084742501 podStartE2EDuration="1.084742501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:18:23.081612659 +0000 UTC m=+769.785581951" watchObservedRunningTime="2026-09-11 21:18:23.084742501 +0000 UTC m=+769.788711792"
Sep 11 21:18:23.685723 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:23.685689    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:23.928561 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:23.928526    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cba43dd5-45a9-4b06-98a3-22105a01de56" path="/var/lib/kubelet/pods/cba43dd5-45a9-4b06-98a3-22105a01de56/volumes"
Sep 11 21:18:24.691506 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:24.691471    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:25.062024 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:25.061944    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:25.063136 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:25.063112    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:27.864206 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:27.864166    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj"]
Sep 11 21:18:27.868672 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:27.868651    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj"
Sep 11 21:18:27.871268 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:27.871241    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-257d9182-dockercfg-zxdcz\""
Sep 11 21:18:27.871389 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:27.871264    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-257d9182\""
Sep 11 21:18:27.878361 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:27.878332    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj"]
Sep 11 21:18:28.035244 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.035201    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08366ec2-1677-4671-84a4-28bb9807b5e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj\" (UID: \"08366ec2-1677-4671-84a4-28bb9807b5e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj"
Sep 11 21:18:28.035460 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.035397    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlk4x\" (UniqueName: \"kubernetes.io/projected/08366ec2-1677-4671-84a4-28bb9807b5e4-kube-api-access-mlk4x\") pod \"payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj\" (UID: \"08366ec2-1677-4671-84a4-28bb9807b5e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj"
Sep 11 21:18:28.136735 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.136636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlk4x\" (UniqueName: \"kubernetes.io/projected/08366ec2-1677-4671-84a4-28bb9807b5e4-kube-api-access-mlk4x\") pod \"payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj\" (UID: \"08366ec2-1677-4671-84a4-28bb9807b5e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj"
Sep 11 21:18:28.136735 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.136695    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08366ec2-1677-4671-84a4-28bb9807b5e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj\" (UID: \"08366ec2-1677-4671-84a4-28bb9807b5e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj"
Sep 11 21:18:28.137437 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.137412    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08366ec2-1677-4671-84a4-28bb9807b5e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj\" (UID: \"08366ec2-1677-4671-84a4-28bb9807b5e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj"
Sep 11 21:18:28.144793 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.144766    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlk4x\" (UniqueName: \"kubernetes.io/projected/08366ec2-1677-4671-84a4-28bb9807b5e4-kube-api-access-mlk4x\") pod \"payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj\" (UID: \"08366ec2-1677-4671-84a4-28bb9807b5e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj"
Sep 11 21:18:28.181458 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.181425    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj"
Sep 11 21:18:28.321420 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.321387    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:18:28.439689 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.439650    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/644cf0ad-7860-4448-937c-d6e195947628-istio-token\") pod \"644cf0ad-7860-4448-937c-d6e195947628\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") "
Sep 11 21:18:28.439930 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.439706    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-istio-envoy\") pod \"644cf0ad-7860-4448-937c-d6e195947628\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") "
Sep 11 21:18:28.439930 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.439729    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhdc2\" (UniqueName: \"kubernetes.io/projected/644cf0ad-7860-4448-937c-d6e195947628-kube-api-access-fhdc2\") pod \"644cf0ad-7860-4448-937c-d6e195947628\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") "
Sep 11 21:18:28.439930 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.439752    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-workload-certs\") pod \"644cf0ad-7860-4448-937c-d6e195947628\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") "
Sep 11 21:18:28.439930 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.439796    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-istio-data\") pod \"644cf0ad-7860-4448-937c-d6e195947628\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") "
Sep 11 21:18:28.439930 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.439850    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-credential-socket\") pod \"644cf0ad-7860-4448-937c-d6e195947628\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") "
Sep 11 21:18:28.439930 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.439909    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-workload-socket\") pod \"644cf0ad-7860-4448-937c-d6e195947628\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") "
Sep 11 21:18:28.440237 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.439947    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/644cf0ad-7860-4448-937c-d6e195947628-istiod-ca-cert\") pod \"644cf0ad-7860-4448-937c-d6e195947628\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") "
Sep 11 21:18:28.440237 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.439978    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/644cf0ad-7860-4448-937c-d6e195947628-istio-podinfo\") pod \"644cf0ad-7860-4448-937c-d6e195947628\" (UID: \"644cf0ad-7860-4448-937c-d6e195947628\") "
Sep 11 21:18:28.440339 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.440228    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "644cf0ad-7860-4448-937c-d6e195947628" (UID: "644cf0ad-7860-4448-937c-d6e195947628"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:18:28.440339 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.440247    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "644cf0ad-7860-4448-937c-d6e195947628" (UID: "644cf0ad-7860-4448-937c-d6e195947628"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:18:28.440339 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.440230    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-istio-data" (OuterVolumeSpecName: "istio-data") pod "644cf0ad-7860-4448-937c-d6e195947628" (UID: "644cf0ad-7860-4448-937c-d6e195947628"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:18:28.440510 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.440477    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "644cf0ad-7860-4448-937c-d6e195947628" (UID: "644cf0ad-7860-4448-937c-d6e195947628"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:18:28.440588 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.440564    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/644cf0ad-7860-4448-937c-d6e195947628-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "644cf0ad-7860-4448-937c-d6e195947628" (UID: "644cf0ad-7860-4448-937c-d6e195947628"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:18:28.442130 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.442101    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/644cf0ad-7860-4448-937c-d6e195947628-istio-token" (OuterVolumeSpecName: "istio-token") pod "644cf0ad-7860-4448-937c-d6e195947628" (UID: "644cf0ad-7860-4448-937c-d6e195947628"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:18:28.442273 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.442152    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "644cf0ad-7860-4448-937c-d6e195947628" (UID: "644cf0ad-7860-4448-937c-d6e195947628"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:18:28.442411 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.442396    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/644cf0ad-7860-4448-937c-d6e195947628-kube-api-access-fhdc2" (OuterVolumeSpecName: "kube-api-access-fhdc2") pod "644cf0ad-7860-4448-937c-d6e195947628" (UID: "644cf0ad-7860-4448-937c-d6e195947628"). InnerVolumeSpecName "kube-api-access-fhdc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:18:28.442486 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.442468    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/644cf0ad-7860-4448-937c-d6e195947628-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "644cf0ad-7860-4448-937c-d6e195947628" (UID: "644cf0ad-7860-4448-937c-d6e195947628"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:18:28.467336 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.467299    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj"]
Sep 11 21:18:28.470369 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:18:28.470343    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08366ec2_1677_4671_84a4_28bb9807b5e4.slice/crio-200c2cc9329a0afc73360b85af64b18227f970c2ba127a5b027153efce5cec7e WatchSource:0}: Error finding container 200c2cc9329a0afc73360b85af64b18227f970c2ba127a5b027153efce5cec7e: Status 404 returned error can't find the container with id 200c2cc9329a0afc73360b85af64b18227f970c2ba127a5b027153efce5cec7e
Sep 11 21:18:28.540891 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.540859    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/644cf0ad-7860-4448-937c-d6e195947628-istio-token\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:18:28.540891 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.540888    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-istio-envoy\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:18:28.540891 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.540900    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhdc2\" (UniqueName: \"kubernetes.io/projected/644cf0ad-7860-4448-937c-d6e195947628-kube-api-access-fhdc2\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:18:28.541134 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.540913    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-workload-certs\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:18:28.541134 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.540926    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-istio-data\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:18:28.541134 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.540935    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-credential-socket\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:18:28.541134 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.540944    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/644cf0ad-7860-4448-937c-d6e195947628-workload-socket\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:18:28.541134 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.540952    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/644cf0ad-7860-4448-937c-d6e195947628-istiod-ca-cert\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:18:28.541134 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:28.540964    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/644cf0ad-7860-4448-937c-d6e195947628-istio-podinfo\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:18:29.081949 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:29.081900    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj" event={"ID":"08366ec2-1677-4671-84a4-28bb9807b5e4","Type":"ContainerStarted","Data":"8b9b12b17b5577131f03b44414f971bc2e5caf973f5f70e1f0daa59479e44c9a"}
Sep 11 21:18:29.082408 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:29.081967    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj" event={"ID":"08366ec2-1677-4671-84a4-28bb9807b5e4","Type":"ContainerStarted","Data":"200c2cc9329a0afc73360b85af64b18227f970c2ba127a5b027153efce5cec7e"}
Sep 11 21:18:29.082408 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:29.081998    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj"
Sep 11 21:18:29.083651 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:29.083625    2578 generic.go:356] "Generic (PLEG): container finished" podID="644cf0ad-7860-4448-937c-d6e195947628" containerID="0d45b7192d053da4093aa0adf607a36ffaa79c52b31a9286ae1ef17f1392a674" exitCode=0
Sep 11 21:18:29.083775 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:29.083675    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"
Sep 11 21:18:29.083775 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:29.083687    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z" event={"ID":"644cf0ad-7860-4448-937c-d6e195947628","Type":"ContainerDied","Data":"0d45b7192d053da4093aa0adf607a36ffaa79c52b31a9286ae1ef17f1392a674"}
Sep 11 21:18:29.083775 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:29.083725    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z" event={"ID":"644cf0ad-7860-4448-937c-d6e195947628","Type":"ContainerDied","Data":"c1945820570445d6c3f03ba6cd4e843c6593d8554095680e50f19fa4cc335473"}
Sep 11 21:18:29.083775 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:29.083745    2578 scope.go:117] "RemoveContainer" containerID="0d45b7192d053da4093aa0adf607a36ffaa79c52b31a9286ae1ef17f1392a674"
Sep 11 21:18:29.094559 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:29.094536    2578 scope.go:117] "RemoveContainer" containerID="a026221b0b7828ca9fcc4b4685da2777176b7a0702770a86f0ace20c191f42fe"
Sep 11 21:18:29.102606 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:29.102556    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj" podStartSLOduration=2.102537888 podStartE2EDuration="2.102537888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:18:29.099985711 +0000 UTC m=+775.803955001" watchObservedRunningTime="2026-09-11 21:18:29.102537888 +0000 UTC m=+775.806507177"
Sep 11 21:18:29.103487 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:29.103462    2578 scope.go:117] "RemoveContainer" containerID="0d45b7192d053da4093aa0adf607a36ffaa79c52b31a9286ae1ef17f1392a674"
Sep 11 21:18:29.103813 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:18:29.103793    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d45b7192d053da4093aa0adf607a36ffaa79c52b31a9286ae1ef17f1392a674\": container with ID starting with 0d45b7192d053da4093aa0adf607a36ffaa79c52b31a9286ae1ef17f1392a674 not found: ID does not exist" containerID="0d45b7192d053da4093aa0adf607a36ffaa79c52b31a9286ae1ef17f1392a674"
Sep 11 21:18:29.103931 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:29.103834    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d45b7192d053da4093aa0adf607a36ffaa79c52b31a9286ae1ef17f1392a674"} err="failed to get container status \"0d45b7192d053da4093aa0adf607a36ffaa79c52b31a9286ae1ef17f1392a674\": rpc error: code = NotFound desc = could not find container \"0d45b7192d053da4093aa0adf607a36ffaa79c52b31a9286ae1ef17f1392a674\": container with ID starting with 0d45b7192d053da4093aa0adf607a36ffaa79c52b31a9286ae1ef17f1392a674 not found: ID does not exist"
Sep 11 21:18:29.103931 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:29.103851    2578 scope.go:117] "RemoveContainer" containerID="a026221b0b7828ca9fcc4b4685da2777176b7a0702770a86f0ace20c191f42fe"
Sep 11 21:18:29.104153 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:18:29.104134    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a026221b0b7828ca9fcc4b4685da2777176b7a0702770a86f0ace20c191f42fe\": container with ID starting with a026221b0b7828ca9fcc4b4685da2777176b7a0702770a86f0ace20c191f42fe not found: ID does not exist" containerID="a026221b0b7828ca9fcc4b4685da2777176b7a0702770a86f0ace20c191f42fe"
Sep 11 21:18:29.104201 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:29.104158    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a026221b0b7828ca9fcc4b4685da2777176b7a0702770a86f0ace20c191f42fe"} err="failed to get container status \"a026221b0b7828ca9fcc4b4685da2777176b7a0702770a86f0ace20c191f42fe\": rpc error: code = NotFound desc = could not find container \"a026221b0b7828ca9fcc4b4685da2777176b7a0702770a86f0ace20c191f42fe\": container with ID starting with a026221b0b7828ca9fcc4b4685da2777176b7a0702770a86f0ace20c191f42fe not found: ID does not exist"
Sep 11 21:18:29.119490 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:29.119453    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"]
Sep 11 21:18:29.126465 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:29.126431    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-2d3d3219-gw-openshift-default-5bff794cd4-5lp6z"]
Sep 11 21:18:29.931940 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:29.931907    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="644cf0ad-7860-4448-937c-d6e195947628" path="/var/lib/kubelet/pods/644cf0ad-7860-4448-937c-d6e195947628/volumes"
Sep 11 21:18:31.612933 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:31.612895    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:18:40.091342 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:40.091301    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj"
Sep 11 21:18:41.098515 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.098471    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"]
Sep 11 21:18:41.099050 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.098829    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="644cf0ad-7860-4448-937c-d6e195947628" containerName="istio-proxy"
Sep 11 21:18:41.099050 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.098841    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="644cf0ad-7860-4448-937c-d6e195947628" containerName="istio-proxy"
Sep 11 21:18:41.099050 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.098911    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="644cf0ad-7860-4448-937c-d6e195947628" containerName="istio-proxy"
Sep 11 21:18:41.099050 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.098919    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="644cf0ad-7860-4448-937c-d6e195947628" containerName="istio-proxy"
Sep 11 21:18:41.101850 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.101808    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:18:41.103976 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.103943    2578 patch_prober.go:28] interesting pod/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 21:18:41.104089 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.103995    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh" podUID="b4cfe8fc-7176-42d0-a27e-a3d05efca597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 21:18:41.104156 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.104134    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-962a7d\"/\"simulator-w4-962a7d-kserve-self-signed-certs\""
Sep 11 21:18:41.104204 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.104152    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-962a7d\"/\"kube-root-ca.crt\""
Sep 11 21:18:41.104771 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.104753    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-962a7d\"/\"openshift-service-ca.crt\""
Sep 11 21:18:41.104860 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.104777    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-962a7d\"/\"default-dockercfg-wqx78\""
Sep 11 21:18:41.110489 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.110463    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"]
Sep 11 21:18:41.153287 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.153245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfn9c\" (UniqueName: \"kubernetes.io/projected/7d2aa425-fe38-4068-8db3-4813b822297e-kube-api-access-nfn9c\") pod \"simulator-w4-962a7d-kserve-5cc984d758-mn6km\" (UID: \"7d2aa425-fe38-4068-8db3-4813b822297e\") " pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:18:41.153481 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.153298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d2aa425-fe38-4068-8db3-4813b822297e-tls-certs\") pod \"simulator-w4-962a7d-kserve-5cc984d758-mn6km\" (UID: \"7d2aa425-fe38-4068-8db3-4813b822297e\") " pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:18:41.153481 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.153376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d2aa425-fe38-4068-8db3-4813b822297e-home\") pod \"simulator-w4-962a7d-kserve-5cc984d758-mn6km\" (UID: \"7d2aa425-fe38-4068-8db3-4813b822297e\") " pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:18:41.153481 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.153408    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d2aa425-fe38-4068-8db3-4813b822297e-dshm\") pod \"simulator-w4-962a7d-kserve-5cc984d758-mn6km\" (UID: \"7d2aa425-fe38-4068-8db3-4813b822297e\") " pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:18:41.153481 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.153449    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d2aa425-fe38-4068-8db3-4813b822297e-tmp-dir\") pod \"simulator-w4-962a7d-kserve-5cc984d758-mn6km\" (UID: \"7d2aa425-fe38-4068-8db3-4813b822297e\") " pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:18:41.153687 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.153510    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d2aa425-fe38-4068-8db3-4813b822297e-model-cache\") pod \"simulator-w4-962a7d-kserve-5cc984d758-mn6km\" (UID: \"7d2aa425-fe38-4068-8db3-4813b822297e\") " pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:18:41.254136 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.254094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d2aa425-fe38-4068-8db3-4813b822297e-tmp-dir\") pod \"simulator-w4-962a7d-kserve-5cc984d758-mn6km\" (UID: \"7d2aa425-fe38-4068-8db3-4813b822297e\") " pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:18:41.254136 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.254137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d2aa425-fe38-4068-8db3-4813b822297e-model-cache\") pod \"simulator-w4-962a7d-kserve-5cc984d758-mn6km\" (UID: \"7d2aa425-fe38-4068-8db3-4813b822297e\") " pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:18:41.254387 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.254285    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfn9c\" (UniqueName: \"kubernetes.io/projected/7d2aa425-fe38-4068-8db3-4813b822297e-kube-api-access-nfn9c\") pod \"simulator-w4-962a7d-kserve-5cc984d758-mn6km\" (UID: \"7d2aa425-fe38-4068-8db3-4813b822297e\") " pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:18:41.254387 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.254344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d2aa425-fe38-4068-8db3-4813b822297e-tls-certs\") pod \"simulator-w4-962a7d-kserve-5cc984d758-mn6km\" (UID: \"7d2aa425-fe38-4068-8db3-4813b822297e\") " pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:18:41.254493 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.254424    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d2aa425-fe38-4068-8db3-4813b822297e-home\") pod \"simulator-w4-962a7d-kserve-5cc984d758-mn6km\" (UID: \"7d2aa425-fe38-4068-8db3-4813b822297e\") " pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:18:41.254493 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.254455    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d2aa425-fe38-4068-8db3-4813b822297e-dshm\") pod \"simulator-w4-962a7d-kserve-5cc984d758-mn6km\" (UID: \"7d2aa425-fe38-4068-8db3-4813b822297e\") " pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:18:41.254589 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.254488    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d2aa425-fe38-4068-8db3-4813b822297e-model-cache\") pod \"simulator-w4-962a7d-kserve-5cc984d758-mn6km\" (UID: \"7d2aa425-fe38-4068-8db3-4813b822297e\") " pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:18:41.254589 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.254535    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d2aa425-fe38-4068-8db3-4813b822297e-tmp-dir\") pod \"simulator-w4-962a7d-kserve-5cc984d758-mn6km\" (UID: \"7d2aa425-fe38-4068-8db3-4813b822297e\") " pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:18:41.254873 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.254847    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d2aa425-fe38-4068-8db3-4813b822297e-home\") pod \"simulator-w4-962a7d-kserve-5cc984d758-mn6km\" (UID: \"7d2aa425-fe38-4068-8db3-4813b822297e\") " pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:18:41.256721 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.256691    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d2aa425-fe38-4068-8db3-4813b822297e-dshm\") pod \"simulator-w4-962a7d-kserve-5cc984d758-mn6km\" (UID: \"7d2aa425-fe38-4068-8db3-4813b822297e\") " pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:18:41.257139 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.257115    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d2aa425-fe38-4068-8db3-4813b822297e-tls-certs\") pod \"simulator-w4-962a7d-kserve-5cc984d758-mn6km\" (UID: \"7d2aa425-fe38-4068-8db3-4813b822297e\") " pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:18:41.262789 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.262768    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfn9c\" (UniqueName: \"kubernetes.io/projected/7d2aa425-fe38-4068-8db3-4813b822297e-kube-api-access-nfn9c\") pod \"simulator-w4-962a7d-kserve-5cc984d758-mn6km\" (UID: \"7d2aa425-fe38-4068-8db3-4813b822297e\") " pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:18:41.413207 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.413122    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:18:41.690322 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:41.690138    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"]
Sep 11 21:18:41.693044 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:18:41.693017    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d2aa425_fe38_4068_8db3_4813b822297e.slice/crio-1ae592aecbc8f5d8a88106fb273b64517a892a6693d35b739c036b579257438a WatchSource:0}: Error finding container 1ae592aecbc8f5d8a88106fb273b64517a892a6693d35b739c036b579257438a: Status 404 returned error can't find the container with id 1ae592aecbc8f5d8a88106fb273b64517a892a6693d35b739c036b579257438a
Sep 11 21:18:42.163449 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:42.163347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km" event={"ID":"7d2aa425-fe38-4068-8db3-4813b822297e","Type":"ContainerStarted","Data":"c52783fc2dbfe7eb9ec35d3084239e316ed0633cc1f1461f2d9f40a21b34cdb1"}
Sep 11 21:18:42.163449 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:42.163395    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km" event={"ID":"7d2aa425-fe38-4068-8db3-4813b822297e","Type":"ContainerStarted","Data":"1ae592aecbc8f5d8a88106fb273b64517a892a6693d35b739c036b579257438a"}
Sep 11 21:18:42.164746 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:42.164722    2578 generic.go:356] "Generic (PLEG): container finished" podID="b4cfe8fc-7176-42d0-a27e-a3d05efca597" containerID="76b290fb3d56ead04ad6cb38535ecc04150d2ec89ec28ab4e36bd9ebc65634ea" exitCode=0
Sep 11 21:18:42.164900 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:42.164761    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh" event={"ID":"b4cfe8fc-7176-42d0-a27e-a3d05efca597","Type":"ContainerDied","Data":"76b290fb3d56ead04ad6cb38535ecc04150d2ec89ec28ab4e36bd9ebc65634ea"}
Sep 11 21:18:42.165108 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:42.165095    2578 scope.go:117] "RemoveContainer" containerID="76b290fb3d56ead04ad6cb38535ecc04150d2ec89ec28ab4e36bd9ebc65634ea"
Sep 11 21:18:42.166103 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:42.166076    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:18:42.166149 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:42.166141    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:18:42.166186 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:42.166172    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:18:42.181068 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:42.181023    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km" podStartSLOduration=1.181006575 podStartE2EDuration="1.181006575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:18:42.179800993 +0000 UTC m=+788.883770283" watchObservedRunningTime="2026-09-11 21:18:42.181006575 +0000 UTC m=+788.884975922"
Sep 11 21:18:43.172781 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:43.172742    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh" event={"ID":"b4cfe8fc-7176-42d0-a27e-a3d05efca597","Type":"ContainerStarted","Data":"584da383a16812a10b012385306589e8855ad4f002c943826f4a633b2b71a634"}
Sep 11 21:18:43.686202 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:43.686162    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:43.686440 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:43.686415    2578 patch_prober.go:28] interesting pod/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 11 21:18:43.686606 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:43.686487    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh" podUID="b4cfe8fc-7176-42d0-a27e-a3d05efca597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 11 21:18:44.686656 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:44.686616    2578 patch_prober.go:28] interesting pod/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 11 21:18:44.687056 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:44.686672    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh" podUID="b4cfe8fc-7176-42d0-a27e-a3d05efca597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 11 21:18:45.686330 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:45.686295    2578 patch_prober.go:28] interesting pod/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 11 21:18:45.686521 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:45.686351    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh" podUID="b4cfe8fc-7176-42d0-a27e-a3d05efca597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 11 21:18:46.686040 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:46.686003    2578 patch_prober.go:28] interesting pod/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 11 21:18:46.686435 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:46.686075    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh" podUID="b4cfe8fc-7176-42d0-a27e-a3d05efca597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 11 21:18:47.686097 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:47.686051    2578 patch_prober.go:28] interesting pod/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 11 21:18:47.686556 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:47.686115    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh" podUID="b4cfe8fc-7176-42d0-a27e-a3d05efca597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 11 21:18:48.860761 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:48.860715    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:48.861303 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:48.861192    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:48.862430 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:48.862406    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:18:50.128283 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:50.128245    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:18:51.413918 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:51.413888    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:18:51.414339 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:51.414162    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:18:51.426748 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:51.426720    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:18:52.226472 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:52.226444    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:18:57.279648 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:57.279562    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj"]
Sep 11 21:18:57.280095 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:57.279810    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj" podUID="08366ec2-1677-4671-84a4-28bb9807b5e4" containerName="payload-pre-processing" containerID="cri-o://8b9b12b17b5577131f03b44414f971bc2e5caf973f5f70e1f0daa59479e44c9a" gracePeriod=30
Sep 11 21:18:57.536890 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:57.536806    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj"
Sep 11 21:18:57.601528 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:57.601495    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlk4x\" (UniqueName: \"kubernetes.io/projected/08366ec2-1677-4671-84a4-28bb9807b5e4-kube-api-access-mlk4x\") pod \"08366ec2-1677-4671-84a4-28bb9807b5e4\" (UID: \"08366ec2-1677-4671-84a4-28bb9807b5e4\") "
Sep 11 21:18:57.601717 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:57.601548    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08366ec2-1677-4671-84a4-28bb9807b5e4-plugins-config-volume\") pod \"08366ec2-1677-4671-84a4-28bb9807b5e4\" (UID: \"08366ec2-1677-4671-84a4-28bb9807b5e4\") "
Sep 11 21:18:57.602004 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:57.601979    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08366ec2-1677-4671-84a4-28bb9807b5e4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "08366ec2-1677-4671-84a4-28bb9807b5e4" (UID: "08366ec2-1677-4671-84a4-28bb9807b5e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:18:57.603770 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:57.603747    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08366ec2-1677-4671-84a4-28bb9807b5e4-kube-api-access-mlk4x" (OuterVolumeSpecName: "kube-api-access-mlk4x") pod "08366ec2-1677-4671-84a4-28bb9807b5e4" (UID: "08366ec2-1677-4671-84a4-28bb9807b5e4"). InnerVolumeSpecName "kube-api-access-mlk4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:18:57.703196 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:57.703153    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlk4x\" (UniqueName: \"kubernetes.io/projected/08366ec2-1677-4671-84a4-28bb9807b5e4-kube-api-access-mlk4x\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:18:57.703196 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:57.703189    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08366ec2-1677-4671-84a4-28bb9807b5e4-plugins-config-volume\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:18:58.250665 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:58.247214    2578 generic.go:356] "Generic (PLEG): container finished" podID="08366ec2-1677-4671-84a4-28bb9807b5e4" containerID="8b9b12b17b5577131f03b44414f971bc2e5caf973f5f70e1f0daa59479e44c9a" exitCode=0
Sep 11 21:18:58.250665 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:58.247329    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj" event={"ID":"08366ec2-1677-4671-84a4-28bb9807b5e4","Type":"ContainerDied","Data":"8b9b12b17b5577131f03b44414f971bc2e5caf973f5f70e1f0daa59479e44c9a"}
Sep 11 21:18:58.250665 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:58.247376    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj" event={"ID":"08366ec2-1677-4671-84a4-28bb9807b5e4","Type":"ContainerDied","Data":"200c2cc9329a0afc73360b85af64b18227f970c2ba127a5b027153efce5cec7e"}
Sep 11 21:18:58.250665 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:58.247404    2578 scope.go:117] "RemoveContainer" containerID="8b9b12b17b5577131f03b44414f971bc2e5caf973f5f70e1f0daa59479e44c9a"
Sep 11 21:18:58.250665 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:58.247575    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj"
Sep 11 21:18:58.265111 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:58.264833    2578 scope.go:117] "RemoveContainer" containerID="8b9b12b17b5577131f03b44414f971bc2e5caf973f5f70e1f0daa59479e44c9a"
Sep 11 21:18:58.265487 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:18:58.265398    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b9b12b17b5577131f03b44414f971bc2e5caf973f5f70e1f0daa59479e44c9a\": container with ID starting with 8b9b12b17b5577131f03b44414f971bc2e5caf973f5f70e1f0daa59479e44c9a not found: ID does not exist" containerID="8b9b12b17b5577131f03b44414f971bc2e5caf973f5f70e1f0daa59479e44c9a"
Sep 11 21:18:58.265487 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:58.265438    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b9b12b17b5577131f03b44414f971bc2e5caf973f5f70e1f0daa59479e44c9a"} err="failed to get container status \"8b9b12b17b5577131f03b44414f971bc2e5caf973f5f70e1f0daa59479e44c9a\": rpc error: code = NotFound desc = could not find container \"8b9b12b17b5577131f03b44414f971bc2e5caf973f5f70e1f0daa59479e44c9a\": container with ID starting with 8b9b12b17b5577131f03b44414f971bc2e5caf973f5f70e1f0daa59479e44c9a not found: ID does not exist"
Sep 11 21:18:58.271806 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:58.271771    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj"]
Sep 11 21:18:58.278285 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:58.278256    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-257d9182-7b8498cd-tqrqj"]
Sep 11 21:18:58.659169 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:58.659077    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"]
Sep 11 21:18:58.659593 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:58.659334    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh" podUID="b4cfe8fc-7176-42d0-a27e-a3d05efca597" containerName="istio-proxy" containerID="cri-o://584da383a16812a10b012385306589e8855ad4f002c943826f4a633b2b71a634" gracePeriod=30
Sep 11 21:18:59.389254 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.389221    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"]
Sep 11 21:18:59.389653 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.389636    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08366ec2-1677-4671-84a4-28bb9807b5e4" containerName="payload-pre-processing"
Sep 11 21:18:59.389715 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.389654    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="08366ec2-1677-4671-84a4-28bb9807b5e4" containerName="payload-pre-processing"
Sep 11 21:18:59.389715 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.389665    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="644cf0ad-7860-4448-937c-d6e195947628" containerName="istio-proxy"
Sep 11 21:18:59.389715 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.389673    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="644cf0ad-7860-4448-937c-d6e195947628" containerName="istio-proxy"
Sep 11 21:18:59.389814 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.389744    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="08366ec2-1677-4671-84a4-28bb9807b5e4" containerName="payload-pre-processing"
Sep 11 21:18:59.392706 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.392680    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.395942 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.395923    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-5fdb611f-openshift-default-dockercfg-wrdpr\""
Sep 11 21:18:59.409268 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.409237    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"]
Sep 11 21:18:59.521001 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.520961    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-istio-data\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.521169 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.521017    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cq6d\" (UniqueName: \"kubernetes.io/projected/e5d86863-2a31-41e3-9e83-c8308e10384c-kube-api-access-2cq6d\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.521169 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.521046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-credential-socket\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.521169 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.521070    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-istio-envoy\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.521169 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.521085    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5d86863-2a31-41e3-9e83-c8308e10384c-istio-podinfo\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.521169 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.521106    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-workload-socket\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.521169 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.521122    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-workload-certs\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.521169 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.521152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5d86863-2a31-41e3-9e83-c8308e10384c-istiod-ca-cert\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.521393 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.521182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5d86863-2a31-41e3-9e83-c8308e10384c-istio-token\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.622479 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.622442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-credential-socket\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.622479 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.622482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-istio-envoy\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.622701 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.622504    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5d86863-2a31-41e3-9e83-c8308e10384c-istio-podinfo\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.622701 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.622525    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-workload-socket\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.622701 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.622547    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-workload-certs\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.622701 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.622584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5d86863-2a31-41e3-9e83-c8308e10384c-istiod-ca-cert\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.622701 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.622629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5d86863-2a31-41e3-9e83-c8308e10384c-istio-token\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.622701 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.622689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-istio-data\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.623006 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.622730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cq6d\" (UniqueName: \"kubernetes.io/projected/e5d86863-2a31-41e3-9e83-c8308e10384c-kube-api-access-2cq6d\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.623006 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.622984    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-workload-certs\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.623111 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.623051    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-credential-socket\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.623259 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.623237    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-workload-socket\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.623351 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.623280    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-istio-data\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.623598 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.623577    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5d86863-2a31-41e3-9e83-c8308e10384c-istiod-ca-cert\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.624918 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.624897    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-istio-envoy\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.625216 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.625199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5d86863-2a31-41e3-9e83-c8308e10384c-istio-podinfo\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.630899 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.630865    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5d86863-2a31-41e3-9e83-c8308e10384c-istio-token\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.631062 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.631043    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cq6d\" (UniqueName: \"kubernetes.io/projected/e5d86863-2a31-41e3-9e83-c8308e10384c-kube-api-access-2cq6d\") pod \"e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") " pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.703370 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.703333    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:18:59.927495 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.927454    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08366ec2-1677-4671-84a4-28bb9807b5e4" path="/var/lib/kubelet/pods/08366ec2-1677-4671-84a4-28bb9807b5e4/volumes"
Sep 11 21:18:59.982635 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.982545    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"]
Sep 11 21:18:59.985229 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:18:59.985193    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d86863_2a31_41e3_9e83_c8308e10384c.slice/crio-1ed8f082eb209920988a43a6b6ab33a72c450814aeb20a0883bd1201bc69ecbe WatchSource:0}: Error finding container 1ed8f082eb209920988a43a6b6ab33a72c450814aeb20a0883bd1201bc69ecbe: Status 404 returned error can't find the container with id 1ed8f082eb209920988a43a6b6ab33a72c450814aeb20a0883bd1201bc69ecbe
Sep 11 21:18:59.987645 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.987614    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:18:59.987711 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.987688    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:18:59.987750 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:18:59.987720    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:19:00.260683 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:00.260582    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6" event={"ID":"e5d86863-2a31-41e3-9e83-c8308e10384c","Type":"ContainerStarted","Data":"5d8493fb6d501a8608b01808539b3d8c148e8f47f7e1181f2e19be785706aefd"}
Sep 11 21:19:00.260683 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:00.260642    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6" event={"ID":"e5d86863-2a31-41e3-9e83-c8308e10384c","Type":"ContainerStarted","Data":"1ed8f082eb209920988a43a6b6ab33a72c450814aeb20a0883bd1201bc69ecbe"}
Sep 11 21:19:00.280962 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:00.280900    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6" podStartSLOduration=1.280876199 podStartE2EDuration="1.280876199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:19:00.279730518 +0000 UTC m=+806.983699810" watchObservedRunningTime="2026-09-11 21:19:00.280876199 +0000 UTC m=+806.984845491"
Sep 11 21:19:00.704446 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:00.704407    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:19:01.709754 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:01.709720    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:19:02.275881 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:02.275847    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:19:02.276745 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:02.276725    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:19:03.904866 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:03.904838    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:19:03.965583 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:03.965552    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istio-podinfo\") pod \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") "
Sep 11 21:19:03.965841 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:03.965605    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istio-data\") pod \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") "
Sep 11 21:19:03.965841 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:03.965629    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2kl8\" (UniqueName: \"kubernetes.io/projected/b4cfe8fc-7176-42d0-a27e-a3d05efca597-kube-api-access-m2kl8\") pod \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") "
Sep 11 21:19:03.965841 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:03.965659    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istio-envoy\") pod \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") "
Sep 11 21:19:03.965841 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:03.965698    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istiod-ca-cert\") pod \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") "
Sep 11 21:19:03.965841 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:03.965760    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istio-token\") pod \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") "
Sep 11 21:19:03.965841 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:03.965801    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-workload-certs\") pod \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") "
Sep 11 21:19:03.966174 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:03.965861    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-credential-socket\") pod \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") "
Sep 11 21:19:03.966174 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:03.965899    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-workload-socket\") pod \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\" (UID: \"b4cfe8fc-7176-42d0-a27e-a3d05efca597\") "
Sep 11 21:19:03.966174 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:03.966018    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istio-data" (OuterVolumeSpecName: "istio-data") pod "b4cfe8fc-7176-42d0-a27e-a3d05efca597" (UID: "b4cfe8fc-7176-42d0-a27e-a3d05efca597"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:19:03.966328 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:03.966207    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istio-data\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:19:03.966328 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:03.966154    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b4cfe8fc-7176-42d0-a27e-a3d05efca597" (UID: "b4cfe8fc-7176-42d0-a27e-a3d05efca597"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:19:03.966328 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:03.966241    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b4cfe8fc-7176-42d0-a27e-a3d05efca597" (UID: "b4cfe8fc-7176-42d0-a27e-a3d05efca597"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:19:03.966476 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:03.966338    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b4cfe8fc-7176-42d0-a27e-a3d05efca597" (UID: "b4cfe8fc-7176-42d0-a27e-a3d05efca597"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:19:03.966476 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:03.966410    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b4cfe8fc-7176-42d0-a27e-a3d05efca597" (UID: "b4cfe8fc-7176-42d0-a27e-a3d05efca597"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:19:03.968131 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:03.968087    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4cfe8fc-7176-42d0-a27e-a3d05efca597-kube-api-access-m2kl8" (OuterVolumeSpecName: "kube-api-access-m2kl8") pod "b4cfe8fc-7176-42d0-a27e-a3d05efca597" (UID: "b4cfe8fc-7176-42d0-a27e-a3d05efca597"). InnerVolumeSpecName "kube-api-access-m2kl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:19:03.968282 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:03.968217    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b4cfe8fc-7176-42d0-a27e-a3d05efca597" (UID: "b4cfe8fc-7176-42d0-a27e-a3d05efca597"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:19:03.968589 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:03.968563    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istio-token" (OuterVolumeSpecName: "istio-token") pod "b4cfe8fc-7176-42d0-a27e-a3d05efca597" (UID: "b4cfe8fc-7176-42d0-a27e-a3d05efca597"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:19:03.968683 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:03.968607    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b4cfe8fc-7176-42d0-a27e-a3d05efca597" (UID: "b4cfe8fc-7176-42d0-a27e-a3d05efca597"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:19:04.067246 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.067201    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istio-token\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:19:04.067246 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.067241    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-workload-certs\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:19:04.067246 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.067255    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-credential-socket\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:19:04.067491 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.067269    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-workload-socket\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:19:04.067491 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.067284    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istio-podinfo\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:19:04.067491 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.067297    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2kl8\" (UniqueName: \"kubernetes.io/projected/b4cfe8fc-7176-42d0-a27e-a3d05efca597-kube-api-access-m2kl8\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:19:04.067491 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.067311    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istio-envoy\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:19:04.067491 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.067324    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cfe8fc-7176-42d0-a27e-a3d05efca597-istiod-ca-cert\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:19:04.287610 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.287517    2578 generic.go:356] "Generic (PLEG): container finished" podID="b4cfe8fc-7176-42d0-a27e-a3d05efca597" containerID="584da383a16812a10b012385306589e8855ad4f002c943826f4a633b2b71a634" exitCode=0
Sep 11 21:19:04.287610 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.287585    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"
Sep 11 21:19:04.287610 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.287601    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh" event={"ID":"b4cfe8fc-7176-42d0-a27e-a3d05efca597","Type":"ContainerDied","Data":"584da383a16812a10b012385306589e8855ad4f002c943826f4a633b2b71a634"}
Sep 11 21:19:04.287900 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.287663    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh" event={"ID":"b4cfe8fc-7176-42d0-a27e-a3d05efca597","Type":"ContainerDied","Data":"2862b70c926a859e85e065a91a2a2bd30e441517b98dd0f7be7fa78604eecd04"}
Sep 11 21:19:04.287900 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.287695    2578 scope.go:117] "RemoveContainer" containerID="584da383a16812a10b012385306589e8855ad4f002c943826f4a633b2b71a634"
Sep 11 21:19:04.297275 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.297230    2578 scope.go:117] "RemoveContainer" containerID="76b290fb3d56ead04ad6cb38535ecc04150d2ec89ec28ab4e36bd9ebc65634ea"
Sep 11 21:19:04.306800 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.306766    2578 scope.go:117] "RemoveContainer" containerID="584da383a16812a10b012385306589e8855ad4f002c943826f4a633b2b71a634"
Sep 11 21:19:04.307157 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:19:04.307133    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"584da383a16812a10b012385306589e8855ad4f002c943826f4a633b2b71a634\": container with ID starting with 584da383a16812a10b012385306589e8855ad4f002c943826f4a633b2b71a634 not found: ID does not exist" containerID="584da383a16812a10b012385306589e8855ad4f002c943826f4a633b2b71a634"
Sep 11 21:19:04.307246 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.307166    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"584da383a16812a10b012385306589e8855ad4f002c943826f4a633b2b71a634"} err="failed to get container status \"584da383a16812a10b012385306589e8855ad4f002c943826f4a633b2b71a634\": rpc error: code = NotFound desc = could not find container \"584da383a16812a10b012385306589e8855ad4f002c943826f4a633b2b71a634\": container with ID starting with 584da383a16812a10b012385306589e8855ad4f002c943826f4a633b2b71a634 not found: ID does not exist"
Sep 11 21:19:04.307246 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.307184    2578 scope.go:117] "RemoveContainer" containerID="76b290fb3d56ead04ad6cb38535ecc04150d2ec89ec28ab4e36bd9ebc65634ea"
Sep 11 21:19:04.307444 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:19:04.307428    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76b290fb3d56ead04ad6cb38535ecc04150d2ec89ec28ab4e36bd9ebc65634ea\": container with ID starting with 76b290fb3d56ead04ad6cb38535ecc04150d2ec89ec28ab4e36bd9ebc65634ea not found: ID does not exist" containerID="76b290fb3d56ead04ad6cb38535ecc04150d2ec89ec28ab4e36bd9ebc65634ea"
Sep 11 21:19:04.307486 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.307447    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76b290fb3d56ead04ad6cb38535ecc04150d2ec89ec28ab4e36bd9ebc65634ea"} err="failed to get container status \"76b290fb3d56ead04ad6cb38535ecc04150d2ec89ec28ab4e36bd9ebc65634ea\": rpc error: code = NotFound desc = could not find container \"76b290fb3d56ead04ad6cb38535ecc04150d2ec89ec28ab4e36bd9ebc65634ea\": container with ID starting with 76b290fb3d56ead04ad6cb38535ecc04150d2ec89ec28ab4e36bd9ebc65634ea not found: ID does not exist"
Sep 11 21:19:04.313755 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.313732    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"]
Sep 11 21:19:04.320949 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.320920    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh"]
Sep 11 21:19:04.576688 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.576591    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh"]
Sep 11 21:19:04.576966 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.576953    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4cfe8fc-7176-42d0-a27e-a3d05efca597" containerName="istio-proxy"
Sep 11 21:19:04.577032 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.576967    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4cfe8fc-7176-42d0-a27e-a3d05efca597" containerName="istio-proxy"
Sep 11 21:19:04.577032 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.576986    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4cfe8fc-7176-42d0-a27e-a3d05efca597" containerName="istio-proxy"
Sep 11 21:19:04.577032 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.576993    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4cfe8fc-7176-42d0-a27e-a3d05efca597" containerName="istio-proxy"
Sep 11 21:19:04.577136 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.577046    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4cfe8fc-7176-42d0-a27e-a3d05efca597" containerName="istio-proxy"
Sep 11 21:19:04.581497 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.581476    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh"
Sep 11 21:19:04.584465 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.584441    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-5fdb611f\""
Sep 11 21:19:04.584601 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.584522    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-5fdb611f-dockercfg-lct28\""
Sep 11 21:19:04.595343 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.595318    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh"]
Sep 11 21:19:04.672107 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.672074    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b94901-f911-45bf-bedf-faa66d25e089-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh\" (UID: \"a9b94901-f911-45bf-bedf-faa66d25e089\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh"
Sep 11 21:19:04.672295 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.672133    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4nx7\" (UniqueName: \"kubernetes.io/projected/a9b94901-f911-45bf-bedf-faa66d25e089-kube-api-access-w4nx7\") pod \"payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh\" (UID: \"a9b94901-f911-45bf-bedf-faa66d25e089\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh"
Sep 11 21:19:04.772916 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.772869    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4nx7\" (UniqueName: \"kubernetes.io/projected/a9b94901-f911-45bf-bedf-faa66d25e089-kube-api-access-w4nx7\") pod \"payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh\" (UID: \"a9b94901-f911-45bf-bedf-faa66d25e089\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh"
Sep 11 21:19:04.773137 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.772975    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b94901-f911-45bf-bedf-faa66d25e089-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh\" (UID: \"a9b94901-f911-45bf-bedf-faa66d25e089\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh"
Sep 11 21:19:04.773661 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.773637    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b94901-f911-45bf-bedf-faa66d25e089-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh\" (UID: \"a9b94901-f911-45bf-bedf-faa66d25e089\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh"
Sep 11 21:19:04.782321 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.782297    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4nx7\" (UniqueName: \"kubernetes.io/projected/a9b94901-f911-45bf-bedf-faa66d25e089-kube-api-access-w4nx7\") pod \"payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh\" (UID: \"a9b94901-f911-45bf-bedf-faa66d25e089\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh"
Sep 11 21:19:04.861774 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.861664    2578 patch_prober.go:28] interesting pod/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 21:19:04.861774 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.861747    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-257d9182-openshift-default-78fd69dc76-ghhlh" podUID="b4cfe8fc-7176-42d0-a27e-a3d05efca597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 21:19:04.891552 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:04.891513    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh"
Sep 11 21:19:05.163059 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:05.162906    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh"]
Sep 11 21:19:05.165804 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:19:05.165776    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b94901_f911_45bf_bedf_faa66d25e089.slice/crio-71cd171d456c845ba6c67413b181ee9b4f099ec4bc0b66c8213f3b23e91a372f WatchSource:0}: Error finding container 71cd171d456c845ba6c67413b181ee9b4f099ec4bc0b66c8213f3b23e91a372f: Status 404 returned error can't find the container with id 71cd171d456c845ba6c67413b181ee9b4f099ec4bc0b66c8213f3b23e91a372f
Sep 11 21:19:05.294552 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:05.294516    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh" event={"ID":"a9b94901-f911-45bf-bedf-faa66d25e089","Type":"ContainerStarted","Data":"0d1aa86351bc0a817605964b389852c33971b7f6fb8516f51bcee67ae511e4d2"}
Sep 11 21:19:05.294715 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:05.294570    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh" event={"ID":"a9b94901-f911-45bf-bedf-faa66d25e089","Type":"ContainerStarted","Data":"71cd171d456c845ba6c67413b181ee9b4f099ec4bc0b66c8213f3b23e91a372f"}
Sep 11 21:19:05.294715 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:05.294612    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh"
Sep 11 21:19:05.314899 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:05.314843    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh" podStartSLOduration=1.314811127 podStartE2EDuration="1.314811127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:19:05.312550221 +0000 UTC m=+812.016519511" watchObservedRunningTime="2026-09-11 21:19:05.314811127 +0000 UTC m=+812.018780416"
Sep 11 21:19:05.927989 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:05.927953    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4cfe8fc-7176-42d0-a27e-a3d05efca597" path="/var/lib/kubelet/pods/b4cfe8fc-7176-42d0-a27e-a3d05efca597/volumes"
Sep 11 21:19:07.112978 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:07.112939    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:19:16.302028 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:16.301990    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh"
Sep 11 21:19:23.716878 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:23.716836    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:19:24.922102 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:24.922069    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"]
Sep 11 21:19:24.922488 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:24.922461    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4cfe8fc-7176-42d0-a27e-a3d05efca597" containerName="istio-proxy"
Sep 11 21:19:24.925235 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:24.925218    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:19:24.928419 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:24.928390    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-765a18\"/\"default-dockercfg-w5s5m\""
Sep 11 21:19:24.928419 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:24.928401    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-765a18\"/\"kube-root-ca.crt\""
Sep 11 21:19:24.928619 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:24.928394    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-765a18\"/\"openshift-service-ca.crt\""
Sep 11 21:19:24.928619 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:24.928390    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-765a18\"/\"autores-model-765a18-kserve-self-signed-certs\""
Sep 11 21:19:24.937116 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:24.937087    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"]
Sep 11 21:19:25.057672 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.057597    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e200df09-7c52-432b-a646-5c6e28d28b0a-model-cache\") pod \"autores-model-765a18-kserve-8d7c47475-k4jw4\" (UID: \"e200df09-7c52-432b-a646-5c6e28d28b0a\") " pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:19:25.057672 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.057630    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e200df09-7c52-432b-a646-5c6e28d28b0a-tls-certs\") pod \"autores-model-765a18-kserve-8d7c47475-k4jw4\" (UID: \"e200df09-7c52-432b-a646-5c6e28d28b0a\") " pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:19:25.057878 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.057700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e200df09-7c52-432b-a646-5c6e28d28b0a-home\") pod \"autores-model-765a18-kserve-8d7c47475-k4jw4\" (UID: \"e200df09-7c52-432b-a646-5c6e28d28b0a\") " pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:19:25.057878 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.057749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsktl\" (UniqueName: \"kubernetes.io/projected/e200df09-7c52-432b-a646-5c6e28d28b0a-kube-api-access-xsktl\") pod \"autores-model-765a18-kserve-8d7c47475-k4jw4\" (UID: \"e200df09-7c52-432b-a646-5c6e28d28b0a\") " pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:19:25.057878 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.057811    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e200df09-7c52-432b-a646-5c6e28d28b0a-dshm\") pod \"autores-model-765a18-kserve-8d7c47475-k4jw4\" (UID: \"e200df09-7c52-432b-a646-5c6e28d28b0a\") " pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:19:25.057878 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.057846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e200df09-7c52-432b-a646-5c6e28d28b0a-tmp-dir\") pod \"autores-model-765a18-kserve-8d7c47475-k4jw4\" (UID: \"e200df09-7c52-432b-a646-5c6e28d28b0a\") " pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:19:25.159234 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.159154    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e200df09-7c52-432b-a646-5c6e28d28b0a-home\") pod \"autores-model-765a18-kserve-8d7c47475-k4jw4\" (UID: \"e200df09-7c52-432b-a646-5c6e28d28b0a\") " pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:19:25.159234 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.159199    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsktl\" (UniqueName: \"kubernetes.io/projected/e200df09-7c52-432b-a646-5c6e28d28b0a-kube-api-access-xsktl\") pod \"autores-model-765a18-kserve-8d7c47475-k4jw4\" (UID: \"e200df09-7c52-432b-a646-5c6e28d28b0a\") " pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:19:25.159426 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.159243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e200df09-7c52-432b-a646-5c6e28d28b0a-dshm\") pod \"autores-model-765a18-kserve-8d7c47475-k4jw4\" (UID: \"e200df09-7c52-432b-a646-5c6e28d28b0a\") " pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:19:25.159426 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.159259    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e200df09-7c52-432b-a646-5c6e28d28b0a-tmp-dir\") pod \"autores-model-765a18-kserve-8d7c47475-k4jw4\" (UID: \"e200df09-7c52-432b-a646-5c6e28d28b0a\") " pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:19:25.159426 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.159284    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e200df09-7c52-432b-a646-5c6e28d28b0a-model-cache\") pod \"autores-model-765a18-kserve-8d7c47475-k4jw4\" (UID: \"e200df09-7c52-432b-a646-5c6e28d28b0a\") " pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:19:25.159426 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.159300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e200df09-7c52-432b-a646-5c6e28d28b0a-tls-certs\") pod \"autores-model-765a18-kserve-8d7c47475-k4jw4\" (UID: \"e200df09-7c52-432b-a646-5c6e28d28b0a\") " pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:19:25.159615 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.159594    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e200df09-7c52-432b-a646-5c6e28d28b0a-home\") pod \"autores-model-765a18-kserve-8d7c47475-k4jw4\" (UID: \"e200df09-7c52-432b-a646-5c6e28d28b0a\") " pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:19:25.159686 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.159669    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e200df09-7c52-432b-a646-5c6e28d28b0a-model-cache\") pod \"autores-model-765a18-kserve-8d7c47475-k4jw4\" (UID: \"e200df09-7c52-432b-a646-5c6e28d28b0a\") " pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:19:25.159736 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.159706    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e200df09-7c52-432b-a646-5c6e28d28b0a-tmp-dir\") pod \"autores-model-765a18-kserve-8d7c47475-k4jw4\" (UID: \"e200df09-7c52-432b-a646-5c6e28d28b0a\") " pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:19:25.161491 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.161467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e200df09-7c52-432b-a646-5c6e28d28b0a-dshm\") pod \"autores-model-765a18-kserve-8d7c47475-k4jw4\" (UID: \"e200df09-7c52-432b-a646-5c6e28d28b0a\") " pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:19:25.161882 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.161863    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e200df09-7c52-432b-a646-5c6e28d28b0a-tls-certs\") pod \"autores-model-765a18-kserve-8d7c47475-k4jw4\" (UID: \"e200df09-7c52-432b-a646-5c6e28d28b0a\") " pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:19:25.167096 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.167072    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsktl\" (UniqueName: \"kubernetes.io/projected/e200df09-7c52-432b-a646-5c6e28d28b0a-kube-api-access-xsktl\") pod \"autores-model-765a18-kserve-8d7c47475-k4jw4\" (UID: \"e200df09-7c52-432b-a646-5c6e28d28b0a\") " pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:19:25.242562 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.242519    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:19:25.391264 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.391231    2578 generic.go:356] "Generic (PLEG): container finished" podID="e5d86863-2a31-41e3-9e83-c8308e10384c" containerID="5d8493fb6d501a8608b01808539b3d8c148e8f47f7e1181f2e19be785706aefd" exitCode=0
Sep 11 21:19:25.391429 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.391285    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6" event={"ID":"e5d86863-2a31-41e3-9e83-c8308e10384c","Type":"ContainerDied","Data":"5d8493fb6d501a8608b01808539b3d8c148e8f47f7e1181f2e19be785706aefd"}
Sep 11 21:19:25.391697 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.391681    2578 scope.go:117] "RemoveContainer" containerID="5d8493fb6d501a8608b01808539b3d8c148e8f47f7e1181f2e19be785706aefd"
Sep 11 21:19:25.392752 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.392723    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:19:25.392864 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.392791    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:19:25.392864 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.392844    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:19:25.529290 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:25.529254    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"]
Sep 11 21:19:25.532529 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:19:25.532492    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode200df09_7c52_432b_a646_5c6e28d28b0a.slice/crio-8e2face21635a3f806666bf2337667ec2dec5b7f82160c1bb0ce6d9b1f3eead5 WatchSource:0}: Error finding container 8e2face21635a3f806666bf2337667ec2dec5b7f82160c1bb0ce6d9b1f3eead5: Status 404 returned error can't find the container with id 8e2face21635a3f806666bf2337667ec2dec5b7f82160c1bb0ce6d9b1f3eead5
Sep 11 21:19:26.397479 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:26.397441    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6" event={"ID":"e5d86863-2a31-41e3-9e83-c8308e10384c","Type":"ContainerStarted","Data":"08f921140180aeb19d1a432f75dea5ab733dd57f21b7cdfa626a63cc2ff77d50"}
Sep 11 21:19:26.399058 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:26.399032    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4" event={"ID":"e200df09-7c52-432b-a646-5c6e28d28b0a","Type":"ContainerStarted","Data":"e1d9b5dcd5493f8e43c4d26f9edc84ef69bb4a3a0c8f1b631bcce6f00f0e257b"}
Sep 11 21:19:26.399192 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:26.399067    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4" event={"ID":"e200df09-7c52-432b-a646-5c6e28d28b0a","Type":"ContainerStarted","Data":"8e2face21635a3f806666bf2337667ec2dec5b7f82160c1bb0ce6d9b1f3eead5"}
Sep 11 21:19:26.447339 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:26.446414    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4" podStartSLOduration=2.446391067 podStartE2EDuration="2.446391067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:19:26.443423909 +0000 UTC m=+833.147393201" watchObservedRunningTime="2026-09-11 21:19:26.446391067 +0000 UTC m=+833.150360357"
Sep 11 21:19:26.704119 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:26.704077    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:19:26.704317 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:26.704284    2578 patch_prober.go:28] interesting pod/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 11 21:19:26.704369 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:26.704328    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6" podUID="e5d86863-2a31-41e3-9e83-c8308e10384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 11 21:19:27.704566 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:27.704529    2578 patch_prober.go:28] interesting pod/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 11 21:19:27.705056 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:27.704591    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6" podUID="e5d86863-2a31-41e3-9e83-c8308e10384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 11 21:19:28.704059 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:28.704020    2578 patch_prober.go:28] interesting pod/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 11 21:19:28.704248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:28.704090    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6" podUID="e5d86863-2a31-41e3-9e83-c8308e10384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 11 21:19:29.704532 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:29.704484    2578 patch_prober.go:28] interesting pod/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 11 21:19:29.704987 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:29.704552    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6" podUID="e5d86863-2a31-41e3-9e83-c8308e10384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 11 21:19:30.703896 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:30.703850    2578 patch_prober.go:28] interesting pod/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 11 21:19:30.704076 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:30.703919    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6" podUID="e5d86863-2a31-41e3-9e83-c8308e10384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 11 21:19:31.708209 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:31.708180    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:19:31.708659 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:31.708463    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:19:31.709165 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:31.709150    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:19:33.730860 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:33.730811    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh"]
Sep 11 21:19:33.731403 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:33.731062    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh" podUID="a9b94901-f911-45bf-bedf-faa66d25e089" containerName="payload-pre-processing" containerID="cri-o://0d1aa86351bc0a817605964b389852c33971b7f6fb8516f51bcee67ae511e4d2" gracePeriod=30
Sep 11 21:19:33.981204 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:33.981156    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh"
Sep 11 21:19:34.023322 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:34.023291    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"]
Sep 11 21:19:34.027904 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:34.027878    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b94901-f911-45bf-bedf-faa66d25e089-plugins-config-volume\") pod \"a9b94901-f911-45bf-bedf-faa66d25e089\" (UID: \"a9b94901-f911-45bf-bedf-faa66d25e089\") "
Sep 11 21:19:34.028069 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:34.027949    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4nx7\" (UniqueName: \"kubernetes.io/projected/a9b94901-f911-45bf-bedf-faa66d25e089-kube-api-access-w4nx7\") pod \"a9b94901-f911-45bf-bedf-faa66d25e089\" (UID: \"a9b94901-f911-45bf-bedf-faa66d25e089\") "
Sep 11 21:19:34.028422 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:34.028391    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b94901-f911-45bf-bedf-faa66d25e089-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a9b94901-f911-45bf-bedf-faa66d25e089" (UID: "a9b94901-f911-45bf-bedf-faa66d25e089"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:19:34.031497 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:34.031465    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b94901-f911-45bf-bedf-faa66d25e089-kube-api-access-w4nx7" (OuterVolumeSpecName: "kube-api-access-w4nx7") pod "a9b94901-f911-45bf-bedf-faa66d25e089" (UID: "a9b94901-f911-45bf-bedf-faa66d25e089"). InnerVolumeSpecName "kube-api-access-w4nx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:19:34.128659 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:34.128628    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b94901-f911-45bf-bedf-faa66d25e089-plugins-config-volume\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:19:34.128870 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:34.128746    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4nx7\" (UniqueName: \"kubernetes.io/projected/a9b94901-f911-45bf-bedf-faa66d25e089-kube-api-access-w4nx7\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:19:34.439732 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:34.439691    2578 generic.go:356] "Generic (PLEG): container finished" podID="a9b94901-f911-45bf-bedf-faa66d25e089" containerID="0d1aa86351bc0a817605964b389852c33971b7f6fb8516f51bcee67ae511e4d2" exitCode=0
Sep 11 21:19:34.439937 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:34.439757    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh"
Sep 11 21:19:34.439937 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:34.439773    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh" event={"ID":"a9b94901-f911-45bf-bedf-faa66d25e089","Type":"ContainerDied","Data":"0d1aa86351bc0a817605964b389852c33971b7f6fb8516f51bcee67ae511e4d2"}
Sep 11 21:19:34.439937 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:34.439868    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh" event={"ID":"a9b94901-f911-45bf-bedf-faa66d25e089","Type":"ContainerDied","Data":"71cd171d456c845ba6c67413b181ee9b4f099ec4bc0b66c8213f3b23e91a372f"}
Sep 11 21:19:34.439937 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:34.439894    2578 scope.go:117] "RemoveContainer" containerID="0d1aa86351bc0a817605964b389852c33971b7f6fb8516f51bcee67ae511e4d2"
Sep 11 21:19:34.440124 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:34.440017    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6" podUID="e5d86863-2a31-41e3-9e83-c8308e10384c" containerName="istio-proxy" containerID="cri-o://08f921140180aeb19d1a432f75dea5ab733dd57f21b7cdfa626a63cc2ff77d50" gracePeriod=30
Sep 11 21:19:34.448201 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:34.448186    2578 scope.go:117] "RemoveContainer" containerID="0d1aa86351bc0a817605964b389852c33971b7f6fb8516f51bcee67ae511e4d2"
Sep 11 21:19:34.448542 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:19:34.448515    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d1aa86351bc0a817605964b389852c33971b7f6fb8516f51bcee67ae511e4d2\": container with ID starting with 0d1aa86351bc0a817605964b389852c33971b7f6fb8516f51bcee67ae511e4d2 not found: ID does not exist" containerID="0d1aa86351bc0a817605964b389852c33971b7f6fb8516f51bcee67ae511e4d2"
Sep 11 21:19:34.448641 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:34.448550    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d1aa86351bc0a817605964b389852c33971b7f6fb8516f51bcee67ae511e4d2"} err="failed to get container status \"0d1aa86351bc0a817605964b389852c33971b7f6fb8516f51bcee67ae511e4d2\": rpc error: code = NotFound desc = could not find container \"0d1aa86351bc0a817605964b389852c33971b7f6fb8516f51bcee67ae511e4d2\": container with ID starting with 0d1aa86351bc0a817605964b389852c33971b7f6fb8516f51bcee67ae511e4d2 not found: ID does not exist"
Sep 11 21:19:34.528943 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:34.528904    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh"]
Sep 11 21:19:34.537058 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:34.537029    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5fdb611f-6f6c9d986-n4khh"]
Sep 11 21:19:35.243615 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:35.243582    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:19:35.244092 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:35.243635    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:19:35.256529 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:35.256486    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:19:35.456983 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:35.456926    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:19:35.927485 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:35.927449    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9b94901-f911-45bf-bedf-faa66d25e089" path="/var/lib/kubelet/pods/a9b94901-f911-45bf-bedf-faa66d25e089/volumes"
Sep 11 21:19:39.688549 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.688523    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:19:39.780600 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.780507    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cq6d\" (UniqueName: \"kubernetes.io/projected/e5d86863-2a31-41e3-9e83-c8308e10384c-kube-api-access-2cq6d\") pod \"e5d86863-2a31-41e3-9e83-c8308e10384c\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") "
Sep 11 21:19:39.780600 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.780554    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-istio-data\") pod \"e5d86863-2a31-41e3-9e83-c8308e10384c\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") "
Sep 11 21:19:39.780600 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.780588    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5d86863-2a31-41e3-9e83-c8308e10384c-istio-podinfo\") pod \"e5d86863-2a31-41e3-9e83-c8308e10384c\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") "
Sep 11 21:19:39.780949 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.780655    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-credential-socket\") pod \"e5d86863-2a31-41e3-9e83-c8308e10384c\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") "
Sep 11 21:19:39.780949 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.780689    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5d86863-2a31-41e3-9e83-c8308e10384c-istiod-ca-cert\") pod \"e5d86863-2a31-41e3-9e83-c8308e10384c\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") "
Sep 11 21:19:39.780949 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.780712    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5d86863-2a31-41e3-9e83-c8308e10384c-istio-token\") pod \"e5d86863-2a31-41e3-9e83-c8308e10384c\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") "
Sep 11 21:19:39.780949 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.780740    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-workload-socket\") pod \"e5d86863-2a31-41e3-9e83-c8308e10384c\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") "
Sep 11 21:19:39.780949 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.780774    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-istio-envoy\") pod \"e5d86863-2a31-41e3-9e83-c8308e10384c\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") "
Sep 11 21:19:39.780949 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.780875    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-workload-certs\") pod \"e5d86863-2a31-41e3-9e83-c8308e10384c\" (UID: \"e5d86863-2a31-41e3-9e83-c8308e10384c\") "
Sep 11 21:19:39.781234 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.781016    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-istio-data" (OuterVolumeSpecName: "istio-data") pod "e5d86863-2a31-41e3-9e83-c8308e10384c" (UID: "e5d86863-2a31-41e3-9e83-c8308e10384c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:19:39.781234 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.781165    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-istio-data\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:19:39.781340 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.781235    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e5d86863-2a31-41e3-9e83-c8308e10384c" (UID: "e5d86863-2a31-41e3-9e83-c8308e10384c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:19:39.781340 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.781179    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5d86863-2a31-41e3-9e83-c8308e10384c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e5d86863-2a31-41e3-9e83-c8308e10384c" (UID: "e5d86863-2a31-41e3-9e83-c8308e10384c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:19:39.781340 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.781256    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e5d86863-2a31-41e3-9e83-c8308e10384c" (UID: "e5d86863-2a31-41e3-9e83-c8308e10384c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:19:39.781495 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.781362    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e5d86863-2a31-41e3-9e83-c8308e10384c" (UID: "e5d86863-2a31-41e3-9e83-c8308e10384c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:19:39.783055 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.783024    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5d86863-2a31-41e3-9e83-c8308e10384c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e5d86863-2a31-41e3-9e83-c8308e10384c" (UID: "e5d86863-2a31-41e3-9e83-c8308e10384c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:19:39.783172 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.783034    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d86863-2a31-41e3-9e83-c8308e10384c-kube-api-access-2cq6d" (OuterVolumeSpecName: "kube-api-access-2cq6d") pod "e5d86863-2a31-41e3-9e83-c8308e10384c" (UID: "e5d86863-2a31-41e3-9e83-c8308e10384c"). InnerVolumeSpecName "kube-api-access-2cq6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:19:39.783172 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.783054    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d86863-2a31-41e3-9e83-c8308e10384c-istio-token" (OuterVolumeSpecName: "istio-token") pod "e5d86863-2a31-41e3-9e83-c8308e10384c" (UID: "e5d86863-2a31-41e3-9e83-c8308e10384c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:19:39.783172 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.783141    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e5d86863-2a31-41e3-9e83-c8308e10384c" (UID: "e5d86863-2a31-41e3-9e83-c8308e10384c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:19:39.882375 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.882337    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-credential-socket\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:19:39.882375 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.882368    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5d86863-2a31-41e3-9e83-c8308e10384c-istiod-ca-cert\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:19:39.882375 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.882377    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5d86863-2a31-41e3-9e83-c8308e10384c-istio-token\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:19:39.882375 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.882385    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-workload-socket\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:19:39.882650 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.882395    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-istio-envoy\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:19:39.882650 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.882403    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5d86863-2a31-41e3-9e83-c8308e10384c-workload-certs\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:19:39.882650 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.882412    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cq6d\" (UniqueName: \"kubernetes.io/projected/e5d86863-2a31-41e3-9e83-c8308e10384c-kube-api-access-2cq6d\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:19:39.882650 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:39.882420    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5d86863-2a31-41e3-9e83-c8308e10384c-istio-podinfo\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:19:40.472535 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:40.472493    2578 generic.go:356] "Generic (PLEG): container finished" podID="e5d86863-2a31-41e3-9e83-c8308e10384c" containerID="08f921140180aeb19d1a432f75dea5ab733dd57f21b7cdfa626a63cc2ff77d50" exitCode=0
Sep 11 21:19:40.472750 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:40.472560    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"
Sep 11 21:19:40.472750 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:40.472579    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6" event={"ID":"e5d86863-2a31-41e3-9e83-c8308e10384c","Type":"ContainerDied","Data":"08f921140180aeb19d1a432f75dea5ab733dd57f21b7cdfa626a63cc2ff77d50"}
Sep 11 21:19:40.472750 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:40.472660    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6" event={"ID":"e5d86863-2a31-41e3-9e83-c8308e10384c","Type":"ContainerDied","Data":"1ed8f082eb209920988a43a6b6ab33a72c450814aeb20a0883bd1201bc69ecbe"}
Sep 11 21:19:40.472750 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:40.472683    2578 scope.go:117] "RemoveContainer" containerID="08f921140180aeb19d1a432f75dea5ab733dd57f21b7cdfa626a63cc2ff77d50"
Sep 11 21:19:40.482178 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:40.482156    2578 scope.go:117] "RemoveContainer" containerID="5d8493fb6d501a8608b01808539b3d8c148e8f47f7e1181f2e19be785706aefd"
Sep 11 21:19:40.490290 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:40.490261    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"]
Sep 11 21:19:40.490539 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:40.490524    2578 scope.go:117] "RemoveContainer" containerID="08f921140180aeb19d1a432f75dea5ab733dd57f21b7cdfa626a63cc2ff77d50"
Sep 11 21:19:40.490855 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:19:40.490815    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08f921140180aeb19d1a432f75dea5ab733dd57f21b7cdfa626a63cc2ff77d50\": container with ID starting with 08f921140180aeb19d1a432f75dea5ab733dd57f21b7cdfa626a63cc2ff77d50 not found: ID does not exist" containerID="08f921140180aeb19d1a432f75dea5ab733dd57f21b7cdfa626a63cc2ff77d50"
Sep 11 21:19:40.490912 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:40.490868    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08f921140180aeb19d1a432f75dea5ab733dd57f21b7cdfa626a63cc2ff77d50"} err="failed to get container status \"08f921140180aeb19d1a432f75dea5ab733dd57f21b7cdfa626a63cc2ff77d50\": rpc error: code = NotFound desc = could not find container \"08f921140180aeb19d1a432f75dea5ab733dd57f21b7cdfa626a63cc2ff77d50\": container with ID starting with 08f921140180aeb19d1a432f75dea5ab733dd57f21b7cdfa626a63cc2ff77d50 not found: ID does not exist"
Sep 11 21:19:40.490912 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:40.490894    2578 scope.go:117] "RemoveContainer" containerID="5d8493fb6d501a8608b01808539b3d8c148e8f47f7e1181f2e19be785706aefd"
Sep 11 21:19:40.491181 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:19:40.491163    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d8493fb6d501a8608b01808539b3d8c148e8f47f7e1181f2e19be785706aefd\": container with ID starting with 5d8493fb6d501a8608b01808539b3d8c148e8f47f7e1181f2e19be785706aefd not found: ID does not exist" containerID="5d8493fb6d501a8608b01808539b3d8c148e8f47f7e1181f2e19be785706aefd"
Sep 11 21:19:40.491234 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:40.491191    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d8493fb6d501a8608b01808539b3d8c148e8f47f7e1181f2e19be785706aefd"} err="failed to get container status \"5d8493fb6d501a8608b01808539b3d8c148e8f47f7e1181f2e19be785706aefd\": rpc error: code = NotFound desc = could not find container \"5d8493fb6d501a8608b01808539b3d8c148e8f47f7e1181f2e19be785706aefd\": container with ID starting with 5d8493fb6d501a8608b01808539b3d8c148e8f47f7e1181f2e19be785706aefd not found: ID does not exist"
Sep 11 21:19:40.494236 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:40.494213    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-5fdb611f-openshift-default-554c776b48-7cdz6"]
Sep 11 21:19:41.126467 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:41.126429    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:19:41.927756 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:41.927723    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5d86863-2a31-41e3-9e83-c8308e10384c" path="/var/lib/kubelet/pods/e5d86863-2a31-41e3-9e83-c8308e10384c/volumes"
Sep 11 21:19:57.819960 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:57.819924    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"]
Sep 11 21:19:57.820569 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:57.820443    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5d86863-2a31-41e3-9e83-c8308e10384c" containerName="istio-proxy"
Sep 11 21:19:57.820569 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:57.820464    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5d86863-2a31-41e3-9e83-c8308e10384c" containerName="istio-proxy"
Sep 11 21:19:57.820569 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:57.820481    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5d86863-2a31-41e3-9e83-c8308e10384c" containerName="istio-proxy"
Sep 11 21:19:57.820569 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:57.820490    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5d86863-2a31-41e3-9e83-c8308e10384c" containerName="istio-proxy"
Sep 11 21:19:57.820569 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:57.820517    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9b94901-f911-45bf-bedf-faa66d25e089" containerName="payload-pre-processing"
Sep 11 21:19:57.820569 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:57.820525    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9b94901-f911-45bf-bedf-faa66d25e089" containerName="payload-pre-processing"
Sep 11 21:19:57.820914 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:57.820607    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5d86863-2a31-41e3-9e83-c8308e10384c" containerName="istio-proxy"
Sep 11 21:19:57.820914 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:57.820618    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9b94901-f911-45bf-bedf-faa66d25e089" containerName="payload-pre-processing"
Sep 11 21:19:57.820914 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:57.820627    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5d86863-2a31-41e3-9e83-c8308e10384c" containerName="istio-proxy"
Sep 11 21:19:57.825120 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:57.825097    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:19:57.827561 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:57.827536    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-26462a\"/\"openshift-service-ca.crt\""
Sep 11 21:19:57.827703 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:57.827594    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-26462a\"/\"test-model-26462a-kserve-self-signed-certs\""
Sep 11 21:19:57.828280 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:57.828265    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-26462a\"/\"default-dockercfg-qq7qw\""
Sep 11 21:19:57.828405 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:57.828385    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-26462a\"/\"kube-root-ca.crt\""
Sep 11 21:19:57.834165 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:57.834141    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"]
Sep 11 21:19:57.946593 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:57.946556    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5699b08a-6f3a-48b3-9f91-3d9cc291a336-tmp-dir\") pod \"test-model-26462a-kserve-68d8f6d567-w87ds\" (UID: \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\") " pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:19:57.946777 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:57.946606    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5699b08a-6f3a-48b3-9f91-3d9cc291a336-home\") pod \"test-model-26462a-kserve-68d8f6d567-w87ds\" (UID: \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\") " pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:19:57.946777 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:57.946682    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5699b08a-6f3a-48b3-9f91-3d9cc291a336-model-cache\") pod \"test-model-26462a-kserve-68d8f6d567-w87ds\" (UID: \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\") " pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:19:57.946777 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:57.946718    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5699b08a-6f3a-48b3-9f91-3d9cc291a336-dshm\") pod \"test-model-26462a-kserve-68d8f6d567-w87ds\" (UID: \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\") " pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:19:57.946777 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:57.946766    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlmxg\" (UniqueName: \"kubernetes.io/projected/5699b08a-6f3a-48b3-9f91-3d9cc291a336-kube-api-access-zlmxg\") pod \"test-model-26462a-kserve-68d8f6d567-w87ds\" (UID: \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\") " pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:19:57.946970 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:57.946796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5699b08a-6f3a-48b3-9f91-3d9cc291a336-tls-certs\") pod \"test-model-26462a-kserve-68d8f6d567-w87ds\" (UID: \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\") " pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:19:58.048161 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:58.048114    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlmxg\" (UniqueName: \"kubernetes.io/projected/5699b08a-6f3a-48b3-9f91-3d9cc291a336-kube-api-access-zlmxg\") pod \"test-model-26462a-kserve-68d8f6d567-w87ds\" (UID: \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\") " pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:19:58.048161 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:58.048162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5699b08a-6f3a-48b3-9f91-3d9cc291a336-tls-certs\") pod \"test-model-26462a-kserve-68d8f6d567-w87ds\" (UID: \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\") " pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:19:58.048407 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:58.048186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5699b08a-6f3a-48b3-9f91-3d9cc291a336-tmp-dir\") pod \"test-model-26462a-kserve-68d8f6d567-w87ds\" (UID: \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\") " pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:19:58.048407 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:58.048212    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5699b08a-6f3a-48b3-9f91-3d9cc291a336-home\") pod \"test-model-26462a-kserve-68d8f6d567-w87ds\" (UID: \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\") " pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:19:58.048407 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:58.048237    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5699b08a-6f3a-48b3-9f91-3d9cc291a336-model-cache\") pod \"test-model-26462a-kserve-68d8f6d567-w87ds\" (UID: \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\") " pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:19:58.048407 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:58.048255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5699b08a-6f3a-48b3-9f91-3d9cc291a336-dshm\") pod \"test-model-26462a-kserve-68d8f6d567-w87ds\" (UID: \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\") " pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:19:58.048726 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:58.048699    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5699b08a-6f3a-48b3-9f91-3d9cc291a336-tmp-dir\") pod \"test-model-26462a-kserve-68d8f6d567-w87ds\" (UID: \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\") " pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:19:58.048726 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:58.048718    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5699b08a-6f3a-48b3-9f91-3d9cc291a336-home\") pod \"test-model-26462a-kserve-68d8f6d567-w87ds\" (UID: \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\") " pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:19:58.048950 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:58.048786    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5699b08a-6f3a-48b3-9f91-3d9cc291a336-model-cache\") pod \"test-model-26462a-kserve-68d8f6d567-w87ds\" (UID: \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\") " pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:19:58.050545 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:58.050527    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5699b08a-6f3a-48b3-9f91-3d9cc291a336-dshm\") pod \"test-model-26462a-kserve-68d8f6d567-w87ds\" (UID: \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\") " pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:19:58.050843 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:58.050806    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5699b08a-6f3a-48b3-9f91-3d9cc291a336-tls-certs\") pod \"test-model-26462a-kserve-68d8f6d567-w87ds\" (UID: \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\") " pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:19:58.057283 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:58.057260    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlmxg\" (UniqueName: \"kubernetes.io/projected/5699b08a-6f3a-48b3-9f91-3d9cc291a336-kube-api-access-zlmxg\") pod \"test-model-26462a-kserve-68d8f6d567-w87ds\" (UID: \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\") " pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:19:58.135225 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:58.135129    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:19:58.409514 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:58.409421    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"]
Sep 11 21:19:58.412471 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:19:58.412448    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5699b08a_6f3a_48b3_9f91_3d9cc291a336.slice/crio-d51b122d99f07752a815fc26541319b5f234645fcdecce721be39ea8d26d2300 WatchSource:0}: Error finding container d51b122d99f07752a815fc26541319b5f234645fcdecce721be39ea8d26d2300: Status 404 returned error can't find the container with id d51b122d99f07752a815fc26541319b5f234645fcdecce721be39ea8d26d2300
Sep 11 21:19:58.563019 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:58.562984    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds" event={"ID":"5699b08a-6f3a-48b3-9f91-3d9cc291a336","Type":"ContainerStarted","Data":"0f3e3acf7256d3aceaf19e7b009cbd86ad9c115ae32e926910055981d75e8191"}
Sep 11 21:19:58.563189 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:58.563029    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds" event={"ID":"5699b08a-6f3a-48b3-9f91-3d9cc291a336","Type":"ContainerStarted","Data":"d51b122d99f07752a815fc26541319b5f234645fcdecce721be39ea8d26d2300"}
Sep 11 21:19:58.582099 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:19:58.582046    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds" podStartSLOduration=1.5820305609999998 podStartE2EDuration="1.582030561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:19:58.580185782 +0000 UTC m=+865.284155075" watchObservedRunningTime="2026-09-11 21:19:58.582030561 +0000 UTC m=+865.285999850"
Sep 11 21:20:05.727899 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:05.727863    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:20:08.136140 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:08.136093    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:20:08.136575 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:08.136241    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:20:08.148934 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:08.148905    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:20:08.620099 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:08.620069    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:20:10.668087 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:10.668053    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-765a18-8576855845-4qwxt"]
Sep 11 21:20:10.668519 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:10.668277    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-765a18-8576855845-4qwxt" podUID="7388aa75-9611-4fab-a5b0-3b914a083584" containerName="payload-processing" containerID="cri-o://706cd964c45fcf393d7a07e0817d2d5673da83b0ece5f40386c98ea72d4253ee" gracePeriod=30
Sep 11 21:20:10.920776 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:10.920712    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-765a18-8576855845-4qwxt"
Sep 11 21:20:11.073212 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:11.073166    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4hjv\" (UniqueName: \"kubernetes.io/projected/7388aa75-9611-4fab-a5b0-3b914a083584-kube-api-access-l4hjv\") pod \"7388aa75-9611-4fab-a5b0-3b914a083584\" (UID: \"7388aa75-9611-4fab-a5b0-3b914a083584\") "
Sep 11 21:20:11.073212 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:11.073216    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7388aa75-9611-4fab-a5b0-3b914a083584-plugins-config-volume\") pod \"7388aa75-9611-4fab-a5b0-3b914a083584\" (UID: \"7388aa75-9611-4fab-a5b0-3b914a083584\") "
Sep 11 21:20:11.073617 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:11.073590    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7388aa75-9611-4fab-a5b0-3b914a083584-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7388aa75-9611-4fab-a5b0-3b914a083584" (UID: "7388aa75-9611-4fab-a5b0-3b914a083584"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:20:11.075378 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:11.075351    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7388aa75-9611-4fab-a5b0-3b914a083584-kube-api-access-l4hjv" (OuterVolumeSpecName: "kube-api-access-l4hjv") pod "7388aa75-9611-4fab-a5b0-3b914a083584" (UID: "7388aa75-9611-4fab-a5b0-3b914a083584"). InnerVolumeSpecName "kube-api-access-l4hjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:20:11.174235 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:11.174203    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4hjv\" (UniqueName: \"kubernetes.io/projected/7388aa75-9611-4fab-a5b0-3b914a083584-kube-api-access-l4hjv\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:11.174235 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:11.174234    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7388aa75-9611-4fab-a5b0-3b914a083584-plugins-config-volume\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:11.624376 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:11.624291    2578 generic.go:356] "Generic (PLEG): container finished" podID="7388aa75-9611-4fab-a5b0-3b914a083584" containerID="706cd964c45fcf393d7a07e0817d2d5673da83b0ece5f40386c98ea72d4253ee" exitCode=0
Sep 11 21:20:11.624376 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:11.624335    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-765a18-8576855845-4qwxt"
Sep 11 21:20:11.624589 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:11.624377    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-765a18-8576855845-4qwxt" event={"ID":"7388aa75-9611-4fab-a5b0-3b914a083584","Type":"ContainerDied","Data":"706cd964c45fcf393d7a07e0817d2d5673da83b0ece5f40386c98ea72d4253ee"}
Sep 11 21:20:11.624589 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:11.624424    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-765a18-8576855845-4qwxt" event={"ID":"7388aa75-9611-4fab-a5b0-3b914a083584","Type":"ContainerDied","Data":"52365cc7e25cb58f4be243b94bf68b0f1df62568baefd373c3df7c5b2f83bde3"}
Sep 11 21:20:11.624589 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:11.624445    2578 scope.go:117] "RemoveContainer" containerID="706cd964c45fcf393d7a07e0817d2d5673da83b0ece5f40386c98ea72d4253ee"
Sep 11 21:20:11.633922 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:11.633903    2578 scope.go:117] "RemoveContainer" containerID="706cd964c45fcf393d7a07e0817d2d5673da83b0ece5f40386c98ea72d4253ee"
Sep 11 21:20:11.634196 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:20:11.634176    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"706cd964c45fcf393d7a07e0817d2d5673da83b0ece5f40386c98ea72d4253ee\": container with ID starting with 706cd964c45fcf393d7a07e0817d2d5673da83b0ece5f40386c98ea72d4253ee not found: ID does not exist" containerID="706cd964c45fcf393d7a07e0817d2d5673da83b0ece5f40386c98ea72d4253ee"
Sep 11 21:20:11.634259 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:11.634206    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"706cd964c45fcf393d7a07e0817d2d5673da83b0ece5f40386c98ea72d4253ee"} err="failed to get container status \"706cd964c45fcf393d7a07e0817d2d5673da83b0ece5f40386c98ea72d4253ee\": rpc error: code = NotFound desc = could not find container \"706cd964c45fcf393d7a07e0817d2d5673da83b0ece5f40386c98ea72d4253ee\": container with ID starting with 706cd964c45fcf393d7a07e0817d2d5673da83b0ece5f40386c98ea72d4253ee not found: ID does not exist"
Sep 11 21:20:11.645611 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:11.645582    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-765a18-8576855845-4qwxt"]
Sep 11 21:20:11.648866 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:11.648811    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-765a18-8576855845-4qwxt"]
Sep 11 21:20:11.927582 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:11.927549    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7388aa75-9611-4fab-a5b0-3b914a083584" path="/var/lib/kubelet/pods/7388aa75-9611-4fab-a5b0-3b914a083584/volumes"
Sep 11 21:20:16.163955 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:16.163919    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"]
Sep 11 21:20:17.805340 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:17.805303    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ea5e33-pfs4n"]
Sep 11 21:20:17.805728 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:17.805633    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7388aa75-9611-4fab-a5b0-3b914a083584" containerName="payload-processing"
Sep 11 21:20:17.805728 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:17.805645    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7388aa75-9611-4fab-a5b0-3b914a083584" containerName="payload-processing"
Sep 11 21:20:17.805728 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:17.805707    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7388aa75-9611-4fab-a5b0-3b914a083584" containerName="payload-processing"
Sep 11 21:20:17.809974 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:17.809950    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ea5e33-pfs4n"
Sep 11 21:20:17.812538 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:17.812515    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:20:17.818248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:17.813007    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfm9g\""
Sep 11 21:20:17.821397 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:17.821370    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ea5e33-pfs4n"]
Sep 11 21:20:17.929535 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:17.929496    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5d61a25-b33a-46bf-ad2b-6bdb20edbce9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ea5e33-pfs4n\" (UID: \"d5d61a25-b33a-46bf-ad2b-6bdb20edbce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ea5e33-pfs4n"
Sep 11 21:20:18.030937 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:18.030900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5d61a25-b33a-46bf-ad2b-6bdb20edbce9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ea5e33-pfs4n\" (UID: \"d5d61a25-b33a-46bf-ad2b-6bdb20edbce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ea5e33-pfs4n"
Sep 11 21:20:18.031765 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:18.031734    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5d61a25-b33a-46bf-ad2b-6bdb20edbce9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ea5e33-pfs4n\" (UID: \"d5d61a25-b33a-46bf-ad2b-6bdb20edbce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ea5e33-pfs4n"
Sep 11 21:20:18.127990 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:18.127899    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ea5e33-pfs4n"
Sep 11 21:20:18.403694 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:18.403669    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ea5e33-pfs4n"]
Sep 11 21:20:18.406151 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:20:18.406125    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d61a25_b33a_46bf_ad2b_6bdb20edbce9.slice/crio-57c9340e965cbb4b65b94e6345d77db5f31a9c1b0ccc1ad32580d3c20e144416 WatchSource:0}: Error finding container 57c9340e965cbb4b65b94e6345d77db5f31a9c1b0ccc1ad32580d3c20e144416: Status 404 returned error can't find the container with id 57c9340e965cbb4b65b94e6345d77db5f31a9c1b0ccc1ad32580d3c20e144416
Sep 11 21:20:18.657275 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:18.657193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ea5e33-pfs4n" event={"ID":"d5d61a25-b33a-46bf-ad2b-6bdb20edbce9","Type":"ContainerStarted","Data":"57c9340e965cbb4b65b94e6345d77db5f31a9c1b0ccc1ad32580d3c20e144416"}
Sep 11 21:20:19.663444 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:19.663405    2578 generic.go:356] "Generic (PLEG): container finished" podID="d5d61a25-b33a-46bf-ad2b-6bdb20edbce9" containerID="bc0138289003f6373df76c3d533377cf101296b7e41134aa1e1d513bd1591ce1" exitCode=0
Sep 11 21:20:19.663850 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:19.663490    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ea5e33-pfs4n" event={"ID":"d5d61a25-b33a-46bf-ad2b-6bdb20edbce9","Type":"ContainerDied","Data":"bc0138289003f6373df76c3d533377cf101296b7e41134aa1e1d513bd1591ce1"}
Sep 11 21:20:20.803244 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:20.803209    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ea5e33-pfs4n"
Sep 11 21:20:20.957724 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:20.957688    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5d61a25-b33a-46bf-ad2b-6bdb20edbce9-maas-api-service-ca\") pod \"d5d61a25-b33a-46bf-ad2b-6bdb20edbce9\" (UID: \"d5d61a25-b33a-46bf-ad2b-6bdb20edbce9\") "
Sep 11 21:20:20.958079 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:20.958056    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d61a25-b33a-46bf-ad2b-6bdb20edbce9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d5d61a25-b33a-46bf-ad2b-6bdb20edbce9" (UID: "d5d61a25-b33a-46bf-ad2b-6bdb20edbce9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:20:21.058749 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:21.058707    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5d61a25-b33a-46bf-ad2b-6bdb20edbce9-maas-api-service-ca\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:21.673553 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:21.673530    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ea5e33-pfs4n"
Sep 11 21:20:21.673729 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:21.673526    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ea5e33-pfs4n" event={"ID":"d5d61a25-b33a-46bf-ad2b-6bdb20edbce9","Type":"ContainerDied","Data":"57c9340e965cbb4b65b94e6345d77db5f31a9c1b0ccc1ad32580d3c20e144416"}
Sep 11 21:20:21.673729 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:21.673647    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57c9340e965cbb4b65b94e6345d77db5f31a9c1b0ccc1ad32580d3c20e144416"
Sep 11 21:20:22.666588 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:22.666553    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"]
Sep 11 21:20:23.015859 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:23.015814    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"]
Sep 11 21:20:24.223072 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:24.223038    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:20:28.610744 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:28.610640    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx"]
Sep 11 21:20:28.611162 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:28.610916    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx" podUID="fb307345-d97a-470b-8d73-1406c761d5c4" containerName="payload-processing" containerID="cri-o://cecedc96b71117cd24eb623b236632a19d0dc6927118014a0d4cf31ac1214689" gracePeriod=30
Sep 11 21:20:28.615737 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:28.615708    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn"]
Sep 11 21:20:28.616140 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:28.616107    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn" podUID="f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9" containerName="payload-pre-processing" containerID="cri-o://22ccde18d7b5851c7bc9e7e996fb835290567331228bafa5a317fd73aa17572a" gracePeriod=30
Sep 11 21:20:28.882665 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:28.882636    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx"
Sep 11 21:20:28.886055 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:28.886028    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn"
Sep 11 21:20:28.922887 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:28.922861    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkkft\" (UniqueName: \"kubernetes.io/projected/fb307345-d97a-470b-8d73-1406c761d5c4-kube-api-access-dkkft\") pod \"fb307345-d97a-470b-8d73-1406c761d5c4\" (UID: \"fb307345-d97a-470b-8d73-1406c761d5c4\") "
Sep 11 21:20:28.922887 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:28.922894    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcn8x\" (UniqueName: \"kubernetes.io/projected/f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9-kube-api-access-qcn8x\") pod \"f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9\" (UID: \"f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9\") "
Sep 11 21:20:28.923498 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:28.922914    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb307345-d97a-470b-8d73-1406c761d5c4-plugins-config-volume\") pod \"fb307345-d97a-470b-8d73-1406c761d5c4\" (UID: \"fb307345-d97a-470b-8d73-1406c761d5c4\") "
Sep 11 21:20:28.923498 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:28.923000    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9-plugins-config-volume\") pod \"f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9\" (UID: \"f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9\") "
Sep 11 21:20:28.923498 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:28.923356    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb307345-d97a-470b-8d73-1406c761d5c4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fb307345-d97a-470b-8d73-1406c761d5c4" (UID: "fb307345-d97a-470b-8d73-1406c761d5c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:20:28.923498 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:28.923454    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9" (UID: "f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:20:28.925213 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:28.925174    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb307345-d97a-470b-8d73-1406c761d5c4-kube-api-access-dkkft" (OuterVolumeSpecName: "kube-api-access-dkkft") pod "fb307345-d97a-470b-8d73-1406c761d5c4" (UID: "fb307345-d97a-470b-8d73-1406c761d5c4"). InnerVolumeSpecName "kube-api-access-dkkft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:20:28.925342 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:28.925230    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9-kube-api-access-qcn8x" (OuterVolumeSpecName: "kube-api-access-qcn8x") pod "f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9" (UID: "f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9"). InnerVolumeSpecName "kube-api-access-qcn8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:20:29.024584 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.024547    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9-plugins-config-volume\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:29.024584 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.024578    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkkft\" (UniqueName: \"kubernetes.io/projected/fb307345-d97a-470b-8d73-1406c761d5c4-kube-api-access-dkkft\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:29.024584 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.024588    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcn8x\" (UniqueName: \"kubernetes.io/projected/f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9-kube-api-access-qcn8x\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:29.024942 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.024598    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb307345-d97a-470b-8d73-1406c761d5c4-plugins-config-volume\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:29.212488 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.212425    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ea5e33-pfs4n"]
Sep 11 21:20:29.215174 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.215149    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ea5e33-pfs4n"]
Sep 11 21:20:29.711557 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.711523    2578 generic.go:356] "Generic (PLEG): container finished" podID="f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9" containerID="22ccde18d7b5851c7bc9e7e996fb835290567331228bafa5a317fd73aa17572a" exitCode=0
Sep 11 21:20:29.712057 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.711576    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn"
Sep 11 21:20:29.712057 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.711606    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn" event={"ID":"f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9","Type":"ContainerDied","Data":"22ccde18d7b5851c7bc9e7e996fb835290567331228bafa5a317fd73aa17572a"}
Sep 11 21:20:29.712057 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.711657    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn" event={"ID":"f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9","Type":"ContainerDied","Data":"fc5046f323855f508c305cbca6943d35916e4f4e32d20459d036fbcba66609fa"}
Sep 11 21:20:29.712057 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.711678    2578 scope.go:117] "RemoveContainer" containerID="22ccde18d7b5851c7bc9e7e996fb835290567331228bafa5a317fd73aa17572a"
Sep 11 21:20:29.713079 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.713060    2578 generic.go:356] "Generic (PLEG): container finished" podID="fb307345-d97a-470b-8d73-1406c761d5c4" containerID="cecedc96b71117cd24eb623b236632a19d0dc6927118014a0d4cf31ac1214689" exitCode=0
Sep 11 21:20:29.713180 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.713091    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx" event={"ID":"fb307345-d97a-470b-8d73-1406c761d5c4","Type":"ContainerDied","Data":"cecedc96b71117cd24eb623b236632a19d0dc6927118014a0d4cf31ac1214689"}
Sep 11 21:20:29.713180 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.713119    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx"
Sep 11 21:20:29.713180 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.713145    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx" event={"ID":"fb307345-d97a-470b-8d73-1406c761d5c4","Type":"ContainerDied","Data":"0959f16e7ac31341a336775c907a5360eff1729f875d4bb0faf7c9535761aa14"}
Sep 11 21:20:29.720907 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.720886    2578 scope.go:117] "RemoveContainer" containerID="22ccde18d7b5851c7bc9e7e996fb835290567331228bafa5a317fd73aa17572a"
Sep 11 21:20:29.721179 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:20:29.721158    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22ccde18d7b5851c7bc9e7e996fb835290567331228bafa5a317fd73aa17572a\": container with ID starting with 22ccde18d7b5851c7bc9e7e996fb835290567331228bafa5a317fd73aa17572a not found: ID does not exist" containerID="22ccde18d7b5851c7bc9e7e996fb835290567331228bafa5a317fd73aa17572a"
Sep 11 21:20:29.721267 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.721189    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ccde18d7b5851c7bc9e7e996fb835290567331228bafa5a317fd73aa17572a"} err="failed to get container status \"22ccde18d7b5851c7bc9e7e996fb835290567331228bafa5a317fd73aa17572a\": rpc error: code = NotFound desc = could not find container \"22ccde18d7b5851c7bc9e7e996fb835290567331228bafa5a317fd73aa17572a\": container with ID starting with 22ccde18d7b5851c7bc9e7e996fb835290567331228bafa5a317fd73aa17572a not found: ID does not exist"
Sep 11 21:20:29.721267 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.721212    2578 scope.go:117] "RemoveContainer" containerID="cecedc96b71117cd24eb623b236632a19d0dc6927118014a0d4cf31ac1214689"
Sep 11 21:20:29.729761 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.729744    2578 scope.go:117] "RemoveContainer" containerID="cecedc96b71117cd24eb623b236632a19d0dc6927118014a0d4cf31ac1214689"
Sep 11 21:20:29.730055 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:20:29.730036    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cecedc96b71117cd24eb623b236632a19d0dc6927118014a0d4cf31ac1214689\": container with ID starting with cecedc96b71117cd24eb623b236632a19d0dc6927118014a0d4cf31ac1214689 not found: ID does not exist" containerID="cecedc96b71117cd24eb623b236632a19d0dc6927118014a0d4cf31ac1214689"
Sep 11 21:20:29.730150 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.730061    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cecedc96b71117cd24eb623b236632a19d0dc6927118014a0d4cf31ac1214689"} err="failed to get container status \"cecedc96b71117cd24eb623b236632a19d0dc6927118014a0d4cf31ac1214689\": rpc error: code = NotFound desc = could not find container \"cecedc96b71117cd24eb623b236632a19d0dc6927118014a0d4cf31ac1214689\": container with ID starting with cecedc96b71117cd24eb623b236632a19d0dc6927118014a0d4cf31ac1214689 not found: ID does not exist"
Sep 11 21:20:29.736537 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.736516    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn"]
Sep 11 21:20:29.738718 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.738699    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ea5e33-5f8584d474-hppxn"]
Sep 11 21:20:29.748244 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.748207    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx"]
Sep 11 21:20:29.755918 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.755895    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ea5e33-7ff4d49959-jfcpx"]
Sep 11 21:20:29.927714 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.927684    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5d61a25-b33a-46bf-ad2b-6bdb20edbce9" path="/var/lib/kubelet/pods/d5d61a25-b33a-46bf-ad2b-6bdb20edbce9/volumes"
Sep 11 21:20:29.928079 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.928065    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9" path="/var/lib/kubelet/pods/f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9/volumes"
Sep 11 21:20:29.928400 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:29.928386    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb307345-d97a-470b-8d73-1406c761d5c4" path="/var/lib/kubelet/pods/fb307345-d97a-470b-8d73-1406c761d5c4/volumes"
Sep 11 21:20:31.182885 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.182847    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4" podUID="e200df09-7c52-432b-a646-5c6e28d28b0a" containerName="main" containerID="cri-o://e1d9b5dcd5493f8e43c4d26f9edc84ef69bb4a3a0c8f1b631bcce6f00f0e257b" gracePeriod=45
Sep 11 21:20:31.427692 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.427662    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:20:31.446248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.446171    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e200df09-7c52-432b-a646-5c6e28d28b0a-tmp-dir\") pod \"e200df09-7c52-432b-a646-5c6e28d28b0a\" (UID: \"e200df09-7c52-432b-a646-5c6e28d28b0a\") "
Sep 11 21:20:31.446248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.446224    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e200df09-7c52-432b-a646-5c6e28d28b0a-dshm\") pod \"e200df09-7c52-432b-a646-5c6e28d28b0a\" (UID: \"e200df09-7c52-432b-a646-5c6e28d28b0a\") "
Sep 11 21:20:31.446460 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.446259    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e200df09-7c52-432b-a646-5c6e28d28b0a-tls-certs\") pod \"e200df09-7c52-432b-a646-5c6e28d28b0a\" (UID: \"e200df09-7c52-432b-a646-5c6e28d28b0a\") "
Sep 11 21:20:31.446460 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.446325    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e200df09-7c52-432b-a646-5c6e28d28b0a-home\") pod \"e200df09-7c52-432b-a646-5c6e28d28b0a\" (UID: \"e200df09-7c52-432b-a646-5c6e28d28b0a\") "
Sep 11 21:20:31.446460 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.446349    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsktl\" (UniqueName: \"kubernetes.io/projected/e200df09-7c52-432b-a646-5c6e28d28b0a-kube-api-access-xsktl\") pod \"e200df09-7c52-432b-a646-5c6e28d28b0a\" (UID: \"e200df09-7c52-432b-a646-5c6e28d28b0a\") "
Sep 11 21:20:31.446460 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.446380    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e200df09-7c52-432b-a646-5c6e28d28b0a-model-cache\") pod \"e200df09-7c52-432b-a646-5c6e28d28b0a\" (UID: \"e200df09-7c52-432b-a646-5c6e28d28b0a\") "
Sep 11 21:20:31.446626 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.446475    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e200df09-7c52-432b-a646-5c6e28d28b0a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e200df09-7c52-432b-a646-5c6e28d28b0a" (UID: "e200df09-7c52-432b-a646-5c6e28d28b0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:20:31.446670 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.446647    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e200df09-7c52-432b-a646-5c6e28d28b0a-home" (OuterVolumeSpecName: "home") pod "e200df09-7c52-432b-a646-5c6e28d28b0a" (UID: "e200df09-7c52-432b-a646-5c6e28d28b0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:20:31.446712 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.446674    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e200df09-7c52-432b-a646-5c6e28d28b0a-model-cache" (OuterVolumeSpecName: "model-cache") pod "e200df09-7c52-432b-a646-5c6e28d28b0a" (UID: "e200df09-7c52-432b-a646-5c6e28d28b0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:20:31.446758 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.446704    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e200df09-7c52-432b-a646-5c6e28d28b0a-tmp-dir\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:31.448714 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.448684    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e200df09-7c52-432b-a646-5c6e28d28b0a-dshm" (OuterVolumeSpecName: "dshm") pod "e200df09-7c52-432b-a646-5c6e28d28b0a" (UID: "e200df09-7c52-432b-a646-5c6e28d28b0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:20:31.448871 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.448747    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e200df09-7c52-432b-a646-5c6e28d28b0a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e200df09-7c52-432b-a646-5c6e28d28b0a" (UID: "e200df09-7c52-432b-a646-5c6e28d28b0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:20:31.448871 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.448777    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e200df09-7c52-432b-a646-5c6e28d28b0a-kube-api-access-xsktl" (OuterVolumeSpecName: "kube-api-access-xsktl") pod "e200df09-7c52-432b-a646-5c6e28d28b0a" (UID: "e200df09-7c52-432b-a646-5c6e28d28b0a"). InnerVolumeSpecName "kube-api-access-xsktl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:20:31.547610 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.547566    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e200df09-7c52-432b-a646-5c6e28d28b0a-dshm\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:31.547610 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.547602    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e200df09-7c52-432b-a646-5c6e28d28b0a-tls-certs\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:31.547610 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.547615    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e200df09-7c52-432b-a646-5c6e28d28b0a-home\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:31.547926 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.547626    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsktl\" (UniqueName: \"kubernetes.io/projected/e200df09-7c52-432b-a646-5c6e28d28b0a-kube-api-access-xsktl\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:31.547926 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.547638    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e200df09-7c52-432b-a646-5c6e28d28b0a-model-cache\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:31.725263 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.725172    2578 generic.go:356] "Generic (PLEG): container finished" podID="e200df09-7c52-432b-a646-5c6e28d28b0a" containerID="e1d9b5dcd5493f8e43c4d26f9edc84ef69bb4a3a0c8f1b631bcce6f00f0e257b" exitCode=0
Sep 11 21:20:31.725263 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.725217    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"
Sep 11 21:20:31.725473 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.725258    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4" event={"ID":"e200df09-7c52-432b-a646-5c6e28d28b0a","Type":"ContainerDied","Data":"e1d9b5dcd5493f8e43c4d26f9edc84ef69bb4a3a0c8f1b631bcce6f00f0e257b"}
Sep 11 21:20:31.725473 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.725301    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4" event={"ID":"e200df09-7c52-432b-a646-5c6e28d28b0a","Type":"ContainerDied","Data":"8e2face21635a3f806666bf2337667ec2dec5b7f82160c1bb0ce6d9b1f3eead5"}
Sep 11 21:20:31.725473 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.725320    2578 scope.go:117] "RemoveContainer" containerID="e1d9b5dcd5493f8e43c4d26f9edc84ef69bb4a3a0c8f1b631bcce6f00f0e257b"
Sep 11 21:20:31.734344 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.734327    2578 scope.go:117] "RemoveContainer" containerID="e1d9b5dcd5493f8e43c4d26f9edc84ef69bb4a3a0c8f1b631bcce6f00f0e257b"
Sep 11 21:20:31.734653 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:20:31.734635    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1d9b5dcd5493f8e43c4d26f9edc84ef69bb4a3a0c8f1b631bcce6f00f0e257b\": container with ID starting with e1d9b5dcd5493f8e43c4d26f9edc84ef69bb4a3a0c8f1b631bcce6f00f0e257b not found: ID does not exist" containerID="e1d9b5dcd5493f8e43c4d26f9edc84ef69bb4a3a0c8f1b631bcce6f00f0e257b"
Sep 11 21:20:31.734709 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.734661    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1d9b5dcd5493f8e43c4d26f9edc84ef69bb4a3a0c8f1b631bcce6f00f0e257b"} err="failed to get container status \"e1d9b5dcd5493f8e43c4d26f9edc84ef69bb4a3a0c8f1b631bcce6f00f0e257b\": rpc error: code = NotFound desc = could not find container \"e1d9b5dcd5493f8e43c4d26f9edc84ef69bb4a3a0c8f1b631bcce6f00f0e257b\": container with ID starting with e1d9b5dcd5493f8e43c4d26f9edc84ef69bb4a3a0c8f1b631bcce6f00f0e257b not found: ID does not exist"
Sep 11 21:20:31.745553 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.745526    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"]
Sep 11 21:20:31.751696 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.751673    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-765a18/autores-model-765a18-kserve-8d7c47475-k4jw4"]
Sep 11 21:20:31.927172 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:31.927141    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e200df09-7c52-432b-a646-5c6e28d28b0a" path="/var/lib/kubelet/pods/e200df09-7c52-432b-a646-5c6e28d28b0a/volumes"
Sep 11 21:20:37.685414 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:37.685351    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7" podUID="7e6496ab-6ff6-476a-9a3b-f3b316c7099e" containerName="main" containerID="cri-o://84c99a94b449e7432646b2b959110905cbcfe4f58d68bd0b8cb8b16b8f19b034" gracePeriod=45
Sep 11 21:20:37.933018 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:37.932991    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:20:37.999976 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:37.999886    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-dshm\") pod \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\" (UID: \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\") "
Sep 11 21:20:37.999976 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:37.999926    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-tls-certs\") pod \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\" (UID: \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\") "
Sep 11 21:20:37.999976 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:37.999970    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-tmp-dir\") pod \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\" (UID: \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\") "
Sep 11 21:20:38.000252 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:37.999986    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-home\") pod \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\" (UID: \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\") "
Sep 11 21:20:38.000252 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.000005    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qclcl\" (UniqueName: \"kubernetes.io/projected/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-kube-api-access-qclcl\") pod \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\" (UID: \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\") "
Sep 11 21:20:38.000252 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.000052    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-model-cache\") pod \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\" (UID: \"7e6496ab-6ff6-476a-9a3b-f3b316c7099e\") "
Sep 11 21:20:38.000415 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.000288    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-home" (OuterVolumeSpecName: "home") pod "7e6496ab-6ff6-476a-9a3b-f3b316c7099e" (UID: "7e6496ab-6ff6-476a-9a3b-f3b316c7099e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:20:38.000415 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.000299    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e6496ab-6ff6-476a-9a3b-f3b316c7099e" (UID: "7e6496ab-6ff6-476a-9a3b-f3b316c7099e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:20:38.000415 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.000393    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e6496ab-6ff6-476a-9a3b-f3b316c7099e" (UID: "7e6496ab-6ff6-476a-9a3b-f3b316c7099e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:20:38.002435 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.002409    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-dshm" (OuterVolumeSpecName: "dshm") pod "7e6496ab-6ff6-476a-9a3b-f3b316c7099e" (UID: "7e6496ab-6ff6-476a-9a3b-f3b316c7099e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:20:38.002435 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.002423    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-kube-api-access-qclcl" (OuterVolumeSpecName: "kube-api-access-qclcl") pod "7e6496ab-6ff6-476a-9a3b-f3b316c7099e" (UID: "7e6496ab-6ff6-476a-9a3b-f3b316c7099e"). InnerVolumeSpecName "kube-api-access-qclcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:20:38.002620 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.002459    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e6496ab-6ff6-476a-9a3b-f3b316c7099e" (UID: "7e6496ab-6ff6-476a-9a3b-f3b316c7099e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:20:38.034231 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.034170    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km" podUID="7d2aa425-fe38-4068-8db3-4813b822297e" containerName="main" containerID="cri-o://c52783fc2dbfe7eb9ec35d3084239e316ed0633cc1f1461f2d9f40a21b34cdb1" gracePeriod=45
Sep 11 21:20:38.101012 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.100980    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-model-cache\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:38.101012 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.101008    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-dshm\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:38.101012 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.101017    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-tls-certs\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:38.101230 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.101026    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-tmp-dir\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:38.101230 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.101033    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-home\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:38.101230 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.101041    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qclcl\" (UniqueName: \"kubernetes.io/projected/7e6496ab-6ff6-476a-9a3b-f3b316c7099e-kube-api-access-qclcl\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:38.272937 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.272910    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:20:38.303009 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.302969    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d2aa425-fe38-4068-8db3-4813b822297e-home\") pod \"7d2aa425-fe38-4068-8db3-4813b822297e\" (UID: \"7d2aa425-fe38-4068-8db3-4813b822297e\") "
Sep 11 21:20:38.303195 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.303040    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d2aa425-fe38-4068-8db3-4813b822297e-tls-certs\") pod \"7d2aa425-fe38-4068-8db3-4813b822297e\" (UID: \"7d2aa425-fe38-4068-8db3-4813b822297e\") "
Sep 11 21:20:38.303195 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.303073    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d2aa425-fe38-4068-8db3-4813b822297e-tmp-dir\") pod \"7d2aa425-fe38-4068-8db3-4813b822297e\" (UID: \"7d2aa425-fe38-4068-8db3-4813b822297e\") "
Sep 11 21:20:38.303195 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.303111    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfn9c\" (UniqueName: \"kubernetes.io/projected/7d2aa425-fe38-4068-8db3-4813b822297e-kube-api-access-nfn9c\") pod \"7d2aa425-fe38-4068-8db3-4813b822297e\" (UID: \"7d2aa425-fe38-4068-8db3-4813b822297e\") "
Sep 11 21:20:38.303195 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.303179    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d2aa425-fe38-4068-8db3-4813b822297e-dshm\") pod \"7d2aa425-fe38-4068-8db3-4813b822297e\" (UID: \"7d2aa425-fe38-4068-8db3-4813b822297e\") "
Sep 11 21:20:38.303437 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.303229    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d2aa425-fe38-4068-8db3-4813b822297e-model-cache\") pod \"7d2aa425-fe38-4068-8db3-4813b822297e\" (UID: \"7d2aa425-fe38-4068-8db3-4813b822297e\") "
Sep 11 21:20:38.303437 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.303274    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2aa425-fe38-4068-8db3-4813b822297e-home" (OuterVolumeSpecName: "home") pod "7d2aa425-fe38-4068-8db3-4813b822297e" (UID: "7d2aa425-fe38-4068-8db3-4813b822297e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:20:38.303585 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.303520    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d2aa425-fe38-4068-8db3-4813b822297e-home\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:38.303715 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.303604    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2aa425-fe38-4068-8db3-4813b822297e-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d2aa425-fe38-4068-8db3-4813b822297e" (UID: "7d2aa425-fe38-4068-8db3-4813b822297e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:20:38.303885 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.303853    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2aa425-fe38-4068-8db3-4813b822297e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d2aa425-fe38-4068-8db3-4813b822297e" (UID: "7d2aa425-fe38-4068-8db3-4813b822297e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:20:38.306375 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.306343    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d2aa425-fe38-4068-8db3-4813b822297e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d2aa425-fe38-4068-8db3-4813b822297e" (UID: "7d2aa425-fe38-4068-8db3-4813b822297e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:20:38.306490 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.306375    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2aa425-fe38-4068-8db3-4813b822297e-dshm" (OuterVolumeSpecName: "dshm") pod "7d2aa425-fe38-4068-8db3-4813b822297e" (UID: "7d2aa425-fe38-4068-8db3-4813b822297e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:20:38.306490 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.306389    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2aa425-fe38-4068-8db3-4813b822297e-kube-api-access-nfn9c" (OuterVolumeSpecName: "kube-api-access-nfn9c") pod "7d2aa425-fe38-4068-8db3-4813b822297e" (UID: "7d2aa425-fe38-4068-8db3-4813b822297e"). InnerVolumeSpecName "kube-api-access-nfn9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:20:38.404491 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.404452    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d2aa425-fe38-4068-8db3-4813b822297e-model-cache\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:38.404491 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.404483    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d2aa425-fe38-4068-8db3-4813b822297e-tls-certs\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:38.404491 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.404493    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d2aa425-fe38-4068-8db3-4813b822297e-tmp-dir\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:38.404755 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.404502    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfn9c\" (UniqueName: \"kubernetes.io/projected/7d2aa425-fe38-4068-8db3-4813b822297e-kube-api-access-nfn9c\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:38.404755 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.404511    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d2aa425-fe38-4068-8db3-4813b822297e-dshm\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:20:38.757100 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.757065    2578 generic.go:356] "Generic (PLEG): container finished" podID="7d2aa425-fe38-4068-8db3-4813b822297e" containerID="c52783fc2dbfe7eb9ec35d3084239e316ed0633cc1f1461f2d9f40a21b34cdb1" exitCode=0
Sep 11 21:20:38.757537 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.757107    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"
Sep 11 21:20:38.757537 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.757149    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km" event={"ID":"7d2aa425-fe38-4068-8db3-4813b822297e","Type":"ContainerDied","Data":"c52783fc2dbfe7eb9ec35d3084239e316ed0633cc1f1461f2d9f40a21b34cdb1"}
Sep 11 21:20:38.757537 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.757192    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km" event={"ID":"7d2aa425-fe38-4068-8db3-4813b822297e","Type":"ContainerDied","Data":"1ae592aecbc8f5d8a88106fb273b64517a892a6693d35b739c036b579257438a"}
Sep 11 21:20:38.757537 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.757214    2578 scope.go:117] "RemoveContainer" containerID="c52783fc2dbfe7eb9ec35d3084239e316ed0633cc1f1461f2d9f40a21b34cdb1"
Sep 11 21:20:38.758742 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.758720    2578 generic.go:356] "Generic (PLEG): container finished" podID="7e6496ab-6ff6-476a-9a3b-f3b316c7099e" containerID="84c99a94b449e7432646b2b959110905cbcfe4f58d68bd0b8cb8b16b8f19b034" exitCode=0
Sep 11 21:20:38.758886 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.758771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7" event={"ID":"7e6496ab-6ff6-476a-9a3b-f3b316c7099e","Type":"ContainerDied","Data":"84c99a94b449e7432646b2b959110905cbcfe4f58d68bd0b8cb8b16b8f19b034"}
Sep 11 21:20:38.758886 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.758778    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"
Sep 11 21:20:38.758886 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.758799    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7" event={"ID":"7e6496ab-6ff6-476a-9a3b-f3b316c7099e","Type":"ContainerDied","Data":"c264a2dfb73ec31d2b3f41687fdf44ef8df5c281fdf47861d42c83283ccfb1c3"}
Sep 11 21:20:38.766692 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.766668    2578 scope.go:117] "RemoveContainer" containerID="c52783fc2dbfe7eb9ec35d3084239e316ed0633cc1f1461f2d9f40a21b34cdb1"
Sep 11 21:20:38.767025 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:20:38.767000    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c52783fc2dbfe7eb9ec35d3084239e316ed0633cc1f1461f2d9f40a21b34cdb1\": container with ID starting with c52783fc2dbfe7eb9ec35d3084239e316ed0633cc1f1461f2d9f40a21b34cdb1 not found: ID does not exist" containerID="c52783fc2dbfe7eb9ec35d3084239e316ed0633cc1f1461f2d9f40a21b34cdb1"
Sep 11 21:20:38.767081 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.767038    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c52783fc2dbfe7eb9ec35d3084239e316ed0633cc1f1461f2d9f40a21b34cdb1"} err="failed to get container status \"c52783fc2dbfe7eb9ec35d3084239e316ed0633cc1f1461f2d9f40a21b34cdb1\": rpc error: code = NotFound desc = could not find container \"c52783fc2dbfe7eb9ec35d3084239e316ed0633cc1f1461f2d9f40a21b34cdb1\": container with ID starting with c52783fc2dbfe7eb9ec35d3084239e316ed0633cc1f1461f2d9f40a21b34cdb1 not found: ID does not exist"
Sep 11 21:20:38.767081 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.767063    2578 scope.go:117] "RemoveContainer" containerID="84c99a94b449e7432646b2b959110905cbcfe4f58d68bd0b8cb8b16b8f19b034"
Sep 11 21:20:38.774816 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.774800    2578 scope.go:117] "RemoveContainer" containerID="84c99a94b449e7432646b2b959110905cbcfe4f58d68bd0b8cb8b16b8f19b034"
Sep 11 21:20:38.775094 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:20:38.775072    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84c99a94b449e7432646b2b959110905cbcfe4f58d68bd0b8cb8b16b8f19b034\": container with ID starting with 84c99a94b449e7432646b2b959110905cbcfe4f58d68bd0b8cb8b16b8f19b034 not found: ID does not exist" containerID="84c99a94b449e7432646b2b959110905cbcfe4f58d68bd0b8cb8b16b8f19b034"
Sep 11 21:20:38.775170 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.775099    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84c99a94b449e7432646b2b959110905cbcfe4f58d68bd0b8cb8b16b8f19b034"} err="failed to get container status \"84c99a94b449e7432646b2b959110905cbcfe4f58d68bd0b8cb8b16b8f19b034\": rpc error: code = NotFound desc = could not find container \"84c99a94b449e7432646b2b959110905cbcfe4f58d68bd0b8cb8b16b8f19b034\": container with ID starting with 84c99a94b449e7432646b2b959110905cbcfe4f58d68bd0b8cb8b16b8f19b034 not found: ID does not exist"
Sep 11 21:20:38.784198 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.784159    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"]
Sep 11 21:20:38.786111 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.786087    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ea5e33/simulator-w0-ea5e33-kserve-b8d8f7df5-kc8b7"]
Sep 11 21:20:38.797749 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.797723    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"]
Sep 11 21:20:38.803958 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:38.803937    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-962a7d/simulator-w4-962a7d-kserve-5cc984d758-mn6km"]
Sep 11 21:20:39.926904 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:39.926871    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d2aa425-fe38-4068-8db3-4813b822297e" path="/var/lib/kubelet/pods/7d2aa425-fe38-4068-8db3-4813b822297e/volumes"
Sep 11 21:20:39.927284 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:39.927258    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e6496ab-6ff6-476a-9a3b-f3b316c7099e" path="/var/lib/kubelet/pods/7e6496ab-6ff6-476a-9a3b-f3b316c7099e/volumes"
Sep 11 21:20:41.214210 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:41.214172    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:20:53.998388 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:53.998350    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"]
Sep 11 21:20:53.998781 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:53.998767    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e6496ab-6ff6-476a-9a3b-f3b316c7099e" containerName="main"
Sep 11 21:20:53.998843 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:53.998784    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e6496ab-6ff6-476a-9a3b-f3b316c7099e" containerName="main"
Sep 11 21:20:53.998843 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:53.998801    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9" containerName="payload-pre-processing"
Sep 11 21:20:53.998843 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:53.998810    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9" containerName="payload-pre-processing"
Sep 11 21:20:53.998937 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:53.998844    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d2aa425-fe38-4068-8db3-4813b822297e" containerName="main"
Sep 11 21:20:53.998937 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:53.998852    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d2aa425-fe38-4068-8db3-4813b822297e" containerName="main"
Sep 11 21:20:53.998937 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:53.998865    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb307345-d97a-470b-8d73-1406c761d5c4" containerName="payload-processing"
Sep 11 21:20:53.998937 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:53.998873    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb307345-d97a-470b-8d73-1406c761d5c4" containerName="payload-processing"
Sep 11 21:20:53.998937 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:53.998882    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5d61a25-b33a-46bf-ad2b-6bdb20edbce9" containerName="revoke-keys"
Sep 11 21:20:53.998937 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:53.998890    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5d61a25-b33a-46bf-ad2b-6bdb20edbce9" containerName="revoke-keys"
Sep 11 21:20:53.998937 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:53.998907    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e200df09-7c52-432b-a646-5c6e28d28b0a" containerName="main"
Sep 11 21:20:53.998937 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:53.998914    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e200df09-7c52-432b-a646-5c6e28d28b0a" containerName="main"
Sep 11 21:20:53.999159 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:53.998981    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d2aa425-fe38-4068-8db3-4813b822297e" containerName="main"
Sep 11 21:20:53.999159 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:53.998992    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb307345-d97a-470b-8d73-1406c761d5c4" containerName="payload-processing"
Sep 11 21:20:53.999159 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:53.999002    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4e9c119-d654-4e0e-9b2a-215f5c0ebfd9" containerName="payload-pre-processing"
Sep 11 21:20:53.999159 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:53.999011    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e200df09-7c52-432b-a646-5c6e28d28b0a" containerName="main"
Sep 11 21:20:53.999159 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:53.999019    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5d61a25-b33a-46bf-ad2b-6bdb20edbce9" containerName="revoke-keys"
Sep 11 21:20:53.999159 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:53.999028    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e6496ab-6ff6-476a-9a3b-f3b316c7099e" containerName="main"
Sep 11 21:20:54.003443 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.003421    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.006131 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.006107    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-7ae28e-openshift-default-dockercfg-s5mjr\""
Sep 11 21:20:54.006258 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.006149    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 21:20:54.013241 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.013216    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"]
Sep 11 21:20:54.130137 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.130103    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-credential-socket\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.130328 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.130173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-istio-envoy\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.130328 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.130243    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-workload-certs\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.130328 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.130311    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-istio-data\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.130501 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.130347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-workload-socket\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.130501 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.130387    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s86d\" (UniqueName: \"kubernetes.io/projected/da5b519a-a0f4-4107-ae63-8d70fe957d12-kube-api-access-4s86d\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.130501 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.130419    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da5b519a-a0f4-4107-ae63-8d70fe957d12-istio-podinfo\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.130501 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.130449    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da5b519a-a0f4-4107-ae63-8d70fe957d12-istio-token\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.130501 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.130500    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da5b519a-a0f4-4107-ae63-8d70fe957d12-istiod-ca-cert\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.230901 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.230868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da5b519a-a0f4-4107-ae63-8d70fe957d12-istiod-ca-cert\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.230901 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.230902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-credential-socket\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.231150 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.230923    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-istio-envoy\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.231150 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.230944    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-workload-certs\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.231150 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.230978    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-istio-data\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.231150 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.231000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-workload-socket\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.231150 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.231050    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s86d\" (UniqueName: \"kubernetes.io/projected/da5b519a-a0f4-4107-ae63-8d70fe957d12-kube-api-access-4s86d\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.231150 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.231107    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da5b519a-a0f4-4107-ae63-8d70fe957d12-istio-podinfo\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.231150 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.231141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da5b519a-a0f4-4107-ae63-8d70fe957d12-istio-token\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.231518 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.231407    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-credential-socket\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.231518 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.231423    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-workload-certs\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.231624 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.231606    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-istio-data\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.231680 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.231664    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-workload-socket\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.231856 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.231816    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da5b519a-a0f4-4107-ae63-8d70fe957d12-istiod-ca-cert\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.233278 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.233255    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-istio-envoy\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.233625 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.233608    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da5b519a-a0f4-4107-ae63-8d70fe957d12-istio-podinfo\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.238989 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.238969    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da5b519a-a0f4-4107-ae63-8d70fe957d12-istio-token\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.239400 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.239380    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s86d\" (UniqueName: \"kubernetes.io/projected/da5b519a-a0f4-4107-ae63-8d70fe957d12-kube-api-access-4s86d\") pod \"e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") " pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.314250 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.314161    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:54.591196 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.591109    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"]
Sep 11 21:20:54.595007 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:20:54.594970    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5b519a_a0f4_4107_ae63_8d70fe957d12.slice/crio-93127ef09ae4ac51e432e63d44bf881c514dd4f22b967ea25a7036b070ee7586 WatchSource:0}: Error finding container 93127ef09ae4ac51e432e63d44bf881c514dd4f22b967ea25a7036b070ee7586: Status 404 returned error can't find the container with id 93127ef09ae4ac51e432e63d44bf881c514dd4f22b967ea25a7036b070ee7586
Sep 11 21:20:54.597127 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.597094    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:20:54.597210 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.597160    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:20:54.597210 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.597187    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:20:54.830578 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.830541    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb" event={"ID":"da5b519a-a0f4-4107-ae63-8d70fe957d12","Type":"ContainerStarted","Data":"b14fca34c124871c6e986f6387b4973222bd3789049ee4a85dd7f7122ee96418"}
Sep 11 21:20:54.830747 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.830592    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb" event={"ID":"da5b519a-a0f4-4107-ae63-8d70fe957d12","Type":"ContainerStarted","Data":"93127ef09ae4ac51e432e63d44bf881c514dd4f22b967ea25a7036b070ee7586"}
Sep 11 21:20:54.855350 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:54.855249    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb" podStartSLOduration=1.855234547 podStartE2EDuration="1.855234547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:20:54.853941917 +0000 UTC m=+921.557911206" watchObservedRunningTime="2026-09-11 21:20:54.855234547 +0000 UTC m=+921.559203835"
Sep 11 21:20:55.314980 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:55.314946    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:55.319778 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:55.319754    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:55.840352 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:55.840320    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:55.841240 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:55.841219    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:20:58.833509 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:20:58.833476    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:21:10.348844 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.342657    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"]
Sep 11 21:21:10.358537 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.358497    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.361257 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.361226    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e85410da-openshift-default-dockercfg-w4h5j\""
Sep 11 21:21:10.362305 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.362277    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"]
Sep 11 21:21:10.476990 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.476941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e53c31c5-a767-4f11-8da8-559476ad3689-istio-podinfo\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.476990 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.476996    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-credential-socket\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.477232 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.477028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-workload-certs\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.477232 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.477096    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-istio-envoy\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.477232 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.477117    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-istio-data\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.477232 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.477134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e53c31c5-a767-4f11-8da8-559476ad3689-istio-token\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.477232 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.477148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n994\" (UniqueName: \"kubernetes.io/projected/e53c31c5-a767-4f11-8da8-559476ad3689-kube-api-access-6n994\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.477232 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.477167    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-workload-socket\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.477232 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.477206    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e53c31c5-a767-4f11-8da8-559476ad3689-istiod-ca-cert\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.577738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.577699    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e53c31c5-a767-4f11-8da8-559476ad3689-istio-podinfo\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.577738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.577738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-credential-socket\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.578010 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.577856    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-workload-certs\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.578010 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.577913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-istio-envoy\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.578010 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.577934    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-istio-data\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.578010 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.577956    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e53c31c5-a767-4f11-8da8-559476ad3689-istio-token\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.578307 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.578252    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-credential-socket\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.578307 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.578286    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-workload-certs\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.578307 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.577980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n994\" (UniqueName: \"kubernetes.io/projected/e53c31c5-a767-4f11-8da8-559476ad3689-kube-api-access-6n994\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.578550 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.578340    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-workload-socket\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.578550 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.578364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e53c31c5-a767-4f11-8da8-559476ad3689-istiod-ca-cert\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.578550 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.578407    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-istio-data\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.578715 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.578613    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-workload-socket\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.579014 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.578993    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e53c31c5-a767-4f11-8da8-559476ad3689-istiod-ca-cert\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.580126 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.580106    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-istio-envoy\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.580250 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.580234    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e53c31c5-a767-4f11-8da8-559476ad3689-istio-podinfo\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.586671 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.586646    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n994\" (UniqueName: \"kubernetes.io/projected/e53c31c5-a767-4f11-8da8-559476ad3689-kube-api-access-6n994\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.586786 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.586773    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e53c31c5-a767-4f11-8da8-559476ad3689-istio-token\") pod \"e2e-mt-e85410da-openshift-default-545cf9b67b-pls99\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") " pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.672795 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.672754    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:10.978715 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.978690    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"]
Sep 11 21:21:10.980715 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:21:10.980690    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode53c31c5_a767_4f11_8da8_559476ad3689.slice/crio-bebbe41981378e4cabba58452bde2f60e86363e7cd1a6d9c59b59eeb5ee61529 WatchSource:0}: Error finding container bebbe41981378e4cabba58452bde2f60e86363e7cd1a6d9c59b59eeb5ee61529: Status 404 returned error can't find the container with id bebbe41981378e4cabba58452bde2f60e86363e7cd1a6d9c59b59eeb5ee61529
Sep 11 21:21:10.982882 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.982845    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:21:10.982945 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.982933    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:21:10.982988 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:10.982973    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:21:11.830584 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:11.830524    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:21:11.914392 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:11.914358    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99" event={"ID":"e53c31c5-a767-4f11-8da8-559476ad3689","Type":"ContainerStarted","Data":"b86c0a06221583ee7c14af876997007a8041c76f057e837c447b46ee552f202a"}
Sep 11 21:21:11.914565 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:11.914407    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99" event={"ID":"e53c31c5-a767-4f11-8da8-559476ad3689","Type":"ContainerStarted","Data":"bebbe41981378e4cabba58452bde2f60e86363e7cd1a6d9c59b59eeb5ee61529"}
Sep 11 21:21:11.936949 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:11.936869    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99" podStartSLOduration=1.9368471760000001 podStartE2EDuration="1.936847176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:21:11.934790461 +0000 UTC m=+938.638759761" watchObservedRunningTime="2026-09-11 21:21:11.936847176 +0000 UTC m=+938.640816466"
Sep 11 21:21:12.673442 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:12.673407    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:12.678651 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:12.678618    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:12.918690 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:12.918659    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:12.919638 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:12.919614    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:16.937164 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:16.937133    2578 generic.go:356] "Generic (PLEG): container finished" podID="da5b519a-a0f4-4107-ae63-8d70fe957d12" containerID="b14fca34c124871c6e986f6387b4973222bd3789049ee4a85dd7f7122ee96418" exitCode=0
Sep 11 21:21:16.937584 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:16.937196    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb" event={"ID":"da5b519a-a0f4-4107-ae63-8d70fe957d12","Type":"ContainerDied","Data":"b14fca34c124871c6e986f6387b4973222bd3789049ee4a85dd7f7122ee96418"}
Sep 11 21:21:16.937584 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:16.937550    2578 scope.go:117] "RemoveContainer" containerID="b14fca34c124871c6e986f6387b4973222bd3789049ee4a85dd7f7122ee96418"
Sep 11 21:21:16.938578 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:16.938544    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:21:16.938676 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:16.938611    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:21:16.938676 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:16.938641    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:21:17.946462 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:17.946425    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb" event={"ID":"da5b519a-a0f4-4107-ae63-8d70fe957d12","Type":"ContainerStarted","Data":"039c3e7420ab20e9cbf41c4eb676940b4c9d8f0e8802cd6388e1a8d0af0acf99"}
Sep 11 21:21:18.314718 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:18.314620    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:21:18.315030 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:18.315003    2578 patch_prober.go:28] interesting pod/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 11 21:21:18.315146 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:18.315072    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb" podUID="da5b519a-a0f4-4107-ae63-8d70fe957d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 11 21:21:19.315087 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:19.315055    2578 patch_prober.go:28] interesting pod/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 11 21:21:19.315481 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:19.315112    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb" podUID="da5b519a-a0f4-4107-ae63-8d70fe957d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 11 21:21:20.314830 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:20.314779    2578 patch_prober.go:28] interesting pod/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 11 21:21:20.315018 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:20.314856    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb" podUID="da5b519a-a0f4-4107-ae63-8d70fe957d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 11 21:21:20.434870 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:20.434810    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg"]
Sep 11 21:21:20.438890 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:20.438864    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg"
Sep 11 21:21:20.443344 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:20.441679    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e85410da-dockercfg-pw7lt\""
Sep 11 21:21:20.443863 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:20.443844    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e85410da\""
Sep 11 21:21:20.449152 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:20.449122    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg"]
Sep 11 21:21:20.466286 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:20.466253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ed1693-9136-461b-8bc7-7b66d5ee3ee3-plugins-config-volume\") pod \"payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg\" (UID: \"36ed1693-9136-461b-8bc7-7b66d5ee3ee3\") " pod="openshift-ingress/payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg"
Sep 11 21:21:20.466435 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:20.466310    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22jhv\" (UniqueName: \"kubernetes.io/projected/36ed1693-9136-461b-8bc7-7b66d5ee3ee3-kube-api-access-22jhv\") pod \"payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg\" (UID: \"36ed1693-9136-461b-8bc7-7b66d5ee3ee3\") " pod="openshift-ingress/payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg"
Sep 11 21:21:20.566883 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:20.566794    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22jhv\" (UniqueName: \"kubernetes.io/projected/36ed1693-9136-461b-8bc7-7b66d5ee3ee3-kube-api-access-22jhv\") pod \"payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg\" (UID: \"36ed1693-9136-461b-8bc7-7b66d5ee3ee3\") " pod="openshift-ingress/payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg"
Sep 11 21:21:20.567057 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:20.566949    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ed1693-9136-461b-8bc7-7b66d5ee3ee3-plugins-config-volume\") pod \"payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg\" (UID: \"36ed1693-9136-461b-8bc7-7b66d5ee3ee3\") " pod="openshift-ingress/payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg"
Sep 11 21:21:20.567644 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:20.567621    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ed1693-9136-461b-8bc7-7b66d5ee3ee3-plugins-config-volume\") pod \"payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg\" (UID: \"36ed1693-9136-461b-8bc7-7b66d5ee3ee3\") " pod="openshift-ingress/payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg"
Sep 11 21:21:20.574770 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:20.574746    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22jhv\" (UniqueName: \"kubernetes.io/projected/36ed1693-9136-461b-8bc7-7b66d5ee3ee3-kube-api-access-22jhv\") pod \"payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg\" (UID: \"36ed1693-9136-461b-8bc7-7b66d5ee3ee3\") " pod="openshift-ingress/payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg"
Sep 11 21:21:20.757178 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:20.757135    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg"
Sep 11 21:21:21.032678 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:21.032638    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg"]
Sep 11 21:21:21.039890 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:21:21.039852    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ed1693_9136_461b_8bc7_7b66d5ee3ee3.slice/crio-32a1c73d5e9f56627059cfa7600a727e77e53beb5b5bc1716937791f9270985a WatchSource:0}: Error finding container 32a1c73d5e9f56627059cfa7600a727e77e53beb5b5bc1716937791f9270985a: Status 404 returned error can't find the container with id 32a1c73d5e9f56627059cfa7600a727e77e53beb5b5bc1716937791f9270985a
Sep 11 21:21:21.314844 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:21.314735    2578 patch_prober.go:28] interesting pod/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 11 21:21:21.314844 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:21.314796    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb" podUID="da5b519a-a0f4-4107-ae63-8d70fe957d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 11 21:21:21.968479 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:21.968422    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg" event={"ID":"36ed1693-9136-461b-8bc7-7b66d5ee3ee3","Type":"ContainerStarted","Data":"9f8795d4996c2b323b67498a4c3781524ce9a5ffa499d6534ffe3cae2c5fa180"}
Sep 11 21:21:21.968909 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:21.968502    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg" event={"ID":"36ed1693-9136-461b-8bc7-7b66d5ee3ee3","Type":"ContainerStarted","Data":"32a1c73d5e9f56627059cfa7600a727e77e53beb5b5bc1716937791f9270985a"}
Sep 11 21:21:21.968909 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:21.968603    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg"
Sep 11 21:21:21.988089 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:21.988031    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg" podStartSLOduration=1.988009265 podStartE2EDuration="1.988009265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:21:21.985491344 +0000 UTC m=+948.689460634" watchObservedRunningTime="2026-09-11 21:21:21.988009265 +0000 UTC m=+948.691978553"
Sep 11 21:21:22.315393 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:22.315302    2578 patch_prober.go:28] interesting pod/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 11 21:21:22.315393 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:22.315373    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb" podUID="da5b519a-a0f4-4107-ae63-8d70fe957d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 11 21:21:23.318996 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:23.318967    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:21:23.319440 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:23.319256    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:21:23.320045 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:23.320031    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:21:25.019004 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:25.018963    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:21:26.999806 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:26.999770    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"]
Sep 11 21:21:27.003300 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.003281    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:21:27.005633 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.005608    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2b95f8\"/\"openshift-service-ca.crt\""
Sep 11 21:21:27.006308 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.006276    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2b95f8\"/\"kube-root-ca.crt\""
Sep 11 21:21:27.006434 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.006360    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2b95f8\"/\"default-dockercfg-j6g5q\""
Sep 11 21:21:27.006434 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.006412    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2b95f8\"/\"test-model-2b95f8-kserve-self-signed-certs\""
Sep 11 21:21:27.012090 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.012072    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlkmx\" (UniqueName: \"kubernetes.io/projected/70727ea4-aa9c-46c4-929e-d3947353fc60-kube-api-access-nlkmx\") pod \"test-model-2b95f8-kserve-647857d89-m298t\" (UID: \"70727ea4-aa9c-46c4-929e-d3947353fc60\") " pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:21:27.012185 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.012100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70727ea4-aa9c-46c4-929e-d3947353fc60-home\") pod \"test-model-2b95f8-kserve-647857d89-m298t\" (UID: \"70727ea4-aa9c-46c4-929e-d3947353fc60\") " pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:21:27.012185 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.012144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70727ea4-aa9c-46c4-929e-d3947353fc60-model-cache\") pod \"test-model-2b95f8-kserve-647857d89-m298t\" (UID: \"70727ea4-aa9c-46c4-929e-d3947353fc60\") " pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:21:27.012185 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.012166    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70727ea4-aa9c-46c4-929e-d3947353fc60-tls-certs\") pod \"test-model-2b95f8-kserve-647857d89-m298t\" (UID: \"70727ea4-aa9c-46c4-929e-d3947353fc60\") " pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:21:27.012291 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.012185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70727ea4-aa9c-46c4-929e-d3947353fc60-dshm\") pod \"test-model-2b95f8-kserve-647857d89-m298t\" (UID: \"70727ea4-aa9c-46c4-929e-d3947353fc60\") " pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:21:27.012291 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.012240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70727ea4-aa9c-46c4-929e-d3947353fc60-tmp-dir\") pod \"test-model-2b95f8-kserve-647857d89-m298t\" (UID: \"70727ea4-aa9c-46c4-929e-d3947353fc60\") " pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:21:27.016282 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.016260    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"]
Sep 11 21:21:27.113300 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.113258    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70727ea4-aa9c-46c4-929e-d3947353fc60-tls-certs\") pod \"test-model-2b95f8-kserve-647857d89-m298t\" (UID: \"70727ea4-aa9c-46c4-929e-d3947353fc60\") " pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:21:27.113491 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.113306    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70727ea4-aa9c-46c4-929e-d3947353fc60-dshm\") pod \"test-model-2b95f8-kserve-647857d89-m298t\" (UID: \"70727ea4-aa9c-46c4-929e-d3947353fc60\") " pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:21:27.113491 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.113352    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70727ea4-aa9c-46c4-929e-d3947353fc60-tmp-dir\") pod \"test-model-2b95f8-kserve-647857d89-m298t\" (UID: \"70727ea4-aa9c-46c4-929e-d3947353fc60\") " pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:21:27.113590 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.113500    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlkmx\" (UniqueName: \"kubernetes.io/projected/70727ea4-aa9c-46c4-929e-d3947353fc60-kube-api-access-nlkmx\") pod \"test-model-2b95f8-kserve-647857d89-m298t\" (UID: \"70727ea4-aa9c-46c4-929e-d3947353fc60\") " pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:21:27.113590 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.113546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70727ea4-aa9c-46c4-929e-d3947353fc60-home\") pod \"test-model-2b95f8-kserve-647857d89-m298t\" (UID: \"70727ea4-aa9c-46c4-929e-d3947353fc60\") " pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:21:27.113681 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.113605    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70727ea4-aa9c-46c4-929e-d3947353fc60-model-cache\") pod \"test-model-2b95f8-kserve-647857d89-m298t\" (UID: \"70727ea4-aa9c-46c4-929e-d3947353fc60\") " pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:21:27.113779 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.113760    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70727ea4-aa9c-46c4-929e-d3947353fc60-tmp-dir\") pod \"test-model-2b95f8-kserve-647857d89-m298t\" (UID: \"70727ea4-aa9c-46c4-929e-d3947353fc60\") " pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:21:27.113967 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.113948    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70727ea4-aa9c-46c4-929e-d3947353fc60-model-cache\") pod \"test-model-2b95f8-kserve-647857d89-m298t\" (UID: \"70727ea4-aa9c-46c4-929e-d3947353fc60\") " pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:21:27.114047 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.113982    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70727ea4-aa9c-46c4-929e-d3947353fc60-home\") pod \"test-model-2b95f8-kserve-647857d89-m298t\" (UID: \"70727ea4-aa9c-46c4-929e-d3947353fc60\") " pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:21:27.115745 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.115723    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70727ea4-aa9c-46c4-929e-d3947353fc60-dshm\") pod \"test-model-2b95f8-kserve-647857d89-m298t\" (UID: \"70727ea4-aa9c-46c4-929e-d3947353fc60\") " pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:21:27.116016 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.115998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70727ea4-aa9c-46c4-929e-d3947353fc60-tls-certs\") pod \"test-model-2b95f8-kserve-647857d89-m298t\" (UID: \"70727ea4-aa9c-46c4-929e-d3947353fc60\") " pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:21:27.122840 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.122795    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlkmx\" (UniqueName: \"kubernetes.io/projected/70727ea4-aa9c-46c4-929e-d3947353fc60-kube-api-access-nlkmx\") pod \"test-model-2b95f8-kserve-647857d89-m298t\" (UID: \"70727ea4-aa9c-46c4-929e-d3947353fc60\") " pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:21:27.313172 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.313077    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:21:27.600842 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.600608    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"]
Sep 11 21:21:27.603416 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:21:27.603383    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70727ea4_aa9c_46c4_929e_d3947353fc60.slice/crio-f91692fde974af98a457b901dedb8e5c055138296e036abc6abe49a229660885 WatchSource:0}: Error finding container f91692fde974af98a457b901dedb8e5c055138296e036abc6abe49a229660885: Status 404 returned error can't find the container with id f91692fde974af98a457b901dedb8e5c055138296e036abc6abe49a229660885
Sep 11 21:21:27.998299 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.998262    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t" event={"ID":"70727ea4-aa9c-46c4-929e-d3947353fc60","Type":"ContainerStarted","Data":"4890399575ddf779fb4cc40eaa07eea832c53c72fbc98cac8828dca7cdc28055"}
Sep 11 21:21:27.998299 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:27.998306    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t" event={"ID":"70727ea4-aa9c-46c4-929e-d3947353fc60","Type":"ContainerStarted","Data":"f91692fde974af98a457b901dedb8e5c055138296e036abc6abe49a229660885"}
Sep 11 21:21:32.975344 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:32.975317    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg"
Sep 11 21:21:32.995079 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:32.995034    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t" podStartSLOduration=6.995017365 podStartE2EDuration="6.995017365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:21:28.028609931 +0000 UTC m=+954.732579221" watchObservedRunningTime="2026-09-11 21:21:32.995017365 +0000 UTC m=+959.698986654"
Sep 11 21:21:36.036303 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:36.036213    2578 generic.go:356] "Generic (PLEG): container finished" podID="e53c31c5-a767-4f11-8da8-559476ad3689" containerID="b86c0a06221583ee7c14af876997007a8041c76f057e837c447b46ee552f202a" exitCode=0
Sep 11 21:21:36.036303 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:36.036284    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99" event={"ID":"e53c31c5-a767-4f11-8da8-559476ad3689","Type":"ContainerDied","Data":"b86c0a06221583ee7c14af876997007a8041c76f057e837c447b46ee552f202a"}
Sep 11 21:21:36.036924 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:36.036787    2578 scope.go:117] "RemoveContainer" containerID="b86c0a06221583ee7c14af876997007a8041c76f057e837c447b46ee552f202a"
Sep 11 21:21:36.037987 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:36.037958    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:21:36.038054 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:36.038029    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:21:36.038090 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:36.038062    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:21:37.046845 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:37.046784    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99" event={"ID":"e53c31c5-a767-4f11-8da8-559476ad3689","Type":"ContainerStarted","Data":"79aa02934b1ab09cf15d817425028a64e8a37a54ac78bdff2309121bedc1db83"}
Sep 11 21:21:37.313883 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:37.313743    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:21:37.313883 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:37.313812    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:21:37.350907 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:37.350873    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:21:37.673306 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:37.673270    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:37.673515 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:37.673493    2578 patch_prober.go:28] interesting pod/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 11 21:21:37.673561 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:37.673541    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99" podUID="e53c31c5-a767-4f11-8da8-559476ad3689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 21:21:38.062685 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:38.062600    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:21:38.673450 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:38.673406    2578 patch_prober.go:28] interesting pod/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 11 21:21:38.673650 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:38.673474    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99" podUID="e53c31c5-a767-4f11-8da8-559476ad3689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 21:21:39.674146 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:39.674104    2578 patch_prober.go:28] interesting pod/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 11 21:21:39.674538 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:39.674161    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99" podUID="e53c31c5-a767-4f11-8da8-559476ad3689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 21:21:40.673280 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:40.673241    2578 patch_prober.go:28] interesting pod/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 11 21:21:40.673498 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:40.673300    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99" podUID="e53c31c5-a767-4f11-8da8-559476ad3689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 21:21:41.673734 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:41.673690    2578 patch_prober.go:28] interesting pod/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 11 21:21:41.674226 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:41.673767    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99" podUID="e53c31c5-a767-4f11-8da8-559476ad3689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 21:21:41.713187 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:41.713148    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:21:42.117278 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.117193    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"]
Sep 11 21:21:42.120811 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.120782    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:21:42.123215 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.123191    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-7ae28e\"/\"openshift-service-ca.crt\""
Sep 11 21:21:42.123798 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.123777    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-7ae28e\"/\"kube-root-ca.crt\""
Sep 11 21:21:42.123941 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.123872    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-7ae28e\"/\"default-dockercfg-jsw9d\""
Sep 11 21:21:42.123997 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.123944    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-7ae28e\"/\"explicit-model-7ae28e-kserve-self-signed-certs\""
Sep 11 21:21:42.138202 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.138167    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"]
Sep 11 21:21:42.178007 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.177974    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f879\" (UniqueName: \"kubernetes.io/projected/615af3ab-3d11-4995-83ba-cffd0d8479a2-kube-api-access-2f879\") pod \"explicit-model-7ae28e-kserve-95b5b6f65-hhmtt\" (UID: \"615af3ab-3d11-4995-83ba-cffd0d8479a2\") " pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:21:42.178201 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.178021    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/615af3ab-3d11-4995-83ba-cffd0d8479a2-model-cache\") pod \"explicit-model-7ae28e-kserve-95b5b6f65-hhmtt\" (UID: \"615af3ab-3d11-4995-83ba-cffd0d8479a2\") " pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:21:42.178201 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.178076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/615af3ab-3d11-4995-83ba-cffd0d8479a2-tls-certs\") pod \"explicit-model-7ae28e-kserve-95b5b6f65-hhmtt\" (UID: \"615af3ab-3d11-4995-83ba-cffd0d8479a2\") " pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:21:42.178201 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.178115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/615af3ab-3d11-4995-83ba-cffd0d8479a2-tmp-dir\") pod \"explicit-model-7ae28e-kserve-95b5b6f65-hhmtt\" (UID: \"615af3ab-3d11-4995-83ba-cffd0d8479a2\") " pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:21:42.178201 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.178150    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/615af3ab-3d11-4995-83ba-cffd0d8479a2-home\") pod \"explicit-model-7ae28e-kserve-95b5b6f65-hhmtt\" (UID: \"615af3ab-3d11-4995-83ba-cffd0d8479a2\") " pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:21:42.178201 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.178186    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/615af3ab-3d11-4995-83ba-cffd0d8479a2-dshm\") pod \"explicit-model-7ae28e-kserve-95b5b6f65-hhmtt\" (UID: \"615af3ab-3d11-4995-83ba-cffd0d8479a2\") " pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:21:42.279581 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.279546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/615af3ab-3d11-4995-83ba-cffd0d8479a2-home\") pod \"explicit-model-7ae28e-kserve-95b5b6f65-hhmtt\" (UID: \"615af3ab-3d11-4995-83ba-cffd0d8479a2\") " pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:21:42.279581 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.279595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/615af3ab-3d11-4995-83ba-cffd0d8479a2-dshm\") pod \"explicit-model-7ae28e-kserve-95b5b6f65-hhmtt\" (UID: \"615af3ab-3d11-4995-83ba-cffd0d8479a2\") " pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:21:42.279874 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.279630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f879\" (UniqueName: \"kubernetes.io/projected/615af3ab-3d11-4995-83ba-cffd0d8479a2-kube-api-access-2f879\") pod \"explicit-model-7ae28e-kserve-95b5b6f65-hhmtt\" (UID: \"615af3ab-3d11-4995-83ba-cffd0d8479a2\") " pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:21:42.279874 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.279792    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/615af3ab-3d11-4995-83ba-cffd0d8479a2-model-cache\") pod \"explicit-model-7ae28e-kserve-95b5b6f65-hhmtt\" (UID: \"615af3ab-3d11-4995-83ba-cffd0d8479a2\") " pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:21:42.279986 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.279923    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/615af3ab-3d11-4995-83ba-cffd0d8479a2-tls-certs\") pod \"explicit-model-7ae28e-kserve-95b5b6f65-hhmtt\" (UID: \"615af3ab-3d11-4995-83ba-cffd0d8479a2\") " pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:21:42.280049 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.280023    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/615af3ab-3d11-4995-83ba-cffd0d8479a2-tmp-dir\") pod \"explicit-model-7ae28e-kserve-95b5b6f65-hhmtt\" (UID: \"615af3ab-3d11-4995-83ba-cffd0d8479a2\") " pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:21:42.280106 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.280054    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/615af3ab-3d11-4995-83ba-cffd0d8479a2-home\") pod \"explicit-model-7ae28e-kserve-95b5b6f65-hhmtt\" (UID: \"615af3ab-3d11-4995-83ba-cffd0d8479a2\") " pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:21:42.280262 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.280241    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/615af3ab-3d11-4995-83ba-cffd0d8479a2-model-cache\") pod \"explicit-model-7ae28e-kserve-95b5b6f65-hhmtt\" (UID: \"615af3ab-3d11-4995-83ba-cffd0d8479a2\") " pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:21:42.280400 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.280377    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/615af3ab-3d11-4995-83ba-cffd0d8479a2-tmp-dir\") pod \"explicit-model-7ae28e-kserve-95b5b6f65-hhmtt\" (UID: \"615af3ab-3d11-4995-83ba-cffd0d8479a2\") " pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:21:42.282256 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.282222    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/615af3ab-3d11-4995-83ba-cffd0d8479a2-dshm\") pod \"explicit-model-7ae28e-kserve-95b5b6f65-hhmtt\" (UID: \"615af3ab-3d11-4995-83ba-cffd0d8479a2\") " pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:21:42.282730 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.282710    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/615af3ab-3d11-4995-83ba-cffd0d8479a2-tls-certs\") pod \"explicit-model-7ae28e-kserve-95b5b6f65-hhmtt\" (UID: \"615af3ab-3d11-4995-83ba-cffd0d8479a2\") " pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:21:42.296734 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.296697    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f879\" (UniqueName: \"kubernetes.io/projected/615af3ab-3d11-4995-83ba-cffd0d8479a2-kube-api-access-2f879\") pod \"explicit-model-7ae28e-kserve-95b5b6f65-hhmtt\" (UID: \"615af3ab-3d11-4995-83ba-cffd0d8479a2\") " pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:21:42.433391 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.433353    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:21:42.677593 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.677559    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:42.677989 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.677920    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:42.678623 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.678603    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:21:42.712304 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:42.712263    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"]
Sep 11 21:21:42.714585 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:21:42.714560    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod615af3ab_3d11_4995_83ba_cffd0d8479a2.slice/crio-401ca3916068d0ad4be659091e722ef5dd3e2be3b33b6a4e5dad28e876a2fed5 WatchSource:0}: Error finding container 401ca3916068d0ad4be659091e722ef5dd3e2be3b33b6a4e5dad28e876a2fed5: Status 404 returned error can't find the container with id 401ca3916068d0ad4be659091e722ef5dd3e2be3b33b6a4e5dad28e876a2fed5
Sep 11 21:21:43.078225 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:43.078130    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt" event={"ID":"615af3ab-3d11-4995-83ba-cffd0d8479a2","Type":"ContainerStarted","Data":"f5c4ffe9cde94ed060520adb30af68f4bc1fa1a874886607ae19ece0dafc4334"}
Sep 11 21:21:43.078225 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:43.078175    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt" event={"ID":"615af3ab-3d11-4995-83ba-cffd0d8479a2","Type":"ContainerStarted","Data":"401ca3916068d0ad4be659091e722ef5dd3e2be3b33b6a4e5dad28e876a2fed5"}
Sep 11 21:21:43.098741 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:43.098687    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt" podStartSLOduration=1.098671405 podStartE2EDuration="1.098671405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:21:43.097131219 +0000 UTC m=+969.801100510" watchObservedRunningTime="2026-09-11 21:21:43.098671405 +0000 UTC m=+969.802640695"
Sep 11 21:21:52.434274 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:52.434237    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:21:52.434670 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:52.434297    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:21:52.447143 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:52.447112    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:21:53.142220 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:53.142189    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:21:56.222045 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:56.221959    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:21:58.067594 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:21:58.067563    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"]
Sep 11 21:22:00.623664 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:00.623633    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg"]
Sep 11 21:22:00.624100 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:00.623855    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg" podUID="36ed1693-9136-461b-8bc7-7b66d5ee3ee3" containerName="payload-processing" containerID="cri-o://9f8795d4996c2b323b67498a4c3781524ce9a5ffa499d6534ffe3cae2c5fa180" gracePeriod=30
Sep 11 21:22:00.878793 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:00.878724    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg"
Sep 11 21:22:01.048969 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:01.048934    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ed1693-9136-461b-8bc7-7b66d5ee3ee3-plugins-config-volume\") pod \"36ed1693-9136-461b-8bc7-7b66d5ee3ee3\" (UID: \"36ed1693-9136-461b-8bc7-7b66d5ee3ee3\") "
Sep 11 21:22:01.048969 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:01.048980    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22jhv\" (UniqueName: \"kubernetes.io/projected/36ed1693-9136-461b-8bc7-7b66d5ee3ee3-kube-api-access-22jhv\") pod \"36ed1693-9136-461b-8bc7-7b66d5ee3ee3\" (UID: \"36ed1693-9136-461b-8bc7-7b66d5ee3ee3\") "
Sep 11 21:22:01.049408 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:01.049381    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ed1693-9136-461b-8bc7-7b66d5ee3ee3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "36ed1693-9136-461b-8bc7-7b66d5ee3ee3" (UID: "36ed1693-9136-461b-8bc7-7b66d5ee3ee3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:22:01.051702 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:01.051670    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ed1693-9136-461b-8bc7-7b66d5ee3ee3-kube-api-access-22jhv" (OuterVolumeSpecName: "kube-api-access-22jhv") pod "36ed1693-9136-461b-8bc7-7b66d5ee3ee3" (UID: "36ed1693-9136-461b-8bc7-7b66d5ee3ee3"). InnerVolumeSpecName "kube-api-access-22jhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:22:01.150601 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:01.150496    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ed1693-9136-461b-8bc7-7b66d5ee3ee3-plugins-config-volume\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:01.150601 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:01.150534    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22jhv\" (UniqueName: \"kubernetes.io/projected/36ed1693-9136-461b-8bc7-7b66d5ee3ee3-kube-api-access-22jhv\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:01.167365 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:01.167332    2578 generic.go:356] "Generic (PLEG): container finished" podID="36ed1693-9136-461b-8bc7-7b66d5ee3ee3" containerID="9f8795d4996c2b323b67498a4c3781524ce9a5ffa499d6534ffe3cae2c5fa180" exitCode=0
Sep 11 21:22:01.167539 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:01.167381    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg"
Sep 11 21:22:01.167539 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:01.167418    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg" event={"ID":"36ed1693-9136-461b-8bc7-7b66d5ee3ee3","Type":"ContainerDied","Data":"9f8795d4996c2b323b67498a4c3781524ce9a5ffa499d6534ffe3cae2c5fa180"}
Sep 11 21:22:01.167539 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:01.167477    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg" event={"ID":"36ed1693-9136-461b-8bc7-7b66d5ee3ee3","Type":"ContainerDied","Data":"32a1c73d5e9f56627059cfa7600a727e77e53beb5b5bc1716937791f9270985a"}
Sep 11 21:22:01.167539 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:01.167500    2578 scope.go:117] "RemoveContainer" containerID="9f8795d4996c2b323b67498a4c3781524ce9a5ffa499d6534ffe3cae2c5fa180"
Sep 11 21:22:01.177460 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:01.177441    2578 scope.go:117] "RemoveContainer" containerID="9f8795d4996c2b323b67498a4c3781524ce9a5ffa499d6534ffe3cae2c5fa180"
Sep 11 21:22:01.177798 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:22:01.177772    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f8795d4996c2b323b67498a4c3781524ce9a5ffa499d6534ffe3cae2c5fa180\": container with ID starting with 9f8795d4996c2b323b67498a4c3781524ce9a5ffa499d6534ffe3cae2c5fa180 not found: ID does not exist" containerID="9f8795d4996c2b323b67498a4c3781524ce9a5ffa499d6534ffe3cae2c5fa180"
Sep 11 21:22:01.177933 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:01.177805    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f8795d4996c2b323b67498a4c3781524ce9a5ffa499d6534ffe3cae2c5fa180"} err="failed to get container status \"9f8795d4996c2b323b67498a4c3781524ce9a5ffa499d6534ffe3cae2c5fa180\": rpc error: code = NotFound desc = could not find container \"9f8795d4996c2b323b67498a4c3781524ce9a5ffa499d6534ffe3cae2c5fa180\": container with ID starting with 9f8795d4996c2b323b67498a4c3781524ce9a5ffa499d6534ffe3cae2c5fa180 not found: ID does not exist"
Sep 11 21:22:01.188525 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:01.188491    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg"]
Sep 11 21:22:01.192972 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:01.192940    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e85410da-7dd8ffb945-z89fg"]
Sep 11 21:22:01.927389 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:01.927354    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36ed1693-9136-461b-8bc7-7b66d5ee3ee3" path="/var/lib/kubelet/pods/36ed1693-9136-461b-8bc7-7b66d5ee3ee3/volumes"
Sep 11 21:22:07.273590 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:07.273548    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"]
Sep 11 21:22:07.274018 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:07.273811    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99" podUID="e53c31c5-a767-4f11-8da8-559476ad3689" containerName="istio-proxy" containerID="cri-o://79aa02934b1ab09cf15d817425028a64e8a37a54ac78bdff2309121bedc1db83" gracePeriod=30
Sep 11 21:22:12.529888 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.529862    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:22:12.549072 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.548974    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e53c31c5-a767-4f11-8da8-559476ad3689-istio-token\") pod \"e53c31c5-a767-4f11-8da8-559476ad3689\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") "
Sep 11 21:22:12.549072 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.549015    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-credential-socket\") pod \"e53c31c5-a767-4f11-8da8-559476ad3689\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") "
Sep 11 21:22:12.549072 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.549060    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-workload-certs\") pod \"e53c31c5-a767-4f11-8da8-559476ad3689\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") "
Sep 11 21:22:12.551006 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.549093    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e53c31c5-a767-4f11-8da8-559476ad3689-istio-podinfo\") pod \"e53c31c5-a767-4f11-8da8-559476ad3689\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") "
Sep 11 21:22:12.551006 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.549108    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-istio-data\") pod \"e53c31c5-a767-4f11-8da8-559476ad3689\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") "
Sep 11 21:22:12.551006 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.549144    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n994\" (UniqueName: \"kubernetes.io/projected/e53c31c5-a767-4f11-8da8-559476ad3689-kube-api-access-6n994\") pod \"e53c31c5-a767-4f11-8da8-559476ad3689\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") "
Sep 11 21:22:12.551006 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.549173    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-istio-envoy\") pod \"e53c31c5-a767-4f11-8da8-559476ad3689\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") "
Sep 11 21:22:12.551006 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.549245    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-workload-socket\") pod \"e53c31c5-a767-4f11-8da8-559476ad3689\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") "
Sep 11 21:22:12.551006 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.549316    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e53c31c5-a767-4f11-8da8-559476ad3689-istiod-ca-cert\") pod \"e53c31c5-a767-4f11-8da8-559476ad3689\" (UID: \"e53c31c5-a767-4f11-8da8-559476ad3689\") "
Sep 11 21:22:12.551006 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.549395    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e53c31c5-a767-4f11-8da8-559476ad3689" (UID: "e53c31c5-a767-4f11-8da8-559476ad3689"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:22:12.551006 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.549422    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e53c31c5-a767-4f11-8da8-559476ad3689" (UID: "e53c31c5-a767-4f11-8da8-559476ad3689"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:22:12.551006 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.549645    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-credential-socket\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:12.551006 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.549665    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-workload-certs\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:12.551006 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.549736    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e53c31c5-a767-4f11-8da8-559476ad3689-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e53c31c5-a767-4f11-8da8-559476ad3689" (UID: "e53c31c5-a767-4f11-8da8-559476ad3689"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:22:12.551006 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.549995    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e53c31c5-a767-4f11-8da8-559476ad3689" (UID: "e53c31c5-a767-4f11-8da8-559476ad3689"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:22:12.551006 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.550171    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-istio-data" (OuterVolumeSpecName: "istio-data") pod "e53c31c5-a767-4f11-8da8-559476ad3689" (UID: "e53c31c5-a767-4f11-8da8-559476ad3689"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:22:12.553334 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.552948    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e53c31c5-a767-4f11-8da8-559476ad3689-istio-token" (OuterVolumeSpecName: "istio-token") pod "e53c31c5-a767-4f11-8da8-559476ad3689" (UID: "e53c31c5-a767-4f11-8da8-559476ad3689"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:22:12.553334 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.553061    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e53c31c5-a767-4f11-8da8-559476ad3689-kube-api-access-6n994" (OuterVolumeSpecName: "kube-api-access-6n994") pod "e53c31c5-a767-4f11-8da8-559476ad3689" (UID: "e53c31c5-a767-4f11-8da8-559476ad3689"). InnerVolumeSpecName "kube-api-access-6n994". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:22:12.553334 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.553115    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e53c31c5-a767-4f11-8da8-559476ad3689" (UID: "e53c31c5-a767-4f11-8da8-559476ad3689"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:22:12.554420 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.554397    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e53c31c5-a767-4f11-8da8-559476ad3689-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e53c31c5-a767-4f11-8da8-559476ad3689" (UID: "e53c31c5-a767-4f11-8da8-559476ad3689"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:22:12.650085 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.650045    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-workload-socket\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:12.650085 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.650077    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e53c31c5-a767-4f11-8da8-559476ad3689-istiod-ca-cert\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:12.650085 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.650086    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e53c31c5-a767-4f11-8da8-559476ad3689-istio-token\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:12.650085 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.650094    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e53c31c5-a767-4f11-8da8-559476ad3689-istio-podinfo\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:12.650478 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.650103    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-istio-data\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:12.650478 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.650112    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n994\" (UniqueName: \"kubernetes.io/projected/e53c31c5-a767-4f11-8da8-559476ad3689-kube-api-access-6n994\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:12.650478 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.650121    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e53c31c5-a767-4f11-8da8-559476ad3689-istio-envoy\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:12.922301 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:12.922209    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:22:13.087420 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.087379    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt" podUID="615af3ab-3d11-4995-83ba-cffd0d8479a2" containerName="main" containerID="cri-o://f5c4ffe9cde94ed060520adb30af68f4bc1fa1a874886607ae19ece0dafc4334" gracePeriod=45
Sep 11 21:22:13.131261 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.131220    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt" podUID="615af3ab-3d11-4995-83ba-cffd0d8479a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 11 21:22:13.235179 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.235145    2578 generic.go:356] "Generic (PLEG): container finished" podID="e53c31c5-a767-4f11-8da8-559476ad3689" containerID="79aa02934b1ab09cf15d817425028a64e8a37a54ac78bdff2309121bedc1db83" exitCode=0
Sep 11 21:22:13.235434 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.235191    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"
Sep 11 21:22:13.235434 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.235231    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99" event={"ID":"e53c31c5-a767-4f11-8da8-559476ad3689","Type":"ContainerDied","Data":"79aa02934b1ab09cf15d817425028a64e8a37a54ac78bdff2309121bedc1db83"}
Sep 11 21:22:13.235434 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.235292    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99" event={"ID":"e53c31c5-a767-4f11-8da8-559476ad3689","Type":"ContainerDied","Data":"bebbe41981378e4cabba58452bde2f60e86363e7cd1a6d9c59b59eeb5ee61529"}
Sep 11 21:22:13.235434 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.235324    2578 scope.go:117] "RemoveContainer" containerID="79aa02934b1ab09cf15d817425028a64e8a37a54ac78bdff2309121bedc1db83"
Sep 11 21:22:13.237167 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.237143    2578 generic.go:356] "Generic (PLEG): container finished" podID="615af3ab-3d11-4995-83ba-cffd0d8479a2" containerID="f5c4ffe9cde94ed060520adb30af68f4bc1fa1a874886607ae19ece0dafc4334" exitCode=0
Sep 11 21:22:13.237269 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.237179    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt" event={"ID":"615af3ab-3d11-4995-83ba-cffd0d8479a2","Type":"ContainerDied","Data":"f5c4ffe9cde94ed060520adb30af68f4bc1fa1a874886607ae19ece0dafc4334"}
Sep 11 21:22:13.244885 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.244863    2578 scope.go:117] "RemoveContainer" containerID="b86c0a06221583ee7c14af876997007a8041c76f057e837c447b46ee552f202a"
Sep 11 21:22:13.253293 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.253271    2578 scope.go:117] "RemoveContainer" containerID="79aa02934b1ab09cf15d817425028a64e8a37a54ac78bdff2309121bedc1db83"
Sep 11 21:22:13.253636 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:22:13.253618    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79aa02934b1ab09cf15d817425028a64e8a37a54ac78bdff2309121bedc1db83\": container with ID starting with 79aa02934b1ab09cf15d817425028a64e8a37a54ac78bdff2309121bedc1db83 not found: ID does not exist" containerID="79aa02934b1ab09cf15d817425028a64e8a37a54ac78bdff2309121bedc1db83"
Sep 11 21:22:13.253684 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.253649    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79aa02934b1ab09cf15d817425028a64e8a37a54ac78bdff2309121bedc1db83"} err="failed to get container status \"79aa02934b1ab09cf15d817425028a64e8a37a54ac78bdff2309121bedc1db83\": rpc error: code = NotFound desc = could not find container \"79aa02934b1ab09cf15d817425028a64e8a37a54ac78bdff2309121bedc1db83\": container with ID starting with 79aa02934b1ab09cf15d817425028a64e8a37a54ac78bdff2309121bedc1db83 not found: ID does not exist"
Sep 11 21:22:13.253684 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.253668    2578 scope.go:117] "RemoveContainer" containerID="b86c0a06221583ee7c14af876997007a8041c76f057e837c447b46ee552f202a"
Sep 11 21:22:13.253992 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:22:13.253973    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b86c0a06221583ee7c14af876997007a8041c76f057e837c447b46ee552f202a\": container with ID starting with b86c0a06221583ee7c14af876997007a8041c76f057e837c447b46ee552f202a not found: ID does not exist" containerID="b86c0a06221583ee7c14af876997007a8041c76f057e837c447b46ee552f202a"
Sep 11 21:22:13.254056 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.253998    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b86c0a06221583ee7c14af876997007a8041c76f057e837c447b46ee552f202a"} err="failed to get container status \"b86c0a06221583ee7c14af876997007a8041c76f057e837c447b46ee552f202a\": rpc error: code = NotFound desc = could not find container \"b86c0a06221583ee7c14af876997007a8041c76f057e837c447b46ee552f202a\": container with ID starting with b86c0a06221583ee7c14af876997007a8041c76f057e837c447b46ee552f202a not found: ID does not exist"
Sep 11 21:22:13.257675 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.257653    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"]
Sep 11 21:22:13.262441 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.262414    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e85410da-openshift-default-545cf9b67b-pls99"]
Sep 11 21:22:13.341358 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.341321    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:22:13.456801 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.456710    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/615af3ab-3d11-4995-83ba-cffd0d8479a2-home\") pod \"615af3ab-3d11-4995-83ba-cffd0d8479a2\" (UID: \"615af3ab-3d11-4995-83ba-cffd0d8479a2\") "
Sep 11 21:22:13.456801 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.456769    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f879\" (UniqueName: \"kubernetes.io/projected/615af3ab-3d11-4995-83ba-cffd0d8479a2-kube-api-access-2f879\") pod \"615af3ab-3d11-4995-83ba-cffd0d8479a2\" (UID: \"615af3ab-3d11-4995-83ba-cffd0d8479a2\") "
Sep 11 21:22:13.457084 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.456806    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/615af3ab-3d11-4995-83ba-cffd0d8479a2-tmp-dir\") pod \"615af3ab-3d11-4995-83ba-cffd0d8479a2\" (UID: \"615af3ab-3d11-4995-83ba-cffd0d8479a2\") "
Sep 11 21:22:13.457084 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.456860    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/615af3ab-3d11-4995-83ba-cffd0d8479a2-tls-certs\") pod \"615af3ab-3d11-4995-83ba-cffd0d8479a2\" (UID: \"615af3ab-3d11-4995-83ba-cffd0d8479a2\") "
Sep 11 21:22:13.457084 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.456879    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/615af3ab-3d11-4995-83ba-cffd0d8479a2-model-cache\") pod \"615af3ab-3d11-4995-83ba-cffd0d8479a2\" (UID: \"615af3ab-3d11-4995-83ba-cffd0d8479a2\") "
Sep 11 21:22:13.457084 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.456907    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/615af3ab-3d11-4995-83ba-cffd0d8479a2-dshm\") pod \"615af3ab-3d11-4995-83ba-cffd0d8479a2\" (UID: \"615af3ab-3d11-4995-83ba-cffd0d8479a2\") "
Sep 11 21:22:13.457084 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.456989    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615af3ab-3d11-4995-83ba-cffd0d8479a2-home" (OuterVolumeSpecName: "home") pod "615af3ab-3d11-4995-83ba-cffd0d8479a2" (UID: "615af3ab-3d11-4995-83ba-cffd0d8479a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:22:13.457339 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.457141    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/615af3ab-3d11-4995-83ba-cffd0d8479a2-home\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:13.457339 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.457150    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615af3ab-3d11-4995-83ba-cffd0d8479a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "615af3ab-3d11-4995-83ba-cffd0d8479a2" (UID: "615af3ab-3d11-4995-83ba-cffd0d8479a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:22:13.457339 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.457177    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615af3ab-3d11-4995-83ba-cffd0d8479a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "615af3ab-3d11-4995-83ba-cffd0d8479a2" (UID: "615af3ab-3d11-4995-83ba-cffd0d8479a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:22:13.459205 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.459172    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615af3ab-3d11-4995-83ba-cffd0d8479a2-dshm" (OuterVolumeSpecName: "dshm") pod "615af3ab-3d11-4995-83ba-cffd0d8479a2" (UID: "615af3ab-3d11-4995-83ba-cffd0d8479a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:22:13.459329 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.459202    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/615af3ab-3d11-4995-83ba-cffd0d8479a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "615af3ab-3d11-4995-83ba-cffd0d8479a2" (UID: "615af3ab-3d11-4995-83ba-cffd0d8479a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:22:13.459329 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.459261    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615af3ab-3d11-4995-83ba-cffd0d8479a2-kube-api-access-2f879" (OuterVolumeSpecName: "kube-api-access-2f879") pod "615af3ab-3d11-4995-83ba-cffd0d8479a2" (UID: "615af3ab-3d11-4995-83ba-cffd0d8479a2"). InnerVolumeSpecName "kube-api-access-2f879". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:22:13.558073 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.558031    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/615af3ab-3d11-4995-83ba-cffd0d8479a2-tmp-dir\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:13.558073 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.558063    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/615af3ab-3d11-4995-83ba-cffd0d8479a2-tls-certs\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:13.558073 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.558072    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/615af3ab-3d11-4995-83ba-cffd0d8479a2-model-cache\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:13.558073 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.558080    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/615af3ab-3d11-4995-83ba-cffd0d8479a2-dshm\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:13.558553 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.558090    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f879\" (UniqueName: \"kubernetes.io/projected/615af3ab-3d11-4995-83ba-cffd0d8479a2-kube-api-access-2f879\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:13.927143 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:13.927115    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e53c31c5-a767-4f11-8da8-559476ad3689" path="/var/lib/kubelet/pods/e53c31c5-a767-4f11-8da8-559476ad3689/volumes"
Sep 11 21:22:14.244167 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:14.244069    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt" event={"ID":"615af3ab-3d11-4995-83ba-cffd0d8479a2","Type":"ContainerDied","Data":"401ca3916068d0ad4be659091e722ef5dd3e2be3b33b6a4e5dad28e876a2fed5"}
Sep 11 21:22:14.244167 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:14.244122    2578 scope.go:117] "RemoveContainer" containerID="f5c4ffe9cde94ed060520adb30af68f4bc1fa1a874886607ae19ece0dafc4334"
Sep 11 21:22:14.244167 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:14.244084    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"
Sep 11 21:22:14.259037 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:14.259001    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"]
Sep 11 21:22:14.262658 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:14.262629    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-7ae28e/explicit-model-7ae28e-kserve-95b5b6f65-hhmtt"]
Sep 11 21:22:15.926752 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:15.926721    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="615af3ab-3d11-4995-83ba-cffd0d8479a2" path="/var/lib/kubelet/pods/615af3ab-3d11-4995-83ba-cffd0d8479a2/volumes"
Sep 11 21:22:16.027838 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:16.027799    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"]
Sep 11 21:22:16.028145 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:16.028087    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb" podUID="da5b519a-a0f4-4107-ae63-8d70fe957d12" containerName="istio-proxy" containerID="cri-o://039c3e7420ab20e9cbf41c4eb676940b4c9d8f0e8802cd6388e1a8d0af0acf99" gracePeriod=30
Sep 11 21:22:21.275074 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.275045    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:22:21.276364 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.276341    2578 generic.go:356] "Generic (PLEG): container finished" podID="da5b519a-a0f4-4107-ae63-8d70fe957d12" containerID="039c3e7420ab20e9cbf41c4eb676940b4c9d8f0e8802cd6388e1a8d0af0acf99" exitCode=0
Sep 11 21:22:21.276475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.276410    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb" event={"ID":"da5b519a-a0f4-4107-ae63-8d70fe957d12","Type":"ContainerDied","Data":"039c3e7420ab20e9cbf41c4eb676940b4c9d8f0e8802cd6388e1a8d0af0acf99"}
Sep 11 21:22:21.276475 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.276471    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb" event={"ID":"da5b519a-a0f4-4107-ae63-8d70fe957d12","Type":"ContainerDied","Data":"93127ef09ae4ac51e432e63d44bf881c514dd4f22b967ea25a7036b070ee7586"}
Sep 11 21:22:21.276549 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.276493    2578 scope.go:117] "RemoveContainer" containerID="039c3e7420ab20e9cbf41c4eb676940b4c9d8f0e8802cd6388e1a8d0af0acf99"
Sep 11 21:22:21.284927 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.284910    2578 scope.go:117] "RemoveContainer" containerID="b14fca34c124871c6e986f6387b4973222bd3789049ee4a85dd7f7122ee96418"
Sep 11 21:22:21.293957 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.293932    2578 scope.go:117] "RemoveContainer" containerID="039c3e7420ab20e9cbf41c4eb676940b4c9d8f0e8802cd6388e1a8d0af0acf99"
Sep 11 21:22:21.294347 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:22:21.294326    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"039c3e7420ab20e9cbf41c4eb676940b4c9d8f0e8802cd6388e1a8d0af0acf99\": container with ID starting with 039c3e7420ab20e9cbf41c4eb676940b4c9d8f0e8802cd6388e1a8d0af0acf99 not found: ID does not exist" containerID="039c3e7420ab20e9cbf41c4eb676940b4c9d8f0e8802cd6388e1a8d0af0acf99"
Sep 11 21:22:21.294431 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.294356    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"039c3e7420ab20e9cbf41c4eb676940b4c9d8f0e8802cd6388e1a8d0af0acf99"} err="failed to get container status \"039c3e7420ab20e9cbf41c4eb676940b4c9d8f0e8802cd6388e1a8d0af0acf99\": rpc error: code = NotFound desc = could not find container \"039c3e7420ab20e9cbf41c4eb676940b4c9d8f0e8802cd6388e1a8d0af0acf99\": container with ID starting with 039c3e7420ab20e9cbf41c4eb676940b4c9d8f0e8802cd6388e1a8d0af0acf99 not found: ID does not exist"
Sep 11 21:22:21.294431 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.294376    2578 scope.go:117] "RemoveContainer" containerID="b14fca34c124871c6e986f6387b4973222bd3789049ee4a85dd7f7122ee96418"
Sep 11 21:22:21.294621 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:22:21.294605    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b14fca34c124871c6e986f6387b4973222bd3789049ee4a85dd7f7122ee96418\": container with ID starting with b14fca34c124871c6e986f6387b4973222bd3789049ee4a85dd7f7122ee96418 not found: ID does not exist" containerID="b14fca34c124871c6e986f6387b4973222bd3789049ee4a85dd7f7122ee96418"
Sep 11 21:22:21.294665 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.294627    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b14fca34c124871c6e986f6387b4973222bd3789049ee4a85dd7f7122ee96418"} err="failed to get container status \"b14fca34c124871c6e986f6387b4973222bd3789049ee4a85dd7f7122ee96418\": rpc error: code = NotFound desc = could not find container \"b14fca34c124871c6e986f6387b4973222bd3789049ee4a85dd7f7122ee96418\": container with ID starting with b14fca34c124871c6e986f6387b4973222bd3789049ee4a85dd7f7122ee96418 not found: ID does not exist"
Sep 11 21:22:21.326923 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.326884    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-workload-certs\") pod \"da5b519a-a0f4-4107-ae63-8d70fe957d12\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") "
Sep 11 21:22:21.326923 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.326926    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-istio-envoy\") pod \"da5b519a-a0f4-4107-ae63-8d70fe957d12\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") "
Sep 11 21:22:21.327183 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.326943    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-workload-socket\") pod \"da5b519a-a0f4-4107-ae63-8d70fe957d12\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") "
Sep 11 21:22:21.327183 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.326968    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-istio-data\") pod \"da5b519a-a0f4-4107-ae63-8d70fe957d12\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") "
Sep 11 21:22:21.327183 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.327009    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da5b519a-a0f4-4107-ae63-8d70fe957d12-istio-token\") pod \"da5b519a-a0f4-4107-ae63-8d70fe957d12\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") "
Sep 11 21:22:21.327183 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.327037    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s86d\" (UniqueName: \"kubernetes.io/projected/da5b519a-a0f4-4107-ae63-8d70fe957d12-kube-api-access-4s86d\") pod \"da5b519a-a0f4-4107-ae63-8d70fe957d12\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") "
Sep 11 21:22:21.327183 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.327086    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da5b519a-a0f4-4107-ae63-8d70fe957d12-istio-podinfo\") pod \"da5b519a-a0f4-4107-ae63-8d70fe957d12\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") "
Sep 11 21:22:21.327183 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.327109    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da5b519a-a0f4-4107-ae63-8d70fe957d12-istiod-ca-cert\") pod \"da5b519a-a0f4-4107-ae63-8d70fe957d12\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") "
Sep 11 21:22:21.327496 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.327176    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-credential-socket\") pod \"da5b519a-a0f4-4107-ae63-8d70fe957d12\" (UID: \"da5b519a-a0f4-4107-ae63-8d70fe957d12\") "
Sep 11 21:22:21.327496 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.327214    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "da5b519a-a0f4-4107-ae63-8d70fe957d12" (UID: "da5b519a-a0f4-4107-ae63-8d70fe957d12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:22:21.327496 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.327266    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "da5b519a-a0f4-4107-ae63-8d70fe957d12" (UID: "da5b519a-a0f4-4107-ae63-8d70fe957d12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:22:21.327496 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.327390    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-istio-data" (OuterVolumeSpecName: "istio-data") pod "da5b519a-a0f4-4107-ae63-8d70fe957d12" (UID: "da5b519a-a0f4-4107-ae63-8d70fe957d12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:22:21.327496 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.327493    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-workload-certs\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:21.327718 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.327512    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-workload-socket\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:21.327718 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.327527    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-istio-data\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:21.327718 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.327616    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "da5b519a-a0f4-4107-ae63-8d70fe957d12" (UID: "da5b519a-a0f4-4107-ae63-8d70fe957d12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:22:21.327718 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.327637    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da5b519a-a0f4-4107-ae63-8d70fe957d12-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "da5b519a-a0f4-4107-ae63-8d70fe957d12" (UID: "da5b519a-a0f4-4107-ae63-8d70fe957d12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:22:21.329326 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.329278    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da5b519a-a0f4-4107-ae63-8d70fe957d12-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "da5b519a-a0f4-4107-ae63-8d70fe957d12" (UID: "da5b519a-a0f4-4107-ae63-8d70fe957d12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:22:21.329450 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.329326    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da5b519a-a0f4-4107-ae63-8d70fe957d12-istio-token" (OuterVolumeSpecName: "istio-token") pod "da5b519a-a0f4-4107-ae63-8d70fe957d12" (UID: "da5b519a-a0f4-4107-ae63-8d70fe957d12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:22:21.329567 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.329547    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "da5b519a-a0f4-4107-ae63-8d70fe957d12" (UID: "da5b519a-a0f4-4107-ae63-8d70fe957d12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:22:21.329626 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.329592    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da5b519a-a0f4-4107-ae63-8d70fe957d12-kube-api-access-4s86d" (OuterVolumeSpecName: "kube-api-access-4s86d") pod "da5b519a-a0f4-4107-ae63-8d70fe957d12" (UID: "da5b519a-a0f4-4107-ae63-8d70fe957d12"). InnerVolumeSpecName "kube-api-access-4s86d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:22:21.428530 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.428489    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-istio-envoy\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:21.428530 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.428520    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da5b519a-a0f4-4107-ae63-8d70fe957d12-istio-token\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:21.428754 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.428554    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s86d\" (UniqueName: \"kubernetes.io/projected/da5b519a-a0f4-4107-ae63-8d70fe957d12-kube-api-access-4s86d\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:21.428754 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.428564    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da5b519a-a0f4-4107-ae63-8d70fe957d12-istio-podinfo\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:21.428754 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.428573    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da5b519a-a0f4-4107-ae63-8d70fe957d12-istiod-ca-cert\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:21.428754 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:21.428583    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5b519a-a0f4-4107-ae63-8d70fe957d12-credential-socket\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:22.282172 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:22.282081    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"
Sep 11 21:22:22.299474 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:22.299432    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"]
Sep 11 21:22:22.303025 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:22.302999    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-7ae28e-openshift-default-8958845b4-4xnsb"]
Sep 11 21:22:23.927076 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:23.927030    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da5b519a-a0f4-4107-ae63-8d70fe957d12" path="/var/lib/kubelet/pods/da5b519a-a0f4-4107-ae63-8d70fe957d12/volumes"
Sep 11 21:22:26.875857 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.875805    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw"]
Sep 11 21:22:26.876284 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.876165    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e53c31c5-a767-4f11-8da8-559476ad3689" containerName="istio-proxy"
Sep 11 21:22:26.876284 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.876177    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e53c31c5-a767-4f11-8da8-559476ad3689" containerName="istio-proxy"
Sep 11 21:22:26.876284 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.876188    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="615af3ab-3d11-4995-83ba-cffd0d8479a2" containerName="main"
Sep 11 21:22:26.876284 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.876193    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="615af3ab-3d11-4995-83ba-cffd0d8479a2" containerName="main"
Sep 11 21:22:26.876284 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.876203    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da5b519a-a0f4-4107-ae63-8d70fe957d12" containerName="istio-proxy"
Sep 11 21:22:26.876284 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.876208    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="da5b519a-a0f4-4107-ae63-8d70fe957d12" containerName="istio-proxy"
Sep 11 21:22:26.876284 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.876214    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36ed1693-9136-461b-8bc7-7b66d5ee3ee3" containerName="payload-processing"
Sep 11 21:22:26.876284 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.876219    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="36ed1693-9136-461b-8bc7-7b66d5ee3ee3" containerName="payload-processing"
Sep 11 21:22:26.876284 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.876227    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e53c31c5-a767-4f11-8da8-559476ad3689" containerName="istio-proxy"
Sep 11 21:22:26.876284 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.876232    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e53c31c5-a767-4f11-8da8-559476ad3689" containerName="istio-proxy"
Sep 11 21:22:26.876284 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.876241    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da5b519a-a0f4-4107-ae63-8d70fe957d12" containerName="istio-proxy"
Sep 11 21:22:26.876284 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.876245    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="da5b519a-a0f4-4107-ae63-8d70fe957d12" containerName="istio-proxy"
Sep 11 21:22:26.876691 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.876298    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="36ed1693-9136-461b-8bc7-7b66d5ee3ee3" containerName="payload-processing"
Sep 11 21:22:26.876691 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.876306    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e53c31c5-a767-4f11-8da8-559476ad3689" containerName="istio-proxy"
Sep 11 21:22:26.876691 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.876312    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="615af3ab-3d11-4995-83ba-cffd0d8479a2" containerName="main"
Sep 11 21:22:26.876691 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.876317    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="da5b519a-a0f4-4107-ae63-8d70fe957d12" containerName="istio-proxy"
Sep 11 21:22:26.876691 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.876323    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="da5b519a-a0f4-4107-ae63-8d70fe957d12" containerName="istio-proxy"
Sep 11 21:22:26.876691 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.876330    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e53c31c5-a767-4f11-8da8-559476ad3689" containerName="istio-proxy"
Sep 11 21:22:26.880934 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.880910    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw"
Sep 11 21:22:26.883983 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.883962    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-1117b2-dockercfg-jsr5h\""
Sep 11 21:22:26.884349 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.884332    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-1117b2\""
Sep 11 21:22:26.889287 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.889257    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5"]
Sep 11 21:22:26.892793 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.892768    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw"]
Sep 11 21:22:26.892942 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.892883    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5"
Sep 11 21:22:26.903154 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.903123    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5"]
Sep 11 21:22:26.976507 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.976465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9b6d\" (UniqueName: \"kubernetes.io/projected/c3178682-f664-4782-8d89-f683b3a69d2f-kube-api-access-c9b6d\") pod \"payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5\" (UID: \"c3178682-f664-4782-8d89-f683b3a69d2f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5"
Sep 11 21:22:26.976507 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.976516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5d9j\" (UniqueName: \"kubernetes.io/projected/46c779ff-b35c-4251-b874-7bdb2821cee7-kube-api-access-x5d9j\") pod \"payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw\" (UID: \"46c779ff-b35c-4251-b874-7bdb2821cee7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw"
Sep 11 21:22:26.976738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.976583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c779ff-b35c-4251-b874-7bdb2821cee7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw\" (UID: \"46c779ff-b35c-4251-b874-7bdb2821cee7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw"
Sep 11 21:22:26.976738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:26.976646    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3178682-f664-4782-8d89-f683b3a69d2f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5\" (UID: \"c3178682-f664-4782-8d89-f683b3a69d2f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5"
Sep 11 21:22:27.077981 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:27.077941    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3178682-f664-4782-8d89-f683b3a69d2f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5\" (UID: \"c3178682-f664-4782-8d89-f683b3a69d2f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5"
Sep 11 21:22:27.078190 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:27.077994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9b6d\" (UniqueName: \"kubernetes.io/projected/c3178682-f664-4782-8d89-f683b3a69d2f-kube-api-access-c9b6d\") pod \"payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5\" (UID: \"c3178682-f664-4782-8d89-f683b3a69d2f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5"
Sep 11 21:22:27.078190 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:27.078020    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5d9j\" (UniqueName: \"kubernetes.io/projected/46c779ff-b35c-4251-b874-7bdb2821cee7-kube-api-access-x5d9j\") pod \"payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw\" (UID: \"46c779ff-b35c-4251-b874-7bdb2821cee7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw"
Sep 11 21:22:27.078190 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:27.078049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c779ff-b35c-4251-b874-7bdb2821cee7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw\" (UID: \"46c779ff-b35c-4251-b874-7bdb2821cee7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw"
Sep 11 21:22:27.078757 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:27.078735    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3178682-f664-4782-8d89-f683b3a69d2f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5\" (UID: \"c3178682-f664-4782-8d89-f683b3a69d2f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5"
Sep 11 21:22:27.078846 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:27.078754    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c779ff-b35c-4251-b874-7bdb2821cee7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw\" (UID: \"46c779ff-b35c-4251-b874-7bdb2821cee7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw"
Sep 11 21:22:27.086257 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:27.086231    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5d9j\" (UniqueName: \"kubernetes.io/projected/46c779ff-b35c-4251-b874-7bdb2821cee7-kube-api-access-x5d9j\") pod \"payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw\" (UID: \"46c779ff-b35c-4251-b874-7bdb2821cee7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw"
Sep 11 21:22:27.086372 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:27.086290    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9b6d\" (UniqueName: \"kubernetes.io/projected/c3178682-f664-4782-8d89-f683b3a69d2f-kube-api-access-c9b6d\") pod \"payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5\" (UID: \"c3178682-f664-4782-8d89-f683b3a69d2f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5"
Sep 11 21:22:27.192563 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:27.192517    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw"
Sep 11 21:22:27.204934 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:27.204902    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5"
Sep 11 21:22:27.471181 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:27.471093    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw"]
Sep 11 21:22:27.474681 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:22:27.474651    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c779ff_b35c_4251_b874_7bdb2821cee7.slice/crio-437083a8f1fccf464dd60c9af981cda17531e77bb351efd8ae6094ebf7d5a253 WatchSource:0}: Error finding container 437083a8f1fccf464dd60c9af981cda17531e77bb351efd8ae6094ebf7d5a253: Status 404 returned error can't find the container with id 437083a8f1fccf464dd60c9af981cda17531e77bb351efd8ae6094ebf7d5a253
Sep 11 21:22:27.487510 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:27.487478    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5"]
Sep 11 21:22:27.491417 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:22:27.491394    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3178682_f664_4782_8d89_f683b3a69d2f.slice/crio-a7e7a8352874678a9485548044d4beafc5d2fda250c11e40f14004381dc3b4f3 WatchSource:0}: Error finding container a7e7a8352874678a9485548044d4beafc5d2fda250c11e40f14004381dc3b4f3: Status 404 returned error can't find the container with id a7e7a8352874678a9485548044d4beafc5d2fda250c11e40f14004381dc3b4f3
Sep 11 21:22:28.314513 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:28.314473    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5" event={"ID":"c3178682-f664-4782-8d89-f683b3a69d2f","Type":"ContainerStarted","Data":"b003a1d88edcd4b99bbf308e4c48e791b5a6b44942eed558be9091f3d4349183"}
Sep 11 21:22:28.315038 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:28.314537    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5" event={"ID":"c3178682-f664-4782-8d89-f683b3a69d2f","Type":"ContainerStarted","Data":"a7e7a8352874678a9485548044d4beafc5d2fda250c11e40f14004381dc3b4f3"}
Sep 11 21:22:28.315038 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:28.314663    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5"
Sep 11 21:22:28.316235 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:28.316210    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw" event={"ID":"46c779ff-b35c-4251-b874-7bdb2821cee7","Type":"ContainerStarted","Data":"bbd98e1f654764bffc479a379c7f1546820ae4fbc5b76311456d7783deca5f1e"}
Sep 11 21:22:28.316364 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:28.316250    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw" event={"ID":"46c779ff-b35c-4251-b874-7bdb2821cee7","Type":"ContainerStarted","Data":"437083a8f1fccf464dd60c9af981cda17531e77bb351efd8ae6094ebf7d5a253"}
Sep 11 21:22:28.316364 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:28.316274    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw"
Sep 11 21:22:28.349178 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:28.349119    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw" podStartSLOduration=2.349099771 podStartE2EDuration="2.349099771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:22:28.347944088 +0000 UTC m=+1015.051913394" watchObservedRunningTime="2026-09-11 21:22:28.349099771 +0000 UTC m=+1015.053069061"
Sep 11 21:22:28.349381 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:28.349228    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5" podStartSLOduration=2.349219724 podStartE2EDuration="2.349219724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:22:28.331996755 +0000 UTC m=+1015.035966048" watchObservedRunningTime="2026-09-11 21:22:28.349219724 +0000 UTC m=+1015.053189015"
Sep 11 21:22:30.521313 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:30.521274    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:22:39.323255 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:39.323220    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw"
Sep 11 21:22:39.323711 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:39.323395    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5"
Sep 11 21:22:41.822700 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:41.822660    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"]
Sep 11 21:22:41.827381 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:41.827363    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:22:41.829648 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:41.829610    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ae7b6a\"/\"openshift-service-ca.crt\""
Sep 11 21:22:41.830426 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:41.830409    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ae7b6a\"/\"noten-model-ae7b6a-kserve-self-signed-certs\""
Sep 11 21:22:41.830533 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:41.830428    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ae7b6a\"/\"kube-root-ca.crt\""
Sep 11 21:22:41.830533 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:41.830440    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ae7b6a\"/\"default-dockercfg-46tt6\""
Sep 11 21:22:41.839800 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:41.839775    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"]
Sep 11 21:22:41.909287 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:41.909253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-model-cache\") pod \"noten-model-ae7b6a-kserve-75547c8b4c-msq22\" (UID: \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\") " pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:22:41.909453 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:41.909302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-tls-certs\") pod \"noten-model-ae7b6a-kserve-75547c8b4c-msq22\" (UID: \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\") " pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:22:41.909453 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:41.909328    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-home\") pod \"noten-model-ae7b6a-kserve-75547c8b4c-msq22\" (UID: \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\") " pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:22:41.909453 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:41.909406    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-dshm\") pod \"noten-model-ae7b6a-kserve-75547c8b4c-msq22\" (UID: \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\") " pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:22:41.909453 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:41.909433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-tmp-dir\") pod \"noten-model-ae7b6a-kserve-75547c8b4c-msq22\" (UID: \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\") " pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:22:41.909606 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:41.909460    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h2lr\" (UniqueName: \"kubernetes.io/projected/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-kube-api-access-6h2lr\") pod \"noten-model-ae7b6a-kserve-75547c8b4c-msq22\" (UID: \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\") " pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:22:42.010516 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:42.010465    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-home\") pod \"noten-model-ae7b6a-kserve-75547c8b4c-msq22\" (UID: \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\") " pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:22:42.010516 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:42.010521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-dshm\") pod \"noten-model-ae7b6a-kserve-75547c8b4c-msq22\" (UID: \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\") " pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:22:42.010737 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:42.010644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-tmp-dir\") pod \"noten-model-ae7b6a-kserve-75547c8b4c-msq22\" (UID: \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\") " pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:22:42.010737 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:42.010699    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h2lr\" (UniqueName: \"kubernetes.io/projected/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-kube-api-access-6h2lr\") pod \"noten-model-ae7b6a-kserve-75547c8b4c-msq22\" (UID: \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\") " pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:22:42.010843 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:42.010773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-model-cache\") pod \"noten-model-ae7b6a-kserve-75547c8b4c-msq22\" (UID: \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\") " pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:22:42.010900 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:42.010864    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-tls-certs\") pod \"noten-model-ae7b6a-kserve-75547c8b4c-msq22\" (UID: \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\") " pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:22:42.010997 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:42.010974    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-home\") pod \"noten-model-ae7b6a-kserve-75547c8b4c-msq22\" (UID: \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\") " pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:22:42.011064 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:42.011025    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-tmp-dir\") pod \"noten-model-ae7b6a-kserve-75547c8b4c-msq22\" (UID: \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\") " pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:22:42.011194 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:42.011170    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-model-cache\") pod \"noten-model-ae7b6a-kserve-75547c8b4c-msq22\" (UID: \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\") " pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:22:42.012749 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:42.012731    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-dshm\") pod \"noten-model-ae7b6a-kserve-75547c8b4c-msq22\" (UID: \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\") " pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:22:42.013158 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:42.013139    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-tls-certs\") pod \"noten-model-ae7b6a-kserve-75547c8b4c-msq22\" (UID: \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\") " pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:22:42.018120 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:42.018098    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h2lr\" (UniqueName: \"kubernetes.io/projected/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-kube-api-access-6h2lr\") pod \"noten-model-ae7b6a-kserve-75547c8b4c-msq22\" (UID: \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\") " pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:22:42.136311 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:42.136218    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:22:42.408884 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:42.408792    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"]
Sep 11 21:22:42.411061 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:22:42.411030    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae0eb382_a5bd_47e9_9bd0_b9b9bb4da95d.slice/crio-6ccd7265daa10d64f4db32486e805118d4333b0e78af15d491c867f3146613c3 WatchSource:0}: Error finding container 6ccd7265daa10d64f4db32486e805118d4333b0e78af15d491c867f3146613c3: Status 404 returned error can't find the container with id 6ccd7265daa10d64f4db32486e805118d4333b0e78af15d491c867f3146613c3
Sep 11 21:22:43.395840 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:43.395782    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22" event={"ID":"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d","Type":"ContainerStarted","Data":"f80278608704f6c33eacac8bbeb1f57db1bd99caf9cba089b2f0d873644c5013"}
Sep 11 21:22:43.395840 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:43.395843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22" event={"ID":"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d","Type":"ContainerStarted","Data":"6ccd7265daa10d64f4db32486e805118d4333b0e78af15d491c867f3146613c3"}
Sep 11 21:22:43.415695 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:43.415646    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22" podStartSLOduration=2.415630087 podStartE2EDuration="2.415630087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:22:43.414387886 +0000 UTC m=+1030.118363666" watchObservedRunningTime="2026-09-11 21:22:43.415630087 +0000 UTC m=+1030.119599377"
Sep 11 21:22:52.099517 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:52.099483    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"]
Sep 11 21:22:52.137127 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:52.137087    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:22:52.137127 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:52.137138    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:22:52.149719 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:52.149685    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:22:52.446618 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:52.446586    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:22:53.151203 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:53.151166    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"]
Sep 11 21:22:53.609606 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:22:53.609563    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-self-signed-certs: secret "noten-model-ae7b6a-kserve-self-signed-certs" not found
Sep 11 21:22:53.609873 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:22:53.609654    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-tls-certs podName:ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d nodeName:}" failed. No retries permitted until 2026-09-11 21:22:54.10963741 +0000 UTC m=+1040.813606694 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-tls-certs") pod "noten-model-ae7b6a-kserve-75547c8b4c-msq22" (UID: "ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d") : secret "noten-model-ae7b6a-kserve-self-signed-certs" not found
Sep 11 21:22:54.114710 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:22:54.114679    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-self-signed-certs: secret "noten-model-ae7b6a-kserve-self-signed-certs" not found
Sep 11 21:22:54.114904 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:22:54.114752    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-tls-certs podName:ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d nodeName:}" failed. No retries permitted until 2026-09-11 21:22:55.114734035 +0000 UTC m=+1041.818703317 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-tls-certs") pod "noten-model-ae7b6a-kserve-75547c8b4c-msq22" (UID: "ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d") : secret "noten-model-ae7b6a-kserve-self-signed-certs" not found
Sep 11 21:22:54.500014 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:54.499969    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"]
Sep 11 21:22:54.620548 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:54.620509    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26462a-tstq6"]
Sep 11 21:22:54.624061 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:54.624043    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26462a-tstq6"
Sep 11 21:22:54.626394 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:54.626365    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:22:54.626516 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:54.626422    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfm9g\""
Sep 11 21:22:54.632807 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:54.632778    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26462a-tstq6"]
Sep 11 21:22:54.822426 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:54.822332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f56870f-2141-48ed-b20f-ac70965287ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-26462a-tstq6\" (UID: \"8f56870f-2141-48ed-b20f-ac70965287ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26462a-tstq6"
Sep 11 21:22:54.923900 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:54.923848    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f56870f-2141-48ed-b20f-ac70965287ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-26462a-tstq6\" (UID: \"8f56870f-2141-48ed-b20f-ac70965287ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26462a-tstq6"
Sep 11 21:22:54.924499 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:54.924479    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f56870f-2141-48ed-b20f-ac70965287ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-26462a-tstq6\" (UID: \"8f56870f-2141-48ed-b20f-ac70965287ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26462a-tstq6"
Sep 11 21:22:54.936777 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:54.936745    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26462a-tstq6"
Sep 11 21:22:55.126260 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:22:55.126160    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-self-signed-certs: secret "noten-model-ae7b6a-kserve-self-signed-certs" not found
Sep 11 21:22:55.126260 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:22:55.126242    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-tls-certs podName:ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d nodeName:}" failed. No retries permitted until 2026-09-11 21:22:57.126219521 +0000 UTC m=+1043.830188789 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-tls-certs") pod "noten-model-ae7b6a-kserve-75547c8b4c-msq22" (UID: "ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d") : secret "noten-model-ae7b6a-kserve-self-signed-certs" not found
Sep 11 21:22:55.209614 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:55.209583    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26462a-tstq6"]
Sep 11 21:22:55.213719 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:22:55.213687    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f56870f_2141_48ed_b20f_ac70965287ac.slice/crio-e7734aca57476316c6d73819e9065dd02080ccba8a3b04d1eb518d76d7a6337c WatchSource:0}: Error finding container e7734aca57476316c6d73819e9065dd02080ccba8a3b04d1eb518d76d7a6337c: Status 404 returned error can't find the container with id e7734aca57476316c6d73819e9065dd02080ccba8a3b04d1eb518d76d7a6337c
Sep 11 21:22:55.452137 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:55.452108    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26462a-tstq6" event={"ID":"8f56870f-2141-48ed-b20f-ac70965287ac","Type":"ContainerStarted","Data":"e7734aca57476316c6d73819e9065dd02080ccba8a3b04d1eb518d76d7a6337c"}
Sep 11 21:22:56.458606 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:56.458577    2578 generic.go:356] "Generic (PLEG): container finished" podID="8f56870f-2141-48ed-b20f-ac70965287ac" containerID="ae306c442fef39b03e3b8d6142d23f2bd57072da41de1e7056657fb4df8bb2aa" exitCode=0
Sep 11 21:22:56.459013 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:56.458665    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26462a-tstq6" event={"ID":"8f56870f-2141-48ed-b20f-ac70965287ac","Type":"ContainerDied","Data":"ae306c442fef39b03e3b8d6142d23f2bd57072da41de1e7056657fb4df8bb2aa"}
Sep 11 21:22:56.923388 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:56.923361    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f"]
Sep 11 21:22:56.926642 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:56.926609    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f"
Sep 11 21:22:56.928890 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:56.928863    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-55fb09-dockercfg-l85cm\""
Sep 11 21:22:56.929036 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:56.928896    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-55fb09\""
Sep 11 21:22:56.940790 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:56.940763    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rz5g\" (UniqueName: \"kubernetes.io/projected/2333549d-7185-4098-94f6-262d7e8dcb39-kube-api-access-4rz5g\") pod \"payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f\" (UID: \"2333549d-7185-4098-94f6-262d7e8dcb39\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f"
Sep 11 21:22:56.940967 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:56.940843    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2333549d-7185-4098-94f6-262d7e8dcb39-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f\" (UID: \"2333549d-7185-4098-94f6-262d7e8dcb39\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f"
Sep 11 21:22:56.941751 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:56.941732    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f"]
Sep 11 21:22:57.042129 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:57.042094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rz5g\" (UniqueName: \"kubernetes.io/projected/2333549d-7185-4098-94f6-262d7e8dcb39-kube-api-access-4rz5g\") pod \"payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f\" (UID: \"2333549d-7185-4098-94f6-262d7e8dcb39\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f"
Sep 11 21:22:57.042308 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:57.042147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2333549d-7185-4098-94f6-262d7e8dcb39-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f\" (UID: \"2333549d-7185-4098-94f6-262d7e8dcb39\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f"
Sep 11 21:22:57.042973 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:57.042954    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2333549d-7185-4098-94f6-262d7e8dcb39-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f\" (UID: \"2333549d-7185-4098-94f6-262d7e8dcb39\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f"
Sep 11 21:22:57.050560 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:57.050535    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rz5g\" (UniqueName: \"kubernetes.io/projected/2333549d-7185-4098-94f6-262d7e8dcb39-kube-api-access-4rz5g\") pod \"payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f\" (UID: \"2333549d-7185-4098-94f6-262d7e8dcb39\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f"
Sep 11 21:22:57.142913 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:22:57.142878    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-self-signed-certs: secret "noten-model-ae7b6a-kserve-self-signed-certs" not found
Sep 11 21:22:57.143091 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:22:57.142949    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-tls-certs podName:ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d nodeName:}" failed. No retries permitted until 2026-09-11 21:23:01.142933201 +0000 UTC m=+1047.846902469 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-tls-certs") pod "noten-model-ae7b6a-kserve-75547c8b4c-msq22" (UID: "ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d") : secret "noten-model-ae7b6a-kserve-self-signed-certs" not found
Sep 11 21:22:57.237293 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:57.237186    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f"
Sep 11 21:22:57.513236 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:57.513203    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f"]
Sep 11 21:22:57.521396 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:22:57.521363    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2333549d_7185_4098_94f6_262d7e8dcb39.slice/crio-5784a82fdaac2357cdb403a97eb42d898c4e8c226837590da6bf7bfe001296c8 WatchSource:0}: Error finding container 5784a82fdaac2357cdb403a97eb42d898c4e8c226837590da6bf7bfe001296c8: Status 404 returned error can't find the container with id 5784a82fdaac2357cdb403a97eb42d898c4e8c226837590da6bf7bfe001296c8
Sep 11 21:22:57.594646 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:57.594622    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26462a-tstq6"
Sep 11 21:22:57.647633 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:57.647595    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f56870f-2141-48ed-b20f-ac70965287ac-maas-api-service-ca\") pod \"8f56870f-2141-48ed-b20f-ac70965287ac\" (UID: \"8f56870f-2141-48ed-b20f-ac70965287ac\") "
Sep 11 21:22:57.648052 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:57.648030    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f56870f-2141-48ed-b20f-ac70965287ac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8f56870f-2141-48ed-b20f-ac70965287ac" (UID: "8f56870f-2141-48ed-b20f-ac70965287ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:22:57.715503 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:57.715469    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:22:57.749130 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:57.749099    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f56870f-2141-48ed-b20f-ac70965287ac-maas-api-service-ca\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:22:58.470240 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:58.470199    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f" event={"ID":"2333549d-7185-4098-94f6-262d7e8dcb39","Type":"ContainerStarted","Data":"7a5e4bd90ffe721d9ce653df13a980cfd6fa9fc45258f3107338791097dd7eee"}
Sep 11 21:22:58.470452 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:58.470264    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f" event={"ID":"2333549d-7185-4098-94f6-262d7e8dcb39","Type":"ContainerStarted","Data":"5784a82fdaac2357cdb403a97eb42d898c4e8c226837590da6bf7bfe001296c8"}
Sep 11 21:22:58.470452 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:58.470294    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f"
Sep 11 21:22:58.471593 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:58.471570    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26462a-tstq6"
Sep 11 21:22:58.471593 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:58.471584    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26462a-tstq6" event={"ID":"8f56870f-2141-48ed-b20f-ac70965287ac","Type":"ContainerDied","Data":"e7734aca57476316c6d73819e9065dd02080ccba8a3b04d1eb518d76d7a6337c"}
Sep 11 21:22:58.471765 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:58.471618    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7734aca57476316c6d73819e9065dd02080ccba8a3b04d1eb518d76d7a6337c"
Sep 11 21:22:58.492328 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:22:58.492273    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f" podStartSLOduration=2.492255782 podStartE2EDuration="2.492255782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:22:58.490350719 +0000 UTC m=+1045.194320010" watchObservedRunningTime="2026-09-11 21:22:58.492255782 +0000 UTC m=+1045.196225066"
Sep 11 21:23:01.180296 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:01.180251    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-self-signed-certs: secret "noten-model-ae7b6a-kserve-self-signed-certs" not found
Sep 11 21:23:01.180720 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:01.180333    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-tls-certs podName:ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d nodeName:}" failed. No retries permitted until 2026-09-11 21:23:09.180317041 +0000 UTC m=+1055.884286310 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-tls-certs") pod "noten-model-ae7b6a-kserve-75547c8b4c-msq22" (UID: "ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d") : secret "noten-model-ae7b6a-kserve-self-signed-certs" not found
Sep 11 21:23:05.754335 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:05.754297    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26462a-tstq6"]
Sep 11 21:23:05.756998 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:05.756971    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26462a-tstq6"]
Sep 11 21:23:05.927285 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:05.927243    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f56870f-2141-48ed-b20f-ac70965287ac" path="/var/lib/kubelet/pods/8f56870f-2141-48ed-b20f-ac70965287ac/volumes"
Sep 11 21:23:07.118689 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.118645    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds" podUID="5699b08a-6f3a-48b3-9f91-3d9cc291a336" containerName="main" containerID="cri-o://0f3e3acf7256d3aceaf19e7b009cbd86ad9c115ae32e926910055981d75e8191" gracePeriod=45
Sep 11 21:23:07.374596 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.374533    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:23:07.436160 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.436126    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlmxg\" (UniqueName: \"kubernetes.io/projected/5699b08a-6f3a-48b3-9f91-3d9cc291a336-kube-api-access-zlmxg\") pod \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\" (UID: \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\") "
Sep 11 21:23:07.436336 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.436197    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5699b08a-6f3a-48b3-9f91-3d9cc291a336-dshm\") pod \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\" (UID: \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\") "
Sep 11 21:23:07.436336 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.436232    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5699b08a-6f3a-48b3-9f91-3d9cc291a336-tmp-dir\") pod \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\" (UID: \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\") "
Sep 11 21:23:07.436336 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.436253    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5699b08a-6f3a-48b3-9f91-3d9cc291a336-home\") pod \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\" (UID: \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\") "
Sep 11 21:23:07.436336 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.436315    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5699b08a-6f3a-48b3-9f91-3d9cc291a336-tls-certs\") pod \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\" (UID: \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\") "
Sep 11 21:23:07.436523 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.436367    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5699b08a-6f3a-48b3-9f91-3d9cc291a336-model-cache\") pod \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\" (UID: \"5699b08a-6f3a-48b3-9f91-3d9cc291a336\") "
Sep 11 21:23:07.436605 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.436577    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5699b08a-6f3a-48b3-9f91-3d9cc291a336-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5699b08a-6f3a-48b3-9f91-3d9cc291a336" (UID: "5699b08a-6f3a-48b3-9f91-3d9cc291a336"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:23:07.436671 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.436592    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5699b08a-6f3a-48b3-9f91-3d9cc291a336-home" (OuterVolumeSpecName: "home") pod "5699b08a-6f3a-48b3-9f91-3d9cc291a336" (UID: "5699b08a-6f3a-48b3-9f91-3d9cc291a336"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:23:07.436729 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.436691    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5699b08a-6f3a-48b3-9f91-3d9cc291a336-tmp-dir\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:07.436729 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.436708    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5699b08a-6f3a-48b3-9f91-3d9cc291a336-home\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:07.436729 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.436706    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5699b08a-6f3a-48b3-9f91-3d9cc291a336-model-cache" (OuterVolumeSpecName: "model-cache") pod "5699b08a-6f3a-48b3-9f91-3d9cc291a336" (UID: "5699b08a-6f3a-48b3-9f91-3d9cc291a336"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:23:07.438693 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.438657    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5699b08a-6f3a-48b3-9f91-3d9cc291a336-dshm" (OuterVolumeSpecName: "dshm") pod "5699b08a-6f3a-48b3-9f91-3d9cc291a336" (UID: "5699b08a-6f3a-48b3-9f91-3d9cc291a336"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:23:07.438780 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.438737    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5699b08a-6f3a-48b3-9f91-3d9cc291a336-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5699b08a-6f3a-48b3-9f91-3d9cc291a336" (UID: "5699b08a-6f3a-48b3-9f91-3d9cc291a336"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:23:07.438780 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.438736    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5699b08a-6f3a-48b3-9f91-3d9cc291a336-kube-api-access-zlmxg" (OuterVolumeSpecName: "kube-api-access-zlmxg") pod "5699b08a-6f3a-48b3-9f91-3d9cc291a336" (UID: "5699b08a-6f3a-48b3-9f91-3d9cc291a336"). InnerVolumeSpecName "kube-api-access-zlmxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:23:07.513503 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.513472    2578 generic.go:356] "Generic (PLEG): container finished" podID="5699b08a-6f3a-48b3-9f91-3d9cc291a336" containerID="0f3e3acf7256d3aceaf19e7b009cbd86ad9c115ae32e926910055981d75e8191" exitCode=0
Sep 11 21:23:07.513677 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.513515    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"
Sep 11 21:23:07.513677 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.513553    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds" event={"ID":"5699b08a-6f3a-48b3-9f91-3d9cc291a336","Type":"ContainerDied","Data":"0f3e3acf7256d3aceaf19e7b009cbd86ad9c115ae32e926910055981d75e8191"}
Sep 11 21:23:07.513677 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.513600    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds" event={"ID":"5699b08a-6f3a-48b3-9f91-3d9cc291a336","Type":"ContainerDied","Data":"d51b122d99f07752a815fc26541319b5f234645fcdecce721be39ea8d26d2300"}
Sep 11 21:23:07.513677 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.513619    2578 scope.go:117] "RemoveContainer" containerID="0f3e3acf7256d3aceaf19e7b009cbd86ad9c115ae32e926910055981d75e8191"
Sep 11 21:23:07.522652 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.522632    2578 scope.go:117] "RemoveContainer" containerID="0f3e3acf7256d3aceaf19e7b009cbd86ad9c115ae32e926910055981d75e8191"
Sep 11 21:23:07.522972 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:07.522954    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f3e3acf7256d3aceaf19e7b009cbd86ad9c115ae32e926910055981d75e8191\": container with ID starting with 0f3e3acf7256d3aceaf19e7b009cbd86ad9c115ae32e926910055981d75e8191 not found: ID does not exist" containerID="0f3e3acf7256d3aceaf19e7b009cbd86ad9c115ae32e926910055981d75e8191"
Sep 11 21:23:07.523031 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.522981    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f3e3acf7256d3aceaf19e7b009cbd86ad9c115ae32e926910055981d75e8191"} err="failed to get container status \"0f3e3acf7256d3aceaf19e7b009cbd86ad9c115ae32e926910055981d75e8191\": rpc error: code = NotFound desc = could not find container \"0f3e3acf7256d3aceaf19e7b009cbd86ad9c115ae32e926910055981d75e8191\": container with ID starting with 0f3e3acf7256d3aceaf19e7b009cbd86ad9c115ae32e926910055981d75e8191 not found: ID does not exist"
Sep 11 21:23:07.534764 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.534736    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"]
Sep 11 21:23:07.536587 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.536562    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-26462a/test-model-26462a-kserve-68d8f6d567-w87ds"]
Sep 11 21:23:07.537195 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.537176    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlmxg\" (UniqueName: \"kubernetes.io/projected/5699b08a-6f3a-48b3-9f91-3d9cc291a336-kube-api-access-zlmxg\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:07.537248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.537198    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5699b08a-6f3a-48b3-9f91-3d9cc291a336-dshm\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:07.537248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.537209    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5699b08a-6f3a-48b3-9f91-3d9cc291a336-tls-certs\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:07.537248 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.537218    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5699b08a-6f3a-48b3-9f91-3d9cc291a336-model-cache\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:07.927280 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:07.927247    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5699b08a-6f3a-48b3-9f91-3d9cc291a336" path="/var/lib/kubelet/pods/5699b08a-6f3a-48b3-9f91-3d9cc291a336/volumes"
Sep 11 21:23:08.555783 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:08.555746    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7699-1e2b07"]
Sep 11 21:23:08.556211 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:08.556129    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f56870f-2141-48ed-b20f-ac70965287ac" containerName="revoke-keys"
Sep 11 21:23:08.556211 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:08.556142    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f56870f-2141-48ed-b20f-ac70965287ac" containerName="revoke-keys"
Sep 11 21:23:08.556211 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:08.556157    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5699b08a-6f3a-48b3-9f91-3d9cc291a336" containerName="main"
Sep 11 21:23:08.556211 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:08.556163    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5699b08a-6f3a-48b3-9f91-3d9cc291a336" containerName="main"
Sep 11 21:23:08.556365 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:08.556215    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5699b08a-6f3a-48b3-9f91-3d9cc291a336" containerName="main"
Sep 11 21:23:08.556365 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:08.556223    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f56870f-2141-48ed-b20f-ac70965287ac" containerName="revoke-keys"
Sep 11 21:23:08.560598 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:08.560578    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-1e2b07"
Sep 11 21:23:08.563381 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:08.563362    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6dwfw\""
Sep 11 21:23:08.574453 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:08.574424    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-1e2b07"]
Sep 11 21:23:08.648072 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:08.648030    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jxhm\" (UniqueName: \"kubernetes.io/projected/1f83316f-02cc-4f3e-bd63-2752d402a1cc-kube-api-access-5jxhm\") pod \"test-curl-7699-1e2b07\" (UID: \"1f83316f-02cc-4f3e-bd63-2752d402a1cc\") " pod="opendatahub/test-curl-7699-1e2b07"
Sep 11 21:23:08.749302 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:08.749259    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jxhm\" (UniqueName: \"kubernetes.io/projected/1f83316f-02cc-4f3e-bd63-2752d402a1cc-kube-api-access-5jxhm\") pod \"test-curl-7699-1e2b07\" (UID: \"1f83316f-02cc-4f3e-bd63-2752d402a1cc\") " pod="opendatahub/test-curl-7699-1e2b07"
Sep 11 21:23:08.758113 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:08.758077    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jxhm\" (UniqueName: \"kubernetes.io/projected/1f83316f-02cc-4f3e-bd63-2752d402a1cc-kube-api-access-5jxhm\") pod \"test-curl-7699-1e2b07\" (UID: \"1f83316f-02cc-4f3e-bd63-2752d402a1cc\") " pod="opendatahub/test-curl-7699-1e2b07"
Sep 11 21:23:08.870368 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:08.870248    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-1e2b07"
Sep 11 21:23:09.146355 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.146316    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-1e2b07"]
Sep 11 21:23:09.149374 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:23:09.149343    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f83316f_02cc_4f3e_bd63_2752d402a1cc.slice/crio-3af27d34a69862922823d61c9678950f401e5aa8a9f3a873f9beee20c18b93a1 WatchSource:0}: Error finding container 3af27d34a69862922823d61c9678950f401e5aa8a9f3a873f9beee20c18b93a1: Status 404 returned error can't find the container with id 3af27d34a69862922823d61c9678950f401e5aa8a9f3a873f9beee20c18b93a1
Sep 11 21:23:09.255273 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:09.255223    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-self-signed-certs: secret "noten-model-ae7b6a-kserve-self-signed-certs" not found
Sep 11 21:23:09.255462 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:09.255308    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-tls-certs podName:ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d nodeName:}" failed. No retries permitted until 2026-09-11 21:23:25.255293321 +0000 UTC m=+1071.959262588 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-tls-certs") pod "noten-model-ae7b6a-kserve-75547c8b4c-msq22" (UID: "ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d") : secret "noten-model-ae7b6a-kserve-self-signed-certs" not found
Sep 11 21:23:09.470027 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.469985    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22" podUID="ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d" containerName="main" containerID="cri-o://f80278608704f6c33eacac8bbeb1f57db1bd99caf9cba089b2f0d873644c5013" gracePeriod=45
Sep 11 21:23:09.479658 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.479632    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f"
Sep 11 21:23:09.519206 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.519135    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t" podUID="70727ea4-aa9c-46c4-929e-d3947353fc60" containerName="main" containerID="cri-o://4890399575ddf779fb4cc40eaa07eea832c53c72fbc98cac8828dca7cdc28055" gracePeriod=45
Sep 11 21:23:09.526642 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.526599    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-1e2b07" event={"ID":"1f83316f-02cc-4f3e-bd63-2752d402a1cc","Type":"ContainerStarted","Data":"faee5f812011d51f0d279f7b6541e247ff46422b617d64712a0baf6fa25587ed"}
Sep 11 21:23:09.526769 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.526649    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-1e2b07" event={"ID":"1f83316f-02cc-4f3e-bd63-2752d402a1cc","Type":"ContainerStarted","Data":"3af27d34a69862922823d61c9678950f401e5aa8a9f3a873f9beee20c18b93a1"}
Sep 11 21:23:09.542391 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.542342    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7699-1e2b07" podStartSLOduration=1.54232388 podStartE2EDuration="1.54232388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:23:09.540813689 +0000 UTC m=+1056.244782982" watchObservedRunningTime="2026-09-11 21:23:09.54232388 +0000 UTC m=+1056.246293170"
Sep 11 21:23:09.792897 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.792869    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:23:09.795932 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.795909    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:23:09.861221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.861183    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-home\") pod \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\" (UID: \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\") "
Sep 11 21:23:09.861221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.861222    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-tmp-dir\") pod \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\" (UID: \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\") "
Sep 11 21:23:09.861437 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.861252    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6h2lr\" (UniqueName: \"kubernetes.io/projected/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-kube-api-access-6h2lr\") pod \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\" (UID: \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\") "
Sep 11 21:23:09.861437 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.861291    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70727ea4-aa9c-46c4-929e-d3947353fc60-tls-certs\") pod \"70727ea4-aa9c-46c4-929e-d3947353fc60\" (UID: \"70727ea4-aa9c-46c4-929e-d3947353fc60\") "
Sep 11 21:23:09.861437 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.861338    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-tls-certs\") pod \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\" (UID: \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\") "
Sep 11 21:23:09.861600 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.861505    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70727ea4-aa9c-46c4-929e-d3947353fc60-home\") pod \"70727ea4-aa9c-46c4-929e-d3947353fc60\" (UID: \"70727ea4-aa9c-46c4-929e-d3947353fc60\") "
Sep 11 21:23:09.861600 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.861514    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d" (UID: "ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:23:09.861600 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.861527    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-home" (OuterVolumeSpecName: "home") pod "ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d" (UID: "ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:23:09.861600 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.861564    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-dshm\") pod \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\" (UID: \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\") "
Sep 11 21:23:09.861600 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.861597    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlkmx\" (UniqueName: \"kubernetes.io/projected/70727ea4-aa9c-46c4-929e-d3947353fc60-kube-api-access-nlkmx\") pod \"70727ea4-aa9c-46c4-929e-d3947353fc60\" (UID: \"70727ea4-aa9c-46c4-929e-d3947353fc60\") "
Sep 11 21:23:09.861813 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.861625    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70727ea4-aa9c-46c4-929e-d3947353fc60-tmp-dir\") pod \"70727ea4-aa9c-46c4-929e-d3947353fc60\" (UID: \"70727ea4-aa9c-46c4-929e-d3947353fc60\") "
Sep 11 21:23:09.861813 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.861763    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-model-cache\") pod \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\" (UID: \"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d\") "
Sep 11 21:23:09.861813 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.861805    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70727ea4-aa9c-46c4-929e-d3947353fc60-home" (OuterVolumeSpecName: "home") pod "70727ea4-aa9c-46c4-929e-d3947353fc60" (UID: "70727ea4-aa9c-46c4-929e-d3947353fc60"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:23:09.861813 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.861841    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70727ea4-aa9c-46c4-929e-d3947353fc60-model-cache\") pod \"70727ea4-aa9c-46c4-929e-d3947353fc60\" (UID: \"70727ea4-aa9c-46c4-929e-d3947353fc60\") "
Sep 11 21:23:09.862032 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.861873    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70727ea4-aa9c-46c4-929e-d3947353fc60-dshm\") pod \"70727ea4-aa9c-46c4-929e-d3947353fc60\" (UID: \"70727ea4-aa9c-46c4-929e-d3947353fc60\") "
Sep 11 21:23:09.862032 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.862001    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d" (UID: "ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:23:09.862139 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.862127    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-model-cache\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:09.862196 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.862172    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-home\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:09.862196 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.862187    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-tmp-dir\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:09.862300 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.862200    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70727ea4-aa9c-46c4-929e-d3947353fc60-home\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:09.862300 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.862198    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70727ea4-aa9c-46c4-929e-d3947353fc60-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "70727ea4-aa9c-46c4-929e-d3947353fc60" (UID: "70727ea4-aa9c-46c4-929e-d3947353fc60"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:23:09.862407 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.862346    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70727ea4-aa9c-46c4-929e-d3947353fc60-model-cache" (OuterVolumeSpecName: "model-cache") pod "70727ea4-aa9c-46c4-929e-d3947353fc60" (UID: "70727ea4-aa9c-46c4-929e-d3947353fc60"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:23:09.864053 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.864022    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-dshm" (OuterVolumeSpecName: "dshm") pod "ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d" (UID: "ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:23:09.864053 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.864032    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-kube-api-access-6h2lr" (OuterVolumeSpecName: "kube-api-access-6h2lr") pod "ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d" (UID: "ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d"). InnerVolumeSpecName "kube-api-access-6h2lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:23:09.864450 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.864110    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d" (UID: "ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:23:09.864450 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.864169    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70727ea4-aa9c-46c4-929e-d3947353fc60-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70727ea4-aa9c-46c4-929e-d3947353fc60" (UID: "70727ea4-aa9c-46c4-929e-d3947353fc60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:23:09.864450 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.864412    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70727ea4-aa9c-46c4-929e-d3947353fc60-dshm" (OuterVolumeSpecName: "dshm") pod "70727ea4-aa9c-46c4-929e-d3947353fc60" (UID: "70727ea4-aa9c-46c4-929e-d3947353fc60"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:23:09.864741 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.864707    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70727ea4-aa9c-46c4-929e-d3947353fc60-kube-api-access-nlkmx" (OuterVolumeSpecName: "kube-api-access-nlkmx") pod "70727ea4-aa9c-46c4-929e-d3947353fc60" (UID: "70727ea4-aa9c-46c4-929e-d3947353fc60"). InnerVolumeSpecName "kube-api-access-nlkmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:23:09.963171 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.963133    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-dshm\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:09.963171 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.963167    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlkmx\" (UniqueName: \"kubernetes.io/projected/70727ea4-aa9c-46c4-929e-d3947353fc60-kube-api-access-nlkmx\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:09.963379 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.963180    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70727ea4-aa9c-46c4-929e-d3947353fc60-tmp-dir\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:09.963379 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.963194    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70727ea4-aa9c-46c4-929e-d3947353fc60-model-cache\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:09.963379 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.963204    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70727ea4-aa9c-46c4-929e-d3947353fc60-dshm\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:09.963379 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.963216    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h2lr\" (UniqueName: \"kubernetes.io/projected/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-kube-api-access-6h2lr\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:09.963379 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.963225    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70727ea4-aa9c-46c4-929e-d3947353fc60-tls-certs\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:09.963379 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:09.963236    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d-tls-certs\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:10.149458 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.149371    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7699-1e2b07"]
Sep 11 21:23:10.156191 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.156156    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7699-1e2b07"]
Sep 11 21:23:10.277393 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.277358    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7699-640f1a"]
Sep 11 21:23:10.277678 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.277666    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70727ea4-aa9c-46c4-929e-d3947353fc60" containerName="main"
Sep 11 21:23:10.277722 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.277680    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="70727ea4-aa9c-46c4-929e-d3947353fc60" containerName="main"
Sep 11 21:23:10.277722 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.277697    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f83316f-02cc-4f3e-bd63-2752d402a1cc" containerName="test-curl-7699-1e2b07"
Sep 11 21:23:10.277722 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.277702    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f83316f-02cc-4f3e-bd63-2752d402a1cc" containerName="test-curl-7699-1e2b07"
Sep 11 21:23:10.277722 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.277718    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d" containerName="main"
Sep 11 21:23:10.277722 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.277724    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d" containerName="main"
Sep 11 21:23:10.277897 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.277776    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d" containerName="main"
Sep 11 21:23:10.277897 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.277786    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="70727ea4-aa9c-46c4-929e-d3947353fc60" containerName="main"
Sep 11 21:23:10.277897 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.277793    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f83316f-02cc-4f3e-bd63-2752d402a1cc" containerName="test-curl-7699-1e2b07"
Sep 11 21:23:10.281531 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.281512    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-640f1a"
Sep 11 21:23:10.285560 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.285528    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-640f1a"]
Sep 11 21:23:10.366041 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.366002    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24nk8\" (UniqueName: \"kubernetes.io/projected/1c6c305f-13bc-4351-87aa-b2c8dd961797-kube-api-access-24nk8\") pod \"test-curl-7699-640f1a\" (UID: \"1c6c305f-13bc-4351-87aa-b2c8dd961797\") " pod="opendatahub/test-curl-7699-640f1a"
Sep 11 21:23:10.466428 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.466389    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24nk8\" (UniqueName: \"kubernetes.io/projected/1c6c305f-13bc-4351-87aa-b2c8dd961797-kube-api-access-24nk8\") pod \"test-curl-7699-640f1a\" (UID: \"1c6c305f-13bc-4351-87aa-b2c8dd961797\") " pod="opendatahub/test-curl-7699-640f1a"
Sep 11 21:23:10.474465 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.474444    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24nk8\" (UniqueName: \"kubernetes.io/projected/1c6c305f-13bc-4351-87aa-b2c8dd961797-kube-api-access-24nk8\") pod \"test-curl-7699-640f1a\" (UID: \"1c6c305f-13bc-4351-87aa-b2c8dd961797\") " pod="opendatahub/test-curl-7699-640f1a"
Sep 11 21:23:10.532873 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.532841    2578 generic.go:356] "Generic (PLEG): container finished" podID="70727ea4-aa9c-46c4-929e-d3947353fc60" containerID="4890399575ddf779fb4cc40eaa07eea832c53c72fbc98cac8828dca7cdc28055" exitCode=0
Sep 11 21:23:10.533030 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.532891    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"
Sep 11 21:23:10.533030 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.532931    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t" event={"ID":"70727ea4-aa9c-46c4-929e-d3947353fc60","Type":"ContainerDied","Data":"4890399575ddf779fb4cc40eaa07eea832c53c72fbc98cac8828dca7cdc28055"}
Sep 11 21:23:10.533030 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.532973    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t" event={"ID":"70727ea4-aa9c-46c4-929e-d3947353fc60","Type":"ContainerDied","Data":"f91692fde974af98a457b901dedb8e5c055138296e036abc6abe49a229660885"}
Sep 11 21:23:10.533030 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.532999    2578 scope.go:117] "RemoveContainer" containerID="4890399575ddf779fb4cc40eaa07eea832c53c72fbc98cac8828dca7cdc28055"
Sep 11 21:23:10.534378 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.534346    2578 generic.go:356] "Generic (PLEG): container finished" podID="ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d" containerID="f80278608704f6c33eacac8bbeb1f57db1bd99caf9cba089b2f0d873644c5013" exitCode=0
Sep 11 21:23:10.534476 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.534398    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"
Sep 11 21:23:10.534476 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.534432    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22" event={"ID":"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d","Type":"ContainerDied","Data":"f80278608704f6c33eacac8bbeb1f57db1bd99caf9cba089b2f0d873644c5013"}
Sep 11 21:23:10.534476 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.534458    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22" event={"ID":"ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d","Type":"ContainerDied","Data":"6ccd7265daa10d64f4db32486e805118d4333b0e78af15d491c867f3146613c3"}
Sep 11 21:23:10.534792 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:10.534724    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7699-1e2b07" container="test-curl-7699-1e2b07"
Sep 11 21:23:10.541901 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.541886    2578 scope.go:117] "RemoveContainer" containerID="4890399575ddf779fb4cc40eaa07eea832c53c72fbc98cac8828dca7cdc28055"
Sep 11 21:23:10.542204 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:10.542179    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4890399575ddf779fb4cc40eaa07eea832c53c72fbc98cac8828dca7cdc28055\": container with ID starting with 4890399575ddf779fb4cc40eaa07eea832c53c72fbc98cac8828dca7cdc28055 not found: ID does not exist" containerID="4890399575ddf779fb4cc40eaa07eea832c53c72fbc98cac8828dca7cdc28055"
Sep 11 21:23:10.542281 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.542204    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4890399575ddf779fb4cc40eaa07eea832c53c72fbc98cac8828dca7cdc28055"} err="failed to get container status \"4890399575ddf779fb4cc40eaa07eea832c53c72fbc98cac8828dca7cdc28055\": rpc error: code = NotFound desc = could not find container \"4890399575ddf779fb4cc40eaa07eea832c53c72fbc98cac8828dca7cdc28055\": container with ID starting with 4890399575ddf779fb4cc40eaa07eea832c53c72fbc98cac8828dca7cdc28055 not found: ID does not exist"
Sep 11 21:23:10.542281 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.542218    2578 scope.go:117] "RemoveContainer" containerID="f80278608704f6c33eacac8bbeb1f57db1bd99caf9cba089b2f0d873644c5013"
Sep 11 21:23:10.549744 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.549721    2578 scope.go:117] "RemoveContainer" containerID="f80278608704f6c33eacac8bbeb1f57db1bd99caf9cba089b2f0d873644c5013"
Sep 11 21:23:10.550081 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:10.550057    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f80278608704f6c33eacac8bbeb1f57db1bd99caf9cba089b2f0d873644c5013\": container with ID starting with f80278608704f6c33eacac8bbeb1f57db1bd99caf9cba089b2f0d873644c5013 not found: ID does not exist" containerID="f80278608704f6c33eacac8bbeb1f57db1bd99caf9cba089b2f0d873644c5013"
Sep 11 21:23:10.550165 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.550090    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f80278608704f6c33eacac8bbeb1f57db1bd99caf9cba089b2f0d873644c5013"} err="failed to get container status \"f80278608704f6c33eacac8bbeb1f57db1bd99caf9cba089b2f0d873644c5013\": rpc error: code = NotFound desc = could not find container \"f80278608704f6c33eacac8bbeb1f57db1bd99caf9cba089b2f0d873644c5013\": container with ID starting with f80278608704f6c33eacac8bbeb1f57db1bd99caf9cba089b2f0d873644c5013 not found: ID does not exist"
Sep 11 21:23:10.553189 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.553167    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"]
Sep 11 21:23:10.557267 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.557246    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-ae7b6a/noten-model-ae7b6a-kserve-75547c8b4c-msq22"]
Sep 11 21:23:10.566392 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.566368    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"]
Sep 11 21:23:10.569967 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.569943    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2b95f8/test-model-2b95f8-kserve-647857d89-m298t"]
Sep 11 21:23:10.593404 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.593371    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-640f1a"
Sep 11 21:23:10.863668 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:10.863588    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-640f1a"]
Sep 11 21:23:10.867928 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:23:10.867897    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c6c305f_13bc_4351_87aa_b2c8dd961797.slice/crio-3c833f5d9c106b4465f03d5e947dacc94ded85dc4cfda15e8dc6fad98bbb9e63 WatchSource:0}: Error finding container 3c833f5d9c106b4465f03d5e947dacc94ded85dc4cfda15e8dc6fad98bbb9e63: Status 404 returned error can't find the container with id 3c833f5d9c106b4465f03d5e947dacc94ded85dc4cfda15e8dc6fad98bbb9e63
Sep 11 21:23:11.546860 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:11.546800    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-640f1a" event={"ID":"1c6c305f-13bc-4351-87aa-b2c8dd961797","Type":"ContainerStarted","Data":"f4c9514117af28c40e144d9577586bf842647987a194c10a4c7de4b4fb7eea2d"}
Sep 11 21:23:11.547063 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:11.546874    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-640f1a" event={"ID":"1c6c305f-13bc-4351-87aa-b2c8dd961797","Type":"ContainerStarted","Data":"3c833f5d9c106b4465f03d5e947dacc94ded85dc4cfda15e8dc6fad98bbb9e63"}
Sep 11 21:23:11.547770 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:11.547743    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7699-1e2b07" podUID="1f83316f-02cc-4f3e-bd63-2752d402a1cc" containerName="test-curl-7699-1e2b07" containerID="cri-o://faee5f812011d51f0d279f7b6541e247ff46422b617d64712a0baf6fa25587ed" gracePeriod=2
Sep 11 21:23:11.561414 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:11.561357    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7699-640f1a" podStartSLOduration=1.56133917 podStartE2EDuration="1.56133917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:23:11.56063437 +0000 UTC m=+1058.264603666" watchObservedRunningTime="2026-09-11 21:23:11.56133917 +0000 UTC m=+1058.265308460"
Sep 11 21:23:11.562518 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:11.562490    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7699-1e2b07\" is forbidden: User \"system:node:ip-10-0-142-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-224.ec2.internal' and this object" podUID="1f83316f-02cc-4f3e-bd63-2752d402a1cc" pod="opendatahub/test-curl-7699-1e2b07"
Sep 11 21:23:11.927489 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:11.927456    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70727ea4-aa9c-46c4-929e-d3947353fc60" path="/var/lib/kubelet/pods/70727ea4-aa9c-46c4-929e-d3947353fc60/volumes"
Sep 11 21:23:11.927877 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:11.927842    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d" path="/var/lib/kubelet/pods/ae0eb382-a5bd-47e9-9bd0-b9b9bb4da95d/volumes"
Sep 11 21:23:12.221642 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:12.221553    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7699-640f1a"]
Sep 11 21:23:12.230809 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:12.230779    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7699-640f1a"]
Sep 11 21:23:12.301282 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:12.301245    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:23:12.365265 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:12.365224    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7699-3ca33a"]
Sep 11 21:23:12.365641 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:12.365623    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c6c305f-13bc-4351-87aa-b2c8dd961797" containerName="test-curl-7699-640f1a"
Sep 11 21:23:12.365726 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:12.365644    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c6c305f-13bc-4351-87aa-b2c8dd961797" containerName="test-curl-7699-640f1a"
Sep 11 21:23:12.365781 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:12.365726    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c6c305f-13bc-4351-87aa-b2c8dd961797" containerName="test-curl-7699-640f1a"
Sep 11 21:23:12.369814 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:12.369792    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-3ca33a"
Sep 11 21:23:12.379281 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:12.379252    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-3ca33a"]
Sep 11 21:23:12.488095 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:12.487994    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-976pt\" (UniqueName: \"kubernetes.io/projected/7b677240-2aa5-48b8-b058-fe834cef3e4d-kube-api-access-976pt\") pod \"test-curl-7699-3ca33a\" (UID: \"7b677240-2aa5-48b8-b058-fe834cef3e4d\") " pod="opendatahub/test-curl-7699-3ca33a"
Sep 11 21:23:12.552452 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:12.552408    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7699-640f1a" container="test-curl-7699-640f1a"
Sep 11 21:23:12.589081 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:12.589044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-976pt\" (UniqueName: \"kubernetes.io/projected/7b677240-2aa5-48b8-b058-fe834cef3e4d-kube-api-access-976pt\") pod \"test-curl-7699-3ca33a\" (UID: \"7b677240-2aa5-48b8-b058-fe834cef3e4d\") " pod="opendatahub/test-curl-7699-3ca33a"
Sep 11 21:23:12.597552 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:12.597521    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-976pt\" (UniqueName: \"kubernetes.io/projected/7b677240-2aa5-48b8-b058-fe834cef3e4d-kube-api-access-976pt\") pod \"test-curl-7699-3ca33a\" (UID: \"7b677240-2aa5-48b8-b058-fe834cef3e4d\") " pod="opendatahub/test-curl-7699-3ca33a"
Sep 11 21:23:12.680725 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:12.680684    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-3ca33a"
Sep 11 21:23:12.956572 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:12.956540    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-3ca33a"]
Sep 11 21:23:12.959609 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:23:12.959582    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b677240_2aa5_48b8_b058_fe834cef3e4d.slice/crio-cc6a5199fdf9e88e20c77fe1d13f45ca8470ecc1381b6501f8a3163bbcc2605b WatchSource:0}: Error finding container cc6a5199fdf9e88e20c77fe1d13f45ca8470ecc1381b6501f8a3163bbcc2605b: Status 404 returned error can't find the container with id cc6a5199fdf9e88e20c77fe1d13f45ca8470ecc1381b6501f8a3163bbcc2605b
Sep 11 21:23:13.558724 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:13.558689    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-3ca33a" event={"ID":"7b677240-2aa5-48b8-b058-fe834cef3e4d","Type":"ContainerStarted","Data":"f0b2e48a31e9edecf957cee7384956ba66710c7fe688bf6fc6fb5c6e982e82b8"}
Sep 11 21:23:13.558911 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:13.558738    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-3ca33a" event={"ID":"7b677240-2aa5-48b8-b058-fe834cef3e4d","Type":"ContainerStarted","Data":"cc6a5199fdf9e88e20c77fe1d13f45ca8470ecc1381b6501f8a3163bbcc2605b"}
Sep 11 21:23:13.558911 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:13.558738    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7699-640f1a" podUID="1c6c305f-13bc-4351-87aa-b2c8dd961797" containerName="test-curl-7699-640f1a" containerID="cri-o://f4c9514117af28c40e144d9577586bf842647987a194c10a4c7de4b4fb7eea2d" gracePeriod=2
Sep 11 21:23:13.579010 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:13.578956    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7699-3ca33a" podStartSLOduration=1.578936103 podStartE2EDuration="1.578936103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:23:13.577957023 +0000 UTC m=+1060.281926315" watchObservedRunningTime="2026-09-11 21:23:13.578936103 +0000 UTC m=+1060.282905390"
Sep 11 21:23:13.582194 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:13.582160    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7699-640f1a\" is forbidden: User \"system:node:ip-10-0-142-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-224.ec2.internal' and this object" podUID="1c6c305f-13bc-4351-87aa-b2c8dd961797" pod="opendatahub/test-curl-7699-640f1a"
Sep 11 21:23:13.584687 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:13.584661    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7699-640f1a\" is forbidden: User \"system:node:ip-10-0-142-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-224.ec2.internal' and this object" podUID="1c6c305f-13bc-4351-87aa-b2c8dd961797" pod="opendatahub/test-curl-7699-640f1a"
Sep 11 21:23:13.719582 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:13.719556    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-1e2b07"
Sep 11 21:23:13.723736 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:13.723705    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7699-640f1a\" is forbidden: User \"system:node:ip-10-0-142-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-224.ec2.internal' and this object" podUID="1c6c305f-13bc-4351-87aa-b2c8dd961797" pod="opendatahub/test-curl-7699-640f1a"
Sep 11 21:23:13.800354 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:13.800316    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jxhm\" (UniqueName: \"kubernetes.io/projected/1f83316f-02cc-4f3e-bd63-2752d402a1cc-kube-api-access-5jxhm\") pod \"1f83316f-02cc-4f3e-bd63-2752d402a1cc\" (UID: \"1f83316f-02cc-4f3e-bd63-2752d402a1cc\") "
Sep 11 21:23:13.802463 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:13.802390    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f83316f-02cc-4f3e-bd63-2752d402a1cc-kube-api-access-5jxhm" (OuterVolumeSpecName: "kube-api-access-5jxhm") pod "1f83316f-02cc-4f3e-bd63-2752d402a1cc" (UID: "1f83316f-02cc-4f3e-bd63-2752d402a1cc"). InnerVolumeSpecName "kube-api-access-5jxhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:23:13.901471 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:13.901438    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jxhm\" (UniqueName: \"kubernetes.io/projected/1f83316f-02cc-4f3e-bd63-2752d402a1cc-kube-api-access-5jxhm\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:13.927051 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:13.927019    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f83316f-02cc-4f3e-bd63-2752d402a1cc" path="/var/lib/kubelet/pods/1f83316f-02cc-4f3e-bd63-2752d402a1cc/volumes"
Sep 11 21:23:13.927276 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:13.927253    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7699-640f1a\" is forbidden: User \"system:node:ip-10-0-142-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-224.ec2.internal' and this object" podUID="1c6c305f-13bc-4351-87aa-b2c8dd961797" pod="opendatahub/test-curl-7699-640f1a"
Sep 11 21:23:14.338838 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:14.338740    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7699-3ca33a"]
Sep 11 21:23:14.346136 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:14.346099    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7699-3ca33a"]
Sep 11 21:23:14.473006 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:14.472972    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7699-6eef12"]
Sep 11 21:23:14.473309 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:14.473296    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b677240-2aa5-48b8-b058-fe834cef3e4d" containerName="test-curl-7699-3ca33a"
Sep 11 21:23:14.473355 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:14.473311    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b677240-2aa5-48b8-b058-fe834cef3e4d" containerName="test-curl-7699-3ca33a"
Sep 11 21:23:14.473387 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:14.473378    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b677240-2aa5-48b8-b058-fe834cef3e4d" containerName="test-curl-7699-3ca33a"
Sep 11 21:23:14.476329 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:14.476313    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-6eef12"
Sep 11 21:23:14.480970 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:14.480944    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-6eef12"]
Sep 11 21:23:14.564962 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:14.564927    2578 generic.go:356] "Generic (PLEG): container finished" podID="1f83316f-02cc-4f3e-bd63-2752d402a1cc" containerID="faee5f812011d51f0d279f7b6541e247ff46422b617d64712a0baf6fa25587ed" exitCode=137
Sep 11 21:23:14.565135 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:14.564993    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-1e2b07"
Sep 11 21:23:14.565135 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:14.565026    2578 scope.go:117] "RemoveContainer" containerID="faee5f812011d51f0d279f7b6541e247ff46422b617d64712a0baf6fa25587ed"
Sep 11 21:23:14.565321 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:14.565305    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7699-3ca33a" container="test-curl-7699-3ca33a"
Sep 11 21:23:14.575464 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:14.575438    2578 scope.go:117] "RemoveContainer" containerID="faee5f812011d51f0d279f7b6541e247ff46422b617d64712a0baf6fa25587ed"
Sep 11 21:23:14.575795 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:14.575771    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faee5f812011d51f0d279f7b6541e247ff46422b617d64712a0baf6fa25587ed\": container with ID starting with faee5f812011d51f0d279f7b6541e247ff46422b617d64712a0baf6fa25587ed not found: ID does not exist" containerID="faee5f812011d51f0d279f7b6541e247ff46422b617d64712a0baf6fa25587ed"
Sep 11 21:23:14.575898 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:14.575805    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faee5f812011d51f0d279f7b6541e247ff46422b617d64712a0baf6fa25587ed"} err="failed to get container status \"faee5f812011d51f0d279f7b6541e247ff46422b617d64712a0baf6fa25587ed\": rpc error: code = NotFound desc = could not find container \"faee5f812011d51f0d279f7b6541e247ff46422b617d64712a0baf6fa25587ed\": container with ID starting with faee5f812011d51f0d279f7b6541e247ff46422b617d64712a0baf6fa25587ed not found: ID does not exist"
Sep 11 21:23:14.607397 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:14.607308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbstg\" (UniqueName: \"kubernetes.io/projected/b7eab14f-4a3a-42f2-ad5b-798bf9c92b38-kube-api-access-sbstg\") pod \"test-curl-7699-6eef12\" (UID: \"b7eab14f-4a3a-42f2-ad5b-798bf9c92b38\") " pod="opendatahub/test-curl-7699-6eef12"
Sep 11 21:23:14.707928 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:14.707886    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbstg\" (UniqueName: \"kubernetes.io/projected/b7eab14f-4a3a-42f2-ad5b-798bf9c92b38-kube-api-access-sbstg\") pod \"test-curl-7699-6eef12\" (UID: \"b7eab14f-4a3a-42f2-ad5b-798bf9c92b38\") " pod="opendatahub/test-curl-7699-6eef12"
Sep 11 21:23:14.716942 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:14.716922    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbstg\" (UniqueName: \"kubernetes.io/projected/b7eab14f-4a3a-42f2-ad5b-798bf9c92b38-kube-api-access-sbstg\") pod \"test-curl-7699-6eef12\" (UID: \"b7eab14f-4a3a-42f2-ad5b-798bf9c92b38\") " pod="opendatahub/test-curl-7699-6eef12"
Sep 11 21:23:14.787991 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:14.787953    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-6eef12"
Sep 11 21:23:15.063109 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:15.063076    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7699-6eef12"]
Sep 11 21:23:15.066431 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:23:15.066405    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7eab14f_4a3a_42f2_ad5b_798bf9c92b38.slice/crio-aba5605826cb013af6250fa4b10886610acd327efeb8c056180e8d11000f0109 WatchSource:0}: Error finding container aba5605826cb013af6250fa4b10886610acd327efeb8c056180e8d11000f0109: Status 404 returned error can't find the container with id aba5605826cb013af6250fa4b10886610acd327efeb8c056180e8d11000f0109
Sep 11 21:23:15.572712 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:15.572682    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-6eef12" event={"ID":"b7eab14f-4a3a-42f2-ad5b-798bf9c92b38","Type":"ContainerStarted","Data":"933c2507774262899964b3a336cccdea88afdaa10f106bbd8c7575670a22aade"}
Sep 11 21:23:15.573113 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:15.572727    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7699-6eef12" event={"ID":"b7eab14f-4a3a-42f2-ad5b-798bf9c92b38","Type":"ContainerStarted","Data":"aba5605826cb013af6250fa4b10886610acd327efeb8c056180e8d11000f0109"}
Sep 11 21:23:15.573113 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:15.572770    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7699-3ca33a" podUID="7b677240-2aa5-48b8-b058-fe834cef3e4d" containerName="test-curl-7699-3ca33a" containerID="cri-o://f0b2e48a31e9edecf957cee7384956ba66710c7fe688bf6fc6fb5c6e982e82b8" gracePeriod=2
Sep 11 21:23:15.574961 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:15.574934    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7699-3ca33a\" is forbidden: User \"system:node:ip-10-0-142-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-224.ec2.internal' and this object" podUID="7b677240-2aa5-48b8-b058-fe834cef3e4d" pod="opendatahub/test-curl-7699-3ca33a"
Sep 11 21:23:15.576873 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:15.576849    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7699-3ca33a\" is forbidden: User \"system:node:ip-10-0-142-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-224.ec2.internal' and this object" podUID="7b677240-2aa5-48b8-b058-fe834cef3e4d" pod="opendatahub/test-curl-7699-3ca33a"
Sep 11 21:23:15.594489 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:15.594434    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7699-6eef12" podStartSLOduration=1.594415649 podStartE2EDuration="1.594415649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:23:15.594177393 +0000 UTC m=+1062.298146683" watchObservedRunningTime="2026-09-11 21:23:15.594415649 +0000 UTC m=+1062.298384941"
Sep 11 21:23:15.745851 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:15.745808    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-640f1a"
Sep 11 21:23:15.748120 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:15.748096    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7699-3ca33a\" is forbidden: User \"system:node:ip-10-0-142-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-224.ec2.internal' and this object" podUID="7b677240-2aa5-48b8-b058-fe834cef3e4d" pod="opendatahub/test-curl-7699-3ca33a"
Sep 11 21:23:15.816271 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:15.816236    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24nk8\" (UniqueName: \"kubernetes.io/projected/1c6c305f-13bc-4351-87aa-b2c8dd961797-kube-api-access-24nk8\") pod \"1c6c305f-13bc-4351-87aa-b2c8dd961797\" (UID: \"1c6c305f-13bc-4351-87aa-b2c8dd961797\") "
Sep 11 21:23:15.818305 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:15.818273    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c6c305f-13bc-4351-87aa-b2c8dd961797-kube-api-access-24nk8" (OuterVolumeSpecName: "kube-api-access-24nk8") pod "1c6c305f-13bc-4351-87aa-b2c8dd961797" (UID: "1c6c305f-13bc-4351-87aa-b2c8dd961797"). InnerVolumeSpecName "kube-api-access-24nk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:23:15.916815 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:15.916775    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24nk8\" (UniqueName: \"kubernetes.io/projected/1c6c305f-13bc-4351-87aa-b2c8dd961797-kube-api-access-24nk8\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:15.927478 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:15.927447    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c6c305f-13bc-4351-87aa-b2c8dd961797" path="/var/lib/kubelet/pods/1c6c305f-13bc-4351-87aa-b2c8dd961797/volumes"
Sep 11 21:23:16.166156 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:16.166120    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7699-6eef12"]
Sep 11 21:23:16.169648 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:16.169603    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7699-6eef12"]
Sep 11 21:23:16.578692 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:16.578654    2578 generic.go:356] "Generic (PLEG): container finished" podID="1c6c305f-13bc-4351-87aa-b2c8dd961797" containerID="f4c9514117af28c40e144d9577586bf842647987a194c10a4c7de4b4fb7eea2d" exitCode=137
Sep 11 21:23:16.579193 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:16.578700    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-640f1a"
Sep 11 21:23:16.579193 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:16.578754    2578 scope.go:117] "RemoveContainer" containerID="f4c9514117af28c40e144d9577586bf842647987a194c10a4c7de4b4fb7eea2d"
Sep 11 21:23:16.579193 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:16.579102    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7699-6eef12" container="test-curl-7699-6eef12"
Sep 11 21:23:16.587138 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:16.587119    2578 scope.go:117] "RemoveContainer" containerID="f4c9514117af28c40e144d9577586bf842647987a194c10a4c7de4b4fb7eea2d"
Sep 11 21:23:16.587462 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:16.587443    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4c9514117af28c40e144d9577586bf842647987a194c10a4c7de4b4fb7eea2d\": container with ID starting with f4c9514117af28c40e144d9577586bf842647987a194c10a4c7de4b4fb7eea2d not found: ID does not exist" containerID="f4c9514117af28c40e144d9577586bf842647987a194c10a4c7de4b4fb7eea2d"
Sep 11 21:23:16.587507 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:16.587472    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c9514117af28c40e144d9577586bf842647987a194c10a4c7de4b4fb7eea2d"} err="failed to get container status \"f4c9514117af28c40e144d9577586bf842647987a194c10a4c7de4b4fb7eea2d\": rpc error: code = NotFound desc = could not find container \"f4c9514117af28c40e144d9577586bf842647987a194c10a4c7de4b4fb7eea2d\": container with ID starting with f4c9514117af28c40e144d9577586bf842647987a194c10a4c7de4b4fb7eea2d not found: ID does not exist"
Sep 11 21:23:17.585482 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:17.585452    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7699-6eef12" podUID="b7eab14f-4a3a-42f2-ad5b-798bf9c92b38" containerName="test-curl-7699-6eef12" containerID="cri-o://933c2507774262899964b3a336cccdea88afdaa10f106bbd8c7575670a22aade" gracePeriod=2
Sep 11 21:23:17.590763 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:17.590736    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7699-6eef12\" is forbidden: User \"system:node:ip-10-0-142-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-224.ec2.internal' and this object" podUID="b7eab14f-4a3a-42f2-ad5b-798bf9c92b38" pod="opendatahub/test-curl-7699-6eef12"
Sep 11 21:23:17.777215 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:17.777188    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-3ca33a"
Sep 11 21:23:17.779426 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:17.779399    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7699-6eef12\" is forbidden: User \"system:node:ip-10-0-142-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-224.ec2.internal' and this object" podUID="b7eab14f-4a3a-42f2-ad5b-798bf9c92b38" pod="opendatahub/test-curl-7699-6eef12"
Sep 11 21:23:17.833178 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:17.833083    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-976pt\" (UniqueName: \"kubernetes.io/projected/7b677240-2aa5-48b8-b058-fe834cef3e4d-kube-api-access-976pt\") pod \"7b677240-2aa5-48b8-b058-fe834cef3e4d\" (UID: \"7b677240-2aa5-48b8-b058-fe834cef3e4d\") "
Sep 11 21:23:17.835273 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:17.835243    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b677240-2aa5-48b8-b058-fe834cef3e4d-kube-api-access-976pt" (OuterVolumeSpecName: "kube-api-access-976pt") pod "7b677240-2aa5-48b8-b058-fe834cef3e4d" (UID: "7b677240-2aa5-48b8-b058-fe834cef3e4d"). InnerVolumeSpecName "kube-api-access-976pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:23:17.927445 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:17.927410    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b677240-2aa5-48b8-b058-fe834cef3e4d" path="/var/lib/kubelet/pods/7b677240-2aa5-48b8-b058-fe834cef3e4d/volumes"
Sep 11 21:23:17.934225 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:17.934191    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-976pt\" (UniqueName: \"kubernetes.io/projected/7b677240-2aa5-48b8-b058-fe834cef3e4d-kube-api-access-976pt\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:18.591649 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:18.591612    2578 generic.go:356] "Generic (PLEG): container finished" podID="7b677240-2aa5-48b8-b058-fe834cef3e4d" containerID="f0b2e48a31e9edecf957cee7384956ba66710c7fe688bf6fc6fb5c6e982e82b8" exitCode=137
Sep 11 21:23:18.592092 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:18.591672    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-3ca33a"
Sep 11 21:23:18.592092 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:18.591723    2578 scope.go:117] "RemoveContainer" containerID="f0b2e48a31e9edecf957cee7384956ba66710c7fe688bf6fc6fb5c6e982e82b8"
Sep 11 21:23:18.600155 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:18.600131    2578 scope.go:117] "RemoveContainer" containerID="f0b2e48a31e9edecf957cee7384956ba66710c7fe688bf6fc6fb5c6e982e82b8"
Sep 11 21:23:18.600514 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:18.600492    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0b2e48a31e9edecf957cee7384956ba66710c7fe688bf6fc6fb5c6e982e82b8\": container with ID starting with f0b2e48a31e9edecf957cee7384956ba66710c7fe688bf6fc6fb5c6e982e82b8 not found: ID does not exist" containerID="f0b2e48a31e9edecf957cee7384956ba66710c7fe688bf6fc6fb5c6e982e82b8"
Sep 11 21:23:18.600576 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:18.600525    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0b2e48a31e9edecf957cee7384956ba66710c7fe688bf6fc6fb5c6e982e82b8"} err="failed to get container status \"f0b2e48a31e9edecf957cee7384956ba66710c7fe688bf6fc6fb5c6e982e82b8\": rpc error: code = NotFound desc = could not find container \"f0b2e48a31e9edecf957cee7384956ba66710c7fe688bf6fc6fb5c6e982e82b8\": container with ID starting with f0b2e48a31e9edecf957cee7384956ba66710c7fe688bf6fc6fb5c6e982e82b8 not found: ID does not exist"
Sep 11 21:23:19.773527 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:19.773495    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-6eef12"
Sep 11 21:23:19.851814 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:19.851704    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbstg\" (UniqueName: \"kubernetes.io/projected/b7eab14f-4a3a-42f2-ad5b-798bf9c92b38-kube-api-access-sbstg\") pod \"b7eab14f-4a3a-42f2-ad5b-798bf9c92b38\" (UID: \"b7eab14f-4a3a-42f2-ad5b-798bf9c92b38\") "
Sep 11 21:23:19.853996 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:19.853956    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7eab14f-4a3a-42f2-ad5b-798bf9c92b38-kube-api-access-sbstg" (OuterVolumeSpecName: "kube-api-access-sbstg") pod "b7eab14f-4a3a-42f2-ad5b-798bf9c92b38" (UID: "b7eab14f-4a3a-42f2-ad5b-798bf9c92b38"). InnerVolumeSpecName "kube-api-access-sbstg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:23:19.927618 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:19.927584    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7eab14f-4a3a-42f2-ad5b-798bf9c92b38" path="/var/lib/kubelet/pods/b7eab14f-4a3a-42f2-ad5b-798bf9c92b38/volumes"
Sep 11 21:23:19.952912 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:19.952876    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbstg\" (UniqueName: \"kubernetes.io/projected/b7eab14f-4a3a-42f2-ad5b-798bf9c92b38-kube-api-access-sbstg\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:20.602732 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:20.602693    2578 generic.go:356] "Generic (PLEG): container finished" podID="b7eab14f-4a3a-42f2-ad5b-798bf9c92b38" containerID="933c2507774262899964b3a336cccdea88afdaa10f106bbd8c7575670a22aade" exitCode=137
Sep 11 21:23:20.602732 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:20.602736    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7699-6eef12"
Sep 11 21:23:20.603093 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:20.602792    2578 scope.go:117] "RemoveContainer" containerID="933c2507774262899964b3a336cccdea88afdaa10f106bbd8c7575670a22aade"
Sep 11 21:23:20.612792 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:20.612769    2578 scope.go:117] "RemoveContainer" containerID="933c2507774262899964b3a336cccdea88afdaa10f106bbd8c7575670a22aade"
Sep 11 21:23:20.613386 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:20.613353    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933c2507774262899964b3a336cccdea88afdaa10f106bbd8c7575670a22aade\": container with ID starting with 933c2507774262899964b3a336cccdea88afdaa10f106bbd8c7575670a22aade not found: ID does not exist" containerID="933c2507774262899964b3a336cccdea88afdaa10f106bbd8c7575670a22aade"
Sep 11 21:23:20.613502 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:20.613396    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933c2507774262899964b3a336cccdea88afdaa10f106bbd8c7575670a22aade"} err="failed to get container status \"933c2507774262899964b3a336cccdea88afdaa10f106bbd8c7575670a22aade\": rpc error: code = NotFound desc = could not find container \"933c2507774262899964b3a336cccdea88afdaa10f106bbd8c7575670a22aade\": container with ID starting with 933c2507774262899964b3a336cccdea88afdaa10f106bbd8c7575670a22aade not found: ID does not exist"
Sep 11 21:23:23.833878 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:23.833843    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj"]
Sep 11 21:23:23.834259 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:23.834157    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7eab14f-4a3a-42f2-ad5b-798bf9c92b38" containerName="test-curl-7699-6eef12"
Sep 11 21:23:23.834259 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:23.834167    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7eab14f-4a3a-42f2-ad5b-798bf9c92b38" containerName="test-curl-7699-6eef12"
Sep 11 21:23:23.834259 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:23.834223    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7eab14f-4a3a-42f2-ad5b-798bf9c92b38" containerName="test-curl-7699-6eef12"
Sep 11 21:23:23.838595 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:23.838575    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj"
Sep 11 21:23:23.841411 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:23.841392    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-908a1435\""
Sep 11 21:23:23.841523 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:23.841392    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-908a1435-dockercfg-696pm\""
Sep 11 21:23:23.848216 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:23.848183    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj"]
Sep 11 21:23:23.988231 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:23.988193    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/055c5601-2aae-442a-932f-ada1ae86ba83-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj\" (UID: \"055c5601-2aae-442a-932f-ada1ae86ba83\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj"
Sep 11 21:23:23.988231 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:23.988232    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcc7l\" (UniqueName: \"kubernetes.io/projected/055c5601-2aae-442a-932f-ada1ae86ba83-kube-api-access-zcc7l\") pod \"payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj\" (UID: \"055c5601-2aae-442a-932f-ada1ae86ba83\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj"
Sep 11 21:23:24.071364 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.071326    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp"]
Sep 11 21:23:24.071623 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.071594    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp" podUID="55889c70-06c0-4fc3-adc1-c122113be4ae" containerName="payload-processing" containerID="cri-o://9c033aee13982f20d72e194645b4078f5b7578e02ac395c3c8c857943bd38b26" gracePeriod=30
Sep 11 21:23:24.076984 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.076956    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982"]
Sep 11 21:23:24.077214 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.077186    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982" podUID="a9ddb125-b7ae-4f13-8e77-f8a18107c88d" containerName="payload-pre-processing" containerID="cri-o://9853d7df85e338e7c901ed4f1f7bb800367060a2577e66e4102729e942ae2a52" gracePeriod=30
Sep 11 21:23:24.089186 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.089155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcc7l\" (UniqueName: \"kubernetes.io/projected/055c5601-2aae-442a-932f-ada1ae86ba83-kube-api-access-zcc7l\") pod \"payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj\" (UID: \"055c5601-2aae-442a-932f-ada1ae86ba83\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj"
Sep 11 21:23:24.089327 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.089290    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/055c5601-2aae-442a-932f-ada1ae86ba83-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj\" (UID: \"055c5601-2aae-442a-932f-ada1ae86ba83\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj"
Sep 11 21:23:24.090206 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.090182    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/055c5601-2aae-442a-932f-ada1ae86ba83-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj\" (UID: \"055c5601-2aae-442a-932f-ada1ae86ba83\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj"
Sep 11 21:23:24.102039 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.102003    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcc7l\" (UniqueName: \"kubernetes.io/projected/055c5601-2aae-442a-932f-ada1ae86ba83-kube-api-access-zcc7l\") pod \"payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj\" (UID: \"055c5601-2aae-442a-932f-ada1ae86ba83\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj"
Sep 11 21:23:24.148793 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.148763    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj"
Sep 11 21:23:24.347306 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.347281    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp"
Sep 11 21:23:24.355656 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.355630    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982"
Sep 11 21:23:24.438896 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.438873    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj"]
Sep 11 21:23:24.441378 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:23:24.441356    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod055c5601_2aae_442a_932f_ada1ae86ba83.slice/crio-1def349e64fcadd77ee7a9158377184c519132b8f62af454de99275e354fa869 WatchSource:0}: Error finding container 1def349e64fcadd77ee7a9158377184c519132b8f62af454de99275e354fa869: Status 404 returned error can't find the container with id 1def349e64fcadd77ee7a9158377184c519132b8f62af454de99275e354fa869
Sep 11 21:23:24.494058 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.494033    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55889c70-06c0-4fc3-adc1-c122113be4ae-plugins-config-volume\") pod \"55889c70-06c0-4fc3-adc1-c122113be4ae\" (UID: \"55889c70-06c0-4fc3-adc1-c122113be4ae\") "
Sep 11 21:23:24.494193 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.494101    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ddb125-b7ae-4f13-8e77-f8a18107c88d-plugins-config-volume\") pod \"a9ddb125-b7ae-4f13-8e77-f8a18107c88d\" (UID: \"a9ddb125-b7ae-4f13-8e77-f8a18107c88d\") "
Sep 11 21:23:24.494193 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.494125    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rd79\" (UniqueName: \"kubernetes.io/projected/a9ddb125-b7ae-4f13-8e77-f8a18107c88d-kube-api-access-6rd79\") pod \"a9ddb125-b7ae-4f13-8e77-f8a18107c88d\" (UID: \"a9ddb125-b7ae-4f13-8e77-f8a18107c88d\") "
Sep 11 21:23:24.494193 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.494190    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gx6fm\" (UniqueName: \"kubernetes.io/projected/55889c70-06c0-4fc3-adc1-c122113be4ae-kube-api-access-gx6fm\") pod \"55889c70-06c0-4fc3-adc1-c122113be4ae\" (UID: \"55889c70-06c0-4fc3-adc1-c122113be4ae\") "
Sep 11 21:23:24.496809 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.494920    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ddb125-b7ae-4f13-8e77-f8a18107c88d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a9ddb125-b7ae-4f13-8e77-f8a18107c88d" (UID: "a9ddb125-b7ae-4f13-8e77-f8a18107c88d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:23:24.501639 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.501603    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ddb125-b7ae-4f13-8e77-f8a18107c88d-kube-api-access-6rd79" (OuterVolumeSpecName: "kube-api-access-6rd79") pod "a9ddb125-b7ae-4f13-8e77-f8a18107c88d" (UID: "a9ddb125-b7ae-4f13-8e77-f8a18107c88d"). InnerVolumeSpecName "kube-api-access-6rd79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:23:24.501763 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.501599    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55889c70-06c0-4fc3-adc1-c122113be4ae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "55889c70-06c0-4fc3-adc1-c122113be4ae" (UID: "55889c70-06c0-4fc3-adc1-c122113be4ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:23:24.501923 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.501903    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55889c70-06c0-4fc3-adc1-c122113be4ae-kube-api-access-gx6fm" (OuterVolumeSpecName: "kube-api-access-gx6fm") pod "55889c70-06c0-4fc3-adc1-c122113be4ae" (UID: "55889c70-06c0-4fc3-adc1-c122113be4ae"). InnerVolumeSpecName "kube-api-access-gx6fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:23:24.595645 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.595558    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx6fm\" (UniqueName: \"kubernetes.io/projected/55889c70-06c0-4fc3-adc1-c122113be4ae-kube-api-access-gx6fm\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:24.595645 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.595588    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55889c70-06c0-4fc3-adc1-c122113be4ae-plugins-config-volume\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:24.595645 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.595599    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ddb125-b7ae-4f13-8e77-f8a18107c88d-plugins-config-volume\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:24.595645 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.595607    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rd79\" (UniqueName: \"kubernetes.io/projected/a9ddb125-b7ae-4f13-8e77-f8a18107c88d-kube-api-access-6rd79\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:23:24.623987 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.623948    2578 generic.go:356] "Generic (PLEG): container finished" podID="a9ddb125-b7ae-4f13-8e77-f8a18107c88d" containerID="9853d7df85e338e7c901ed4f1f7bb800367060a2577e66e4102729e942ae2a52" exitCode=0
Sep 11 21:23:24.624167 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.624022    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982"
Sep 11 21:23:24.624167 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.624027    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982" event={"ID":"a9ddb125-b7ae-4f13-8e77-f8a18107c88d","Type":"ContainerDied","Data":"9853d7df85e338e7c901ed4f1f7bb800367060a2577e66e4102729e942ae2a52"}
Sep 11 21:23:24.624287 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.624210    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982" event={"ID":"a9ddb125-b7ae-4f13-8e77-f8a18107c88d","Type":"ContainerDied","Data":"7e2bbbd602d09da001702e437409010b1b97a137422955b7277c135b2d3d8893"}
Sep 11 21:23:24.624287 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.624251    2578 scope.go:117] "RemoveContainer" containerID="9853d7df85e338e7c901ed4f1f7bb800367060a2577e66e4102729e942ae2a52"
Sep 11 21:23:24.625615 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.625589    2578 generic.go:356] "Generic (PLEG): container finished" podID="55889c70-06c0-4fc3-adc1-c122113be4ae" containerID="9c033aee13982f20d72e194645b4078f5b7578e02ac395c3c8c857943bd38b26" exitCode=0
Sep 11 21:23:24.625720 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.625650    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp"
Sep 11 21:23:24.625720 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.625669    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp" event={"ID":"55889c70-06c0-4fc3-adc1-c122113be4ae","Type":"ContainerDied","Data":"9c033aee13982f20d72e194645b4078f5b7578e02ac395c3c8c857943bd38b26"}
Sep 11 21:23:24.625851 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.625732    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp" event={"ID":"55889c70-06c0-4fc3-adc1-c122113be4ae","Type":"ContainerDied","Data":"ae5f84d7e8bcfcd09c7def7e58c785c4b855e9541d2092a06a755b2d6b35694c"}
Sep 11 21:23:24.627589 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.627559    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj" event={"ID":"055c5601-2aae-442a-932f-ada1ae86ba83","Type":"ContainerStarted","Data":"335874d1a7887b548ff8602cb3f19a0413c87877858a2a6adb9d0a20fa2ba8f9"}
Sep 11 21:23:24.627710 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.627609    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj" event={"ID":"055c5601-2aae-442a-932f-ada1ae86ba83","Type":"ContainerStarted","Data":"1def349e64fcadd77ee7a9158377184c519132b8f62af454de99275e354fa869"}
Sep 11 21:23:24.627710 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.627665    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj"
Sep 11 21:23:24.635231 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.635165    2578 scope.go:117] "RemoveContainer" containerID="9853d7df85e338e7c901ed4f1f7bb800367060a2577e66e4102729e942ae2a52"
Sep 11 21:23:24.635575 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:24.635548    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9853d7df85e338e7c901ed4f1f7bb800367060a2577e66e4102729e942ae2a52\": container with ID starting with 9853d7df85e338e7c901ed4f1f7bb800367060a2577e66e4102729e942ae2a52 not found: ID does not exist" containerID="9853d7df85e338e7c901ed4f1f7bb800367060a2577e66e4102729e942ae2a52"
Sep 11 21:23:24.635677 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.635586    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9853d7df85e338e7c901ed4f1f7bb800367060a2577e66e4102729e942ae2a52"} err="failed to get container status \"9853d7df85e338e7c901ed4f1f7bb800367060a2577e66e4102729e942ae2a52\": rpc error: code = NotFound desc = could not find container \"9853d7df85e338e7c901ed4f1f7bb800367060a2577e66e4102729e942ae2a52\": container with ID starting with 9853d7df85e338e7c901ed4f1f7bb800367060a2577e66e4102729e942ae2a52 not found: ID does not exist"
Sep 11 21:23:24.635677 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.635609    2578 scope.go:117] "RemoveContainer" containerID="9c033aee13982f20d72e194645b4078f5b7578e02ac395c3c8c857943bd38b26"
Sep 11 21:23:24.644587 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.644559    2578 scope.go:117] "RemoveContainer" containerID="9c033aee13982f20d72e194645b4078f5b7578e02ac395c3c8c857943bd38b26"
Sep 11 21:23:24.644957 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:23:24.644935    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c033aee13982f20d72e194645b4078f5b7578e02ac395c3c8c857943bd38b26\": container with ID starting with 9c033aee13982f20d72e194645b4078f5b7578e02ac395c3c8c857943bd38b26 not found: ID does not exist" containerID="9c033aee13982f20d72e194645b4078f5b7578e02ac395c3c8c857943bd38b26"
Sep 11 21:23:24.645023 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.644967    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c033aee13982f20d72e194645b4078f5b7578e02ac395c3c8c857943bd38b26"} err="failed to get container status \"9c033aee13982f20d72e194645b4078f5b7578e02ac395c3c8c857943bd38b26\": rpc error: code = NotFound desc = could not find container \"9c033aee13982f20d72e194645b4078f5b7578e02ac395c3c8c857943bd38b26\": container with ID starting with 9c033aee13982f20d72e194645b4078f5b7578e02ac395c3c8c857943bd38b26 not found: ID does not exist"
Sep 11 21:23:24.647483 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.647437    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj" podStartSLOduration=1.647418622 podStartE2EDuration="1.647418622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:23:24.647240826 +0000 UTC m=+1071.351210118" watchObservedRunningTime="2026-09-11 21:23:24.647418622 +0000 UTC m=+1071.351387906"
Sep 11 21:23:24.661686 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.661657    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982"]
Sep 11 21:23:24.665635 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.665602    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2b95f8-8d7467674-ms982"]
Sep 11 21:23:24.675267 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.675241    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp"]
Sep 11 21:23:24.680177 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:24.680150    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2b95f8-67577b6c95-7b7hp"]
Sep 11 21:23:25.926970 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:25.926899    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55889c70-06c0-4fc3-adc1-c122113be4ae" path="/var/lib/kubelet/pods/55889c70-06c0-4fc3-adc1-c122113be4ae/volumes"
Sep 11 21:23:25.927320 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:25.927299    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9ddb125-b7ae-4f13-8e77-f8a18107c88d" path="/var/lib/kubelet/pods/a9ddb125-b7ae-4f13-8e77-f8a18107c88d/volumes"
Sep 11 21:23:28.824274 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:28.824242    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:23:31.524335 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.524304    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"]
Sep 11 21:23:31.524715 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.524659    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9ddb125-b7ae-4f13-8e77-f8a18107c88d" containerName="payload-pre-processing"
Sep 11 21:23:31.524715 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.524671    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9ddb125-b7ae-4f13-8e77-f8a18107c88d" containerName="payload-pre-processing"
Sep 11 21:23:31.524715 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.524685    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55889c70-06c0-4fc3-adc1-c122113be4ae" containerName="payload-processing"
Sep 11 21:23:31.524715 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.524691    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="55889c70-06c0-4fc3-adc1-c122113be4ae" containerName="payload-processing"
Sep 11 21:23:31.524870 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.524738    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9ddb125-b7ae-4f13-8e77-f8a18107c88d" containerName="payload-pre-processing"
Sep 11 21:23:31.524870 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.524747    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="55889c70-06c0-4fc3-adc1-c122113be4ae" containerName="payload-processing"
Sep 11 21:23:31.527689 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.527669    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.530047 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.530020    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 21:23:31.530188 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.530169    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-2bd772-openshift-default-dockercfg-cx6hr\""
Sep 11 21:23:31.547062 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.547032    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"]
Sep 11 21:23:31.656487 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.656450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-istio-envoy\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.656487 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.656491    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d37d20ec-69f2-49ce-a03f-5970c06d7177-istio-token\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.656679 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.656514    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-workload-certs\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.656679 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.656531    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp7rl\" (UniqueName: \"kubernetes.io/projected/d37d20ec-69f2-49ce-a03f-5970c06d7177-kube-api-access-mp7rl\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.656679 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.656643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-workload-socket\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.656802 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.656691    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d37d20ec-69f2-49ce-a03f-5970c06d7177-istio-podinfo\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.656802 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.656738    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d37d20ec-69f2-49ce-a03f-5970c06d7177-istiod-ca-cert\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.656802 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.656763    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-istio-data\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.656918 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.656812    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-credential-socket\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.757235 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.757194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d37d20ec-69f2-49ce-a03f-5970c06d7177-istiod-ca-cert\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.757235 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.757238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-istio-data\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.757478 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.757272    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-credential-socket\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.757478 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.757297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-istio-envoy\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.757478 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.757317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d37d20ec-69f2-49ce-a03f-5970c06d7177-istio-token\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.757478 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.757339    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-workload-certs\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.757478 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.757362    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp7rl\" (UniqueName: \"kubernetes.io/projected/d37d20ec-69f2-49ce-a03f-5970c06d7177-kube-api-access-mp7rl\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.757478 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.757409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-workload-socket\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.757478 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.757430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d37d20ec-69f2-49ce-a03f-5970c06d7177-istio-podinfo\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.757859 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.757790    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-workload-certs\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.757937 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.757896    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-credential-socket\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.757975 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.757935    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-istio-data\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.758006 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.757985    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-workload-socket\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.758395 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.758374    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d37d20ec-69f2-49ce-a03f-5970c06d7177-istiod-ca-cert\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.759869 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.759846    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-istio-envoy\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.760030 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.760011    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d37d20ec-69f2-49ce-a03f-5970c06d7177-istio-podinfo\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.765866 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.765833    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d37d20ec-69f2-49ce-a03f-5970c06d7177-istio-token\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.765984 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.765965    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp7rl\" (UniqueName: \"kubernetes.io/projected/d37d20ec-69f2-49ce-a03f-5970c06d7177-kube-api-access-mp7rl\") pod \"e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") " pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:31.838197 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:31.838102    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:32.123503 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:32.123411    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"]
Sep 11 21:23:32.127317 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:23:32.127284    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37d20ec_69f2_49ce_a03f_5970c06d7177.slice/crio-802cbe64baa1799982321d3560a4364625ffb85a797cc4907b2db445eb52b1ee WatchSource:0}: Error finding container 802cbe64baa1799982321d3560a4364625ffb85a797cc4907b2db445eb52b1ee: Status 404 returned error can't find the container with id 802cbe64baa1799982321d3560a4364625ffb85a797cc4907b2db445eb52b1ee
Sep 11 21:23:32.129855 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:32.129808    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:23:32.129923 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:32.129893    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:23:32.129970 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:32.129920    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:23:32.671643 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:32.671610    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd" event={"ID":"d37d20ec-69f2-49ce-a03f-5970c06d7177","Type":"ContainerStarted","Data":"34d288c8fee081297b61f09ff4657425b2efdc703b492e16e3c91e26695ba7eb"}
Sep 11 21:23:32.672041 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:32.671661    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd" event={"ID":"d37d20ec-69f2-49ce-a03f-5970c06d7177","Type":"ContainerStarted","Data":"802cbe64baa1799982321d3560a4364625ffb85a797cc4907b2db445eb52b1ee"}
Sep 11 21:23:32.692531 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:32.692479    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd" podStartSLOduration=1.692461651 podStartE2EDuration="1.692461651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:23:32.690802665 +0000 UTC m=+1079.394771955" watchObservedRunningTime="2026-09-11 21:23:32.692461651 +0000 UTC m=+1079.396430941"
Sep 11 21:23:32.839174 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:32.839135    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:33.843761 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:33.843732    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:34.681480 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:34.681441    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:34.682810 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:34.682784    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:35.637195 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:35.637164    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj"
Sep 11 21:23:37.800125 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:37.800090    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb"]
Sep 11 21:23:37.805128 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:37.805111    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb"
Sep 11 21:23:37.808117 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:37.808099    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-2bd772\""
Sep 11 21:23:37.808117 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:37.808110    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-2bd772-dockercfg-htwc5\""
Sep 11 21:23:37.829667 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:37.829637    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb"]
Sep 11 21:23:37.851130 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:37.851089    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd"]
Sep 11 21:23:37.854539 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:37.854512    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd"
Sep 11 21:23:37.872627 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:37.872596    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd"]
Sep 11 21:23:37.908792 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:37.908752    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr5l7\" (UniqueName: \"kubernetes.io/projected/b9d7fc44-e0c4-4933-8079-f0cd21d243a3-kube-api-access-hr5l7\") pod \"payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb\" (UID: \"b9d7fc44-e0c4-4933-8079-f0cd21d243a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb"
Sep 11 21:23:37.908792 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:37.908795    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9d7fc44-e0c4-4933-8079-f0cd21d243a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb\" (UID: \"b9d7fc44-e0c4-4933-8079-f0cd21d243a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb"
Sep 11 21:23:38.009893 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:38.009852    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr5l7\" (UniqueName: \"kubernetes.io/projected/b9d7fc44-e0c4-4933-8079-f0cd21d243a3-kube-api-access-hr5l7\") pod \"payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb\" (UID: \"b9d7fc44-e0c4-4933-8079-f0cd21d243a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb"
Sep 11 21:23:38.010093 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:38.009904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9d7fc44-e0c4-4933-8079-f0cd21d243a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb\" (UID: \"b9d7fc44-e0c4-4933-8079-f0cd21d243a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb"
Sep 11 21:23:38.010093 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:38.010000    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdj7b\" (UniqueName: \"kubernetes.io/projected/43b41491-60a0-4dbf-8aa7-32e9344f0497-kube-api-access-qdj7b\") pod \"payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd\" (UID: \"43b41491-60a0-4dbf-8aa7-32e9344f0497\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd"
Sep 11 21:23:38.010093 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:38.010031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b41491-60a0-4dbf-8aa7-32e9344f0497-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd\" (UID: \"43b41491-60a0-4dbf-8aa7-32e9344f0497\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd"
Sep 11 21:23:38.010548 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:38.010527    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9d7fc44-e0c4-4933-8079-f0cd21d243a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb\" (UID: \"b9d7fc44-e0c4-4933-8079-f0cd21d243a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb"
Sep 11 21:23:38.019050 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:38.019016    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr5l7\" (UniqueName: \"kubernetes.io/projected/b9d7fc44-e0c4-4933-8079-f0cd21d243a3-kube-api-access-hr5l7\") pod \"payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb\" (UID: \"b9d7fc44-e0c4-4933-8079-f0cd21d243a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb"
Sep 11 21:23:38.111224 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:38.111058    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdj7b\" (UniqueName: \"kubernetes.io/projected/43b41491-60a0-4dbf-8aa7-32e9344f0497-kube-api-access-qdj7b\") pod \"payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd\" (UID: \"43b41491-60a0-4dbf-8aa7-32e9344f0497\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd"
Sep 11 21:23:38.111224 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:38.111202    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b41491-60a0-4dbf-8aa7-32e9344f0497-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd\" (UID: \"43b41491-60a0-4dbf-8aa7-32e9344f0497\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd"
Sep 11 21:23:38.111949 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:38.111928    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b41491-60a0-4dbf-8aa7-32e9344f0497-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd\" (UID: \"43b41491-60a0-4dbf-8aa7-32e9344f0497\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd"
Sep 11 21:23:38.115946 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:38.115922    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb"
Sep 11 21:23:38.119754 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:38.119731    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdj7b\" (UniqueName: \"kubernetes.io/projected/43b41491-60a0-4dbf-8aa7-32e9344f0497-kube-api-access-qdj7b\") pod \"payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd\" (UID: \"43b41491-60a0-4dbf-8aa7-32e9344f0497\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd"
Sep 11 21:23:38.164866 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:38.164805    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd"
Sep 11 21:23:38.388875 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:38.388705    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb"]
Sep 11 21:23:38.391514 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:23:38.391483    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d7fc44_e0c4_4933_8079_f0cd21d243a3.slice/crio-683dc98b11a3255bc51fe7d3926c844ff96bc8831c9b84b45804796700c0e13c WatchSource:0}: Error finding container 683dc98b11a3255bc51fe7d3926c844ff96bc8831c9b84b45804796700c0e13c: Status 404 returned error can't find the container with id 683dc98b11a3255bc51fe7d3926c844ff96bc8831c9b84b45804796700c0e13c
Sep 11 21:23:38.459574 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:38.459527    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd"]
Sep 11 21:23:38.463275 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:23:38.463249    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b41491_60a0_4dbf_8aa7_32e9344f0497.slice/crio-d6bfc7807894b28f8e3c2aa369d2f32ceeb3ebf21348c62afb1487d0f01e3337 WatchSource:0}: Error finding container d6bfc7807894b28f8e3c2aa369d2f32ceeb3ebf21348c62afb1487d0f01e3337: Status 404 returned error can't find the container with id d6bfc7807894b28f8e3c2aa369d2f32ceeb3ebf21348c62afb1487d0f01e3337
Sep 11 21:23:38.700341 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:38.700286    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb" event={"ID":"b9d7fc44-e0c4-4933-8079-f0cd21d243a3","Type":"ContainerStarted","Data":"e2650615b7bd185105f757b5ddacde347cca55c20aa0250245480e76f4f4966a"}
Sep 11 21:23:38.700560 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:38.700496    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb"
Sep 11 21:23:38.700560 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:38.700541    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb" event={"ID":"b9d7fc44-e0c4-4933-8079-f0cd21d243a3","Type":"ContainerStarted","Data":"683dc98b11a3255bc51fe7d3926c844ff96bc8831c9b84b45804796700c0e13c"}
Sep 11 21:23:38.701939 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:38.701914    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd" event={"ID":"43b41491-60a0-4dbf-8aa7-32e9344f0497","Type":"ContainerStarted","Data":"2069b3c4f78148d5c69eb4b4253c3d5a5df9271441784801db30cf745cb98409"}
Sep 11 21:23:38.702057 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:38.701957    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd" event={"ID":"43b41491-60a0-4dbf-8aa7-32e9344f0497","Type":"ContainerStarted","Data":"d6bfc7807894b28f8e3c2aa369d2f32ceeb3ebf21348c62afb1487d0f01e3337"}
Sep 11 21:23:38.702057 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:38.702039    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd"
Sep 11 21:23:38.718899 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:38.718849    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb" podStartSLOduration=1.71880816 podStartE2EDuration="1.71880816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:23:38.717126781 +0000 UTC m=+1085.421096065" watchObservedRunningTime="2026-09-11 21:23:38.71880816 +0000 UTC m=+1085.422777449"
Sep 11 21:23:38.735988 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:38.735936    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd" podStartSLOduration=1.7358956939999999 podStartE2EDuration="1.735895694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:23:38.73441763 +0000 UTC m=+1085.438386923" watchObservedRunningTime="2026-09-11 21:23:38.735895694 +0000 UTC m=+1085.439864984"
Sep 11 21:23:46.241141 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:46.241103    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:23:49.708625 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:49.708596    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb"
Sep 11 21:23:49.709100 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:49.708664    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd"
Sep 11 21:23:56.790926 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:56.790891    2578 generic.go:356] "Generic (PLEG): container finished" podID="d37d20ec-69f2-49ce-a03f-5970c06d7177" containerID="34d288c8fee081297b61f09ff4657425b2efdc703b492e16e3c91e26695ba7eb" exitCode=0
Sep 11 21:23:56.791335 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:56.790970    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd" event={"ID":"d37d20ec-69f2-49ce-a03f-5970c06d7177","Type":"ContainerDied","Data":"34d288c8fee081297b61f09ff4657425b2efdc703b492e16e3c91e26695ba7eb"}
Sep 11 21:23:56.791377 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:56.791359    2578 scope.go:117] "RemoveContainer" containerID="34d288c8fee081297b61f09ff4657425b2efdc703b492e16e3c91e26695ba7eb"
Sep 11 21:23:56.792361 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:56.792336    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:23:56.792415 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:56.792400    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:23:56.792457 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:56.792430    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:23:57.797374 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:57.797336    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd" event={"ID":"d37d20ec-69f2-49ce-a03f-5970c06d7177","Type":"ContainerStarted","Data":"484dac02c7842e069e115bdc141f917c6efea6641a2fdb8c0a0ecdb69d881b29"}
Sep 11 21:23:57.839168 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:57.839127    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:23:57.839696 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:57.839668    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 11 21:23:57.839853 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:57.839742    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd" podUID="d37d20ec-69f2-49ce-a03f-5970c06d7177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 11 21:23:58.839108 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:58.839056    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 11 21:23:58.839630 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:58.839153    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd" podUID="d37d20ec-69f2-49ce-a03f-5970c06d7177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 11 21:23:59.838895 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:59.838848    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 11 21:23:59.839086 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:23:59.838937    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd" podUID="d37d20ec-69f2-49ce-a03f-5970c06d7177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 11 21:24:00.839545 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:00.839505    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 11 21:24:00.840204 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:00.840166    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd" podUID="d37d20ec-69f2-49ce-a03f-5970c06d7177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 11 21:24:01.839409 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:01.839362    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Sep 11 21:24:01.839627 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:01.839436    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd" podUID="d37d20ec-69f2-49ce-a03f-5970c06d7177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Sep 11 21:24:02.984883 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:02.984840    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:24:02.985407 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:02.985183    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:24:02.986542 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:02.986496    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:24:08.297799 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:08.297762    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb"]
Sep 11 21:24:08.301130 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:08.301109    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb"
Sep 11 21:24:08.303412 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:08.303391    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-713791\""
Sep 11 21:24:08.303735 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:08.303719    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-713791-dockercfg-kvhm4\""
Sep 11 21:24:08.314158 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:08.314135    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb"]
Sep 11 21:24:08.377676 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:08.377639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrqdc\" (UniqueName: \"kubernetes.io/projected/b781df93-f077-495c-b3c0-9c817514e6a9-kube-api-access-zrqdc\") pod \"payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb\" (UID: \"b781df93-f077-495c-b3c0-9c817514e6a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb"
Sep 11 21:24:08.377676 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:08.377680    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b781df93-f077-495c-b3c0-9c817514e6a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb\" (UID: \"b781df93-f077-495c-b3c0-9c817514e6a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb"
Sep 11 21:24:08.478156 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:08.478112    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrqdc\" (UniqueName: \"kubernetes.io/projected/b781df93-f077-495c-b3c0-9c817514e6a9-kube-api-access-zrqdc\") pod \"payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb\" (UID: \"b781df93-f077-495c-b3c0-9c817514e6a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb"
Sep 11 21:24:08.478156 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:08.478154    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b781df93-f077-495c-b3c0-9c817514e6a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb\" (UID: \"b781df93-f077-495c-b3c0-9c817514e6a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb"
Sep 11 21:24:08.479052 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:08.479032    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b781df93-f077-495c-b3c0-9c817514e6a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb\" (UID: \"b781df93-f077-495c-b3c0-9c817514e6a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb"
Sep 11 21:24:08.486268 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:08.486249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrqdc\" (UniqueName: \"kubernetes.io/projected/b781df93-f077-495c-b3c0-9c817514e6a9-kube-api-access-zrqdc\") pod \"payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb\" (UID: \"b781df93-f077-495c-b3c0-9c817514e6a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb"
Sep 11 21:24:08.611418 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:08.611327    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb"
Sep 11 21:24:08.884256 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:08.884179    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb"]
Sep 11 21:24:08.887266 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:24:08.887238    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb781df93_f077_495c_b3c0_9c817514e6a9.slice/crio-b1e8761090f366ecdee0214ecd67e952efae4e6717fdcc6a53555a6ee7435b58 WatchSource:0}: Error finding container b1e8761090f366ecdee0214ecd67e952efae4e6717fdcc6a53555a6ee7435b58: Status 404 returned error can't find the container with id b1e8761090f366ecdee0214ecd67e952efae4e6717fdcc6a53555a6ee7435b58
Sep 11 21:24:09.851978 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:09.851940    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb" event={"ID":"b781df93-f077-495c-b3c0-9c817514e6a9","Type":"ContainerStarted","Data":"c2c82d935203b25761848f3262677c522d01f5b498612fa4b8cdd437e476dfdb"}
Sep 11 21:24:09.852366 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:09.852000    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb" event={"ID":"b781df93-f077-495c-b3c0-9c817514e6a9","Type":"ContainerStarted","Data":"b1e8761090f366ecdee0214ecd67e952efae4e6717fdcc6a53555a6ee7435b58"}
Sep 11 21:24:09.852366 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:09.852067    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb"
Sep 11 21:24:09.871349 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:09.871291    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb" podStartSLOduration=1.871269627 podStartE2EDuration="1.871269627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:24:09.869784887 +0000 UTC m=+1116.573754177" watchObservedRunningTime="2026-09-11 21:24:09.871269627 +0000 UTC m=+1116.575238921"
Sep 11 21:24:13.697170 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:13.697133    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:24:15.115618 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.115582    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"]
Sep 11 21:24:15.120620 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.120598    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:24:15.123116 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.123092    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-908a1435\"/\"default-dockercfg-x84qb\""
Sep 11 21:24:15.123838 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.123798    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-908a1435\"/\"openshift-service-ca.crt\""
Sep 11 21:24:15.123953 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.123860    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-908a1435\"/\"kube-root-ca.crt\""
Sep 11 21:24:15.123953 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.123930    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-908a1435\"/\"e2e-lifecycle-model-908a1435-kserve-self-signed-certs\""
Sep 11 21:24:15.130619 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.130598    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"]
Sep 11 21:24:15.245055 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.245019    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/217c78d8-f167-4fd0-b118-bcb8d524bef0-model-cache\") pod \"e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46\" (UID: \"217c78d8-f167-4fd0-b118-bcb8d524bef0\") " pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:24:15.245055 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.245059    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmvdj\" (UniqueName: \"kubernetes.io/projected/217c78d8-f167-4fd0-b118-bcb8d524bef0-kube-api-access-zmvdj\") pod \"e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46\" (UID: \"217c78d8-f167-4fd0-b118-bcb8d524bef0\") " pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:24:15.245261 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.245127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/217c78d8-f167-4fd0-b118-bcb8d524bef0-dshm\") pod \"e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46\" (UID: \"217c78d8-f167-4fd0-b118-bcb8d524bef0\") " pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:24:15.245261 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.245160    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/217c78d8-f167-4fd0-b118-bcb8d524bef0-tmp-dir\") pod \"e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46\" (UID: \"217c78d8-f167-4fd0-b118-bcb8d524bef0\") " pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:24:15.245261 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.245193    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/217c78d8-f167-4fd0-b118-bcb8d524bef0-tls-certs\") pod \"e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46\" (UID: \"217c78d8-f167-4fd0-b118-bcb8d524bef0\") " pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:24:15.245261 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.245226    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/217c78d8-f167-4fd0-b118-bcb8d524bef0-home\") pod \"e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46\" (UID: \"217c78d8-f167-4fd0-b118-bcb8d524bef0\") " pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:24:15.345864 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.345793    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/217c78d8-f167-4fd0-b118-bcb8d524bef0-dshm\") pod \"e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46\" (UID: \"217c78d8-f167-4fd0-b118-bcb8d524bef0\") " pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:24:15.345864 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.345859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/217c78d8-f167-4fd0-b118-bcb8d524bef0-tmp-dir\") pod \"e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46\" (UID: \"217c78d8-f167-4fd0-b118-bcb8d524bef0\") " pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:24:15.346087 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.345887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/217c78d8-f167-4fd0-b118-bcb8d524bef0-tls-certs\") pod \"e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46\" (UID: \"217c78d8-f167-4fd0-b118-bcb8d524bef0\") " pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:24:15.346087 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.345915    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/217c78d8-f167-4fd0-b118-bcb8d524bef0-home\") pod \"e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46\" (UID: \"217c78d8-f167-4fd0-b118-bcb8d524bef0\") " pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:24:15.346087 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.345984    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/217c78d8-f167-4fd0-b118-bcb8d524bef0-model-cache\") pod \"e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46\" (UID: \"217c78d8-f167-4fd0-b118-bcb8d524bef0\") " pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:24:15.346087 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.346003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmvdj\" (UniqueName: \"kubernetes.io/projected/217c78d8-f167-4fd0-b118-bcb8d524bef0-kube-api-access-zmvdj\") pod \"e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46\" (UID: \"217c78d8-f167-4fd0-b118-bcb8d524bef0\") " pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:24:15.346370 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.346339    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/217c78d8-f167-4fd0-b118-bcb8d524bef0-tmp-dir\") pod \"e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46\" (UID: \"217c78d8-f167-4fd0-b118-bcb8d524bef0\") " pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:24:15.346485 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.346377    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/217c78d8-f167-4fd0-b118-bcb8d524bef0-model-cache\") pod \"e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46\" (UID: \"217c78d8-f167-4fd0-b118-bcb8d524bef0\") " pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:24:15.346485 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.346406    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/217c78d8-f167-4fd0-b118-bcb8d524bef0-home\") pod \"e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46\" (UID: \"217c78d8-f167-4fd0-b118-bcb8d524bef0\") " pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:24:15.348165 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.348145    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/217c78d8-f167-4fd0-b118-bcb8d524bef0-dshm\") pod \"e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46\" (UID: \"217c78d8-f167-4fd0-b118-bcb8d524bef0\") " pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:24:15.348260 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.348238    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/217c78d8-f167-4fd0-b118-bcb8d524bef0-tls-certs\") pod \"e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46\" (UID: \"217c78d8-f167-4fd0-b118-bcb8d524bef0\") " pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:24:15.354287 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.354266    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmvdj\" (UniqueName: \"kubernetes.io/projected/217c78d8-f167-4fd0-b118-bcb8d524bef0-kube-api-access-zmvdj\") pod \"e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46\" (UID: \"217c78d8-f167-4fd0-b118-bcb8d524bef0\") " pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:24:15.430780 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.430747    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:24:15.700625 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.700598    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"]
Sep 11 21:24:15.702754 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:24:15.702723    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod217c78d8_f167_4fd0_b118_bcb8d524bef0.slice/crio-bc806fd34e9c12a038036473efcc3666d45a65dc42c07ebe30ada393be618604 WatchSource:0}: Error finding container bc806fd34e9c12a038036473efcc3666d45a65dc42c07ebe30ada393be618604: Status 404 returned error can't find the container with id bc806fd34e9c12a038036473efcc3666d45a65dc42c07ebe30ada393be618604
Sep 11 21:24:15.883775 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.883741    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46" event={"ID":"217c78d8-f167-4fd0-b118-bcb8d524bef0","Type":"ContainerStarted","Data":"1de69bee65148a8ef8d66063f4df3670dcfc7b82dc2d95b69ee26408306e35e0"}
Sep 11 21:24:15.883969 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.883789    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46" event={"ID":"217c78d8-f167-4fd0-b118-bcb8d524bef0","Type":"ContainerStarted","Data":"bc806fd34e9c12a038036473efcc3666d45a65dc42c07ebe30ada393be618604"}
Sep 11 21:24:15.902303 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:15.902254    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46" podStartSLOduration=0.90223614 podStartE2EDuration="902.23614ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:24:15.900559351 +0000 UTC m=+1122.604528641" watchObservedRunningTime="2026-09-11 21:24:15.90223614 +0000 UTC m=+1122.606205428"
Sep 11 21:24:20.859884 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:20.859851    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb"
Sep 11 21:24:25.431735 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:25.431697    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:24:25.432176 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:25.431772    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:24:25.444235 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:25.444209    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:24:25.941029 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:25.940997    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:24:29.559032 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:29.558992    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:24:34.088330 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:34.088298    2578 scope.go:117] "RemoveContainer" containerID="57a149534803a00b53dd2d0bc300762d6da9c915ead78f32ac45b5600c816fca"
Sep 11 21:24:34.097331 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:34.097306    2578 scope.go:117] "RemoveContainer" containerID="4614e5f9c432316ae8271c6803a0f256dc0be148feb05927e23eb36a84f79210"
Sep 11 21:24:43.927293 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:43.927261    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:24:52.717364 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.717325    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"]
Sep 11 21:24:52.722054 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.722026    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:24:52.724704 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.724681    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2bd772\"/\"kube-root-ca.crt\""
Sep 11 21:24:52.725431 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.725411    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2bd772\"/\"openshift-service-ca.crt\""
Sep 11 21:24:52.725490 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.725426    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2bd772\"/\"default-dockercfg-fk6bh\""
Sep 11 21:24:52.726042 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.726019    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2bd772\"/\"auth-test-model-2bd772-kserve-self-signed-certs\""
Sep 11 21:24:52.733395 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.733371    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"]
Sep 11 21:24:52.753634 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.753598    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj255\" (UniqueName: \"kubernetes.io/projected/e34bde6e-b402-4be5-af5d-e0f16a3efabe-kube-api-access-xj255\") pod \"auth-test-model-2bd772-kserve-c956569b7-wvgwc\" (UID: \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:24:52.753794 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.753707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e34bde6e-b402-4be5-af5d-e0f16a3efabe-tmp-dir\") pod \"auth-test-model-2bd772-kserve-c956569b7-wvgwc\" (UID: \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:24:52.753864 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.753805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e34bde6e-b402-4be5-af5d-e0f16a3efabe-dshm\") pod \"auth-test-model-2bd772-kserve-c956569b7-wvgwc\" (UID: \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:24:52.753916 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.753902    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e34bde6e-b402-4be5-af5d-e0f16a3efabe-tls-certs\") pod \"auth-test-model-2bd772-kserve-c956569b7-wvgwc\" (UID: \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:24:52.753952 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.753930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e34bde6e-b402-4be5-af5d-e0f16a3efabe-home\") pod \"auth-test-model-2bd772-kserve-c956569b7-wvgwc\" (UID: \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:24:52.753994 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.753952    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e34bde6e-b402-4be5-af5d-e0f16a3efabe-model-cache\") pod \"auth-test-model-2bd772-kserve-c956569b7-wvgwc\" (UID: \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:24:52.855112 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.855069    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj255\" (UniqueName: \"kubernetes.io/projected/e34bde6e-b402-4be5-af5d-e0f16a3efabe-kube-api-access-xj255\") pod \"auth-test-model-2bd772-kserve-c956569b7-wvgwc\" (UID: \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:24:52.855319 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.855123    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e34bde6e-b402-4be5-af5d-e0f16a3efabe-tmp-dir\") pod \"auth-test-model-2bd772-kserve-c956569b7-wvgwc\" (UID: \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:24:52.855319 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.855175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e34bde6e-b402-4be5-af5d-e0f16a3efabe-dshm\") pod \"auth-test-model-2bd772-kserve-c956569b7-wvgwc\" (UID: \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:24:52.855319 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.855231    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e34bde6e-b402-4be5-af5d-e0f16a3efabe-tls-certs\") pod \"auth-test-model-2bd772-kserve-c956569b7-wvgwc\" (UID: \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:24:52.855319 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.855255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e34bde6e-b402-4be5-af5d-e0f16a3efabe-home\") pod \"auth-test-model-2bd772-kserve-c956569b7-wvgwc\" (UID: \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:24:52.855319 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.855283    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e34bde6e-b402-4be5-af5d-e0f16a3efabe-model-cache\") pod \"auth-test-model-2bd772-kserve-c956569b7-wvgwc\" (UID: \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:24:52.855639 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.855615    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e34bde6e-b402-4be5-af5d-e0f16a3efabe-tmp-dir\") pod \"auth-test-model-2bd772-kserve-c956569b7-wvgwc\" (UID: \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:24:52.855710 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.855650    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e34bde6e-b402-4be5-af5d-e0f16a3efabe-home\") pod \"auth-test-model-2bd772-kserve-c956569b7-wvgwc\" (UID: \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:24:52.855765 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.855705    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e34bde6e-b402-4be5-af5d-e0f16a3efabe-model-cache\") pod \"auth-test-model-2bd772-kserve-c956569b7-wvgwc\" (UID: \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:24:52.857499 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.857478    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e34bde6e-b402-4be5-af5d-e0f16a3efabe-dshm\") pod \"auth-test-model-2bd772-kserve-c956569b7-wvgwc\" (UID: \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:24:52.857731 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.857715    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e34bde6e-b402-4be5-af5d-e0f16a3efabe-tls-certs\") pod \"auth-test-model-2bd772-kserve-c956569b7-wvgwc\" (UID: \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:24:52.866848 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:52.866814    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj255\" (UniqueName: \"kubernetes.io/projected/e34bde6e-b402-4be5-af5d-e0f16a3efabe-kube-api-access-xj255\") pod \"auth-test-model-2bd772-kserve-c956569b7-wvgwc\" (UID: \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:24:53.032444 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:53.032359    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:24:53.329902 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:53.329810    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"]
Sep 11 21:24:53.333740 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:24:53.333708    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode34bde6e_b402_4be5_af5d_e0f16a3efabe.slice/crio-3d9b878e620b854764dc42f523c50e218f9ab576ee19c64b076e5d3738d50542 WatchSource:0}: Error finding container 3d9b878e620b854764dc42f523c50e218f9ab576ee19c64b076e5d3738d50542: Status 404 returned error can't find the container with id 3d9b878e620b854764dc42f523c50e218f9ab576ee19c64b076e5d3738d50542
Sep 11 21:24:54.059428 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:54.059400    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc" event={"ID":"e34bde6e-b402-4be5-af5d-e0f16a3efabe","Type":"ContainerStarted","Data":"48cbad06f9f2a08f1a4a7df064952f0b66c3ed0eb3f4ed361c3e31f493ac6f54"}
Sep 11 21:24:54.059813 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:54.059436    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc" event={"ID":"e34bde6e-b402-4be5-af5d-e0f16a3efabe","Type":"ContainerStarted","Data":"3d9b878e620b854764dc42f523c50e218f9ab576ee19c64b076e5d3738d50542"}
Sep 11 21:24:54.078323 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:54.078189    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc" podStartSLOduration=2.078170432 podStartE2EDuration="2.078170432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:24:54.07797774 +0000 UTC m=+1160.781947031" watchObservedRunningTime="2026-09-11 21:24:54.078170432 +0000 UTC m=+1160.782139738"
Sep 11 21:24:56.025610 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:56.025529    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj"]
Sep 11 21:24:56.026066 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:56.025760    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj" podUID="055c5601-2aae-442a-932f-ada1ae86ba83" containerName="payload-pre-processing" containerID="cri-o://335874d1a7887b548ff8602cb3f19a0413c87877858a2a6adb9d0a20fa2ba8f9" gracePeriod=30
Sep 11 21:24:56.273514 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:56.273488    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj"
Sep 11 21:24:56.386787 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:56.386699    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/055c5601-2aae-442a-932f-ada1ae86ba83-plugins-config-volume\") pod \"055c5601-2aae-442a-932f-ada1ae86ba83\" (UID: \"055c5601-2aae-442a-932f-ada1ae86ba83\") "
Sep 11 21:24:56.386787 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:56.386745    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcc7l\" (UniqueName: \"kubernetes.io/projected/055c5601-2aae-442a-932f-ada1ae86ba83-kube-api-access-zcc7l\") pod \"055c5601-2aae-442a-932f-ada1ae86ba83\" (UID: \"055c5601-2aae-442a-932f-ada1ae86ba83\") "
Sep 11 21:24:56.387213 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:56.387181    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/055c5601-2aae-442a-932f-ada1ae86ba83-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "055c5601-2aae-442a-932f-ada1ae86ba83" (UID: "055c5601-2aae-442a-932f-ada1ae86ba83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:24:56.389017 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:56.388994    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/055c5601-2aae-442a-932f-ada1ae86ba83-kube-api-access-zcc7l" (OuterVolumeSpecName: "kube-api-access-zcc7l") pod "055c5601-2aae-442a-932f-ada1ae86ba83" (UID: "055c5601-2aae-442a-932f-ada1ae86ba83"). InnerVolumeSpecName "kube-api-access-zcc7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:24:56.488111 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:56.488062    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/055c5601-2aae-442a-932f-ada1ae86ba83-plugins-config-volume\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:24:56.488111 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:56.488104    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcc7l\" (UniqueName: \"kubernetes.io/projected/055c5601-2aae-442a-932f-ada1ae86ba83-kube-api-access-zcc7l\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:24:57.073649 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:57.073620    2578 generic.go:356] "Generic (PLEG): container finished" podID="055c5601-2aae-442a-932f-ada1ae86ba83" containerID="335874d1a7887b548ff8602cb3f19a0413c87877858a2a6adb9d0a20fa2ba8f9" exitCode=0
Sep 11 21:24:57.074068 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:57.073667    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj"
Sep 11 21:24:57.074068 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:57.073718    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj" event={"ID":"055c5601-2aae-442a-932f-ada1ae86ba83","Type":"ContainerDied","Data":"335874d1a7887b548ff8602cb3f19a0413c87877858a2a6adb9d0a20fa2ba8f9"}
Sep 11 21:24:57.074068 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:57.073770    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj" event={"ID":"055c5601-2aae-442a-932f-ada1ae86ba83","Type":"ContainerDied","Data":"1def349e64fcadd77ee7a9158377184c519132b8f62af454de99275e354fa869"}
Sep 11 21:24:57.074068 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:57.073792    2578 scope.go:117] "RemoveContainer" containerID="335874d1a7887b548ff8602cb3f19a0413c87877858a2a6adb9d0a20fa2ba8f9"
Sep 11 21:24:57.082547 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:57.082531    2578 scope.go:117] "RemoveContainer" containerID="335874d1a7887b548ff8602cb3f19a0413c87877858a2a6adb9d0a20fa2ba8f9"
Sep 11 21:24:57.082807 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:24:57.082789    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"335874d1a7887b548ff8602cb3f19a0413c87877858a2a6adb9d0a20fa2ba8f9\": container with ID starting with 335874d1a7887b548ff8602cb3f19a0413c87877858a2a6adb9d0a20fa2ba8f9 not found: ID does not exist" containerID="335874d1a7887b548ff8602cb3f19a0413c87877858a2a6adb9d0a20fa2ba8f9"
Sep 11 21:24:57.082876 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:57.082835    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"335874d1a7887b548ff8602cb3f19a0413c87877858a2a6adb9d0a20fa2ba8f9"} err="failed to get container status \"335874d1a7887b548ff8602cb3f19a0413c87877858a2a6adb9d0a20fa2ba8f9\": rpc error: code = NotFound desc = could not find container \"335874d1a7887b548ff8602cb3f19a0413c87877858a2a6adb9d0a20fa2ba8f9\": container with ID starting with 335874d1a7887b548ff8602cb3f19a0413c87877858a2a6adb9d0a20fa2ba8f9 not found: ID does not exist"
Sep 11 21:24:57.093860 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:57.093838    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj"]
Sep 11 21:24:57.096457 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:57.096433    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-908a1435-67f77f65f8-dqkhj"]
Sep 11 21:24:57.927055 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:57.927024    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="055c5601-2aae-442a-932f-ada1ae86ba83" path="/var/lib/kubelet/pods/055c5601-2aae-442a-932f-ada1ae86ba83/volumes"
Sep 11 21:24:59.735373 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:24:59.735339    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:25:03.032659 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:03.032625    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:25:03.033114 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:03.032913    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:25:03.045069 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:03.045042    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:25:03.112568 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:03.112539    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:25:04.419172 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:04.419139    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"]
Sep 11 21:25:19.437751 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:19.437694    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46" podUID="217c78d8-f167-4fd0-b118-bcb8d524bef0" containerName="main" containerID="cri-o://1de69bee65148a8ef8d66063f4df3670dcfc7b82dc2d95b69ee26408306e35e0" gracePeriod=45
Sep 11 21:25:19.681429 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:19.681402    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:25:19.791644 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:19.791558    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/217c78d8-f167-4fd0-b118-bcb8d524bef0-tls-certs\") pod \"217c78d8-f167-4fd0-b118-bcb8d524bef0\" (UID: \"217c78d8-f167-4fd0-b118-bcb8d524bef0\") "
Sep 11 21:25:19.791644 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:19.791602    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/217c78d8-f167-4fd0-b118-bcb8d524bef0-model-cache\") pod \"217c78d8-f167-4fd0-b118-bcb8d524bef0\" (UID: \"217c78d8-f167-4fd0-b118-bcb8d524bef0\") "
Sep 11 21:25:19.791904 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:19.791734    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/217c78d8-f167-4fd0-b118-bcb8d524bef0-tmp-dir\") pod \"217c78d8-f167-4fd0-b118-bcb8d524bef0\" (UID: \"217c78d8-f167-4fd0-b118-bcb8d524bef0\") "
Sep 11 21:25:19.791904 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:19.791794    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmvdj\" (UniqueName: \"kubernetes.io/projected/217c78d8-f167-4fd0-b118-bcb8d524bef0-kube-api-access-zmvdj\") pod \"217c78d8-f167-4fd0-b118-bcb8d524bef0\" (UID: \"217c78d8-f167-4fd0-b118-bcb8d524bef0\") "
Sep 11 21:25:19.791904 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:19.791869    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/217c78d8-f167-4fd0-b118-bcb8d524bef0-dshm\") pod \"217c78d8-f167-4fd0-b118-bcb8d524bef0\" (UID: \"217c78d8-f167-4fd0-b118-bcb8d524bef0\") "
Sep 11 21:25:19.791904 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:19.791896    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/217c78d8-f167-4fd0-b118-bcb8d524bef0-home\") pod \"217c78d8-f167-4fd0-b118-bcb8d524bef0\" (UID: \"217c78d8-f167-4fd0-b118-bcb8d524bef0\") "
Sep 11 21:25:19.792114 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:19.791865    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/217c78d8-f167-4fd0-b118-bcb8d524bef0-model-cache" (OuterVolumeSpecName: "model-cache") pod "217c78d8-f167-4fd0-b118-bcb8d524bef0" (UID: "217c78d8-f167-4fd0-b118-bcb8d524bef0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:25:19.792114 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:19.791973    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/217c78d8-f167-4fd0-b118-bcb8d524bef0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "217c78d8-f167-4fd0-b118-bcb8d524bef0" (UID: "217c78d8-f167-4fd0-b118-bcb8d524bef0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:25:19.792217 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:19.792180    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/217c78d8-f167-4fd0-b118-bcb8d524bef0-home" (OuterVolumeSpecName: "home") pod "217c78d8-f167-4fd0-b118-bcb8d524bef0" (UID: "217c78d8-f167-4fd0-b118-bcb8d524bef0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:25:19.792258 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:19.792213    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/217c78d8-f167-4fd0-b118-bcb8d524bef0-model-cache\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:25:19.792258 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:19.792229    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/217c78d8-f167-4fd0-b118-bcb8d524bef0-tmp-dir\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:25:19.793792 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:19.793768    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/217c78d8-f167-4fd0-b118-bcb8d524bef0-dshm" (OuterVolumeSpecName: "dshm") pod "217c78d8-f167-4fd0-b118-bcb8d524bef0" (UID: "217c78d8-f167-4fd0-b118-bcb8d524bef0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:25:19.793792 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:19.793781    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/217c78d8-f167-4fd0-b118-bcb8d524bef0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "217c78d8-f167-4fd0-b118-bcb8d524bef0" (UID: "217c78d8-f167-4fd0-b118-bcb8d524bef0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:25:19.793792 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:19.793791    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/217c78d8-f167-4fd0-b118-bcb8d524bef0-kube-api-access-zmvdj" (OuterVolumeSpecName: "kube-api-access-zmvdj") pod "217c78d8-f167-4fd0-b118-bcb8d524bef0" (UID: "217c78d8-f167-4fd0-b118-bcb8d524bef0"). InnerVolumeSpecName "kube-api-access-zmvdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:25:19.893511 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:19.893464    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmvdj\" (UniqueName: \"kubernetes.io/projected/217c78d8-f167-4fd0-b118-bcb8d524bef0-kube-api-access-zmvdj\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:25:19.893511 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:19.893506    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/217c78d8-f167-4fd0-b118-bcb8d524bef0-dshm\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:25:19.893511 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:19.893515    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/217c78d8-f167-4fd0-b118-bcb8d524bef0-home\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:25:19.893511 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:19.893523    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/217c78d8-f167-4fd0-b118-bcb8d524bef0-tls-certs\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:25:20.175228 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:20.175196    2578 generic.go:356] "Generic (PLEG): container finished" podID="217c78d8-f167-4fd0-b118-bcb8d524bef0" containerID="1de69bee65148a8ef8d66063f4df3670dcfc7b82dc2d95b69ee26408306e35e0" exitCode=0
Sep 11 21:25:20.175404 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:20.175236    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"
Sep 11 21:25:20.175404 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:20.175230    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46" event={"ID":"217c78d8-f167-4fd0-b118-bcb8d524bef0","Type":"ContainerDied","Data":"1de69bee65148a8ef8d66063f4df3670dcfc7b82dc2d95b69ee26408306e35e0"}
Sep 11 21:25:20.175404 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:20.175344    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46" event={"ID":"217c78d8-f167-4fd0-b118-bcb8d524bef0","Type":"ContainerDied","Data":"bc806fd34e9c12a038036473efcc3666d45a65dc42c07ebe30ada393be618604"}
Sep 11 21:25:20.175404 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:20.175364    2578 scope.go:117] "RemoveContainer" containerID="1de69bee65148a8ef8d66063f4df3670dcfc7b82dc2d95b69ee26408306e35e0"
Sep 11 21:25:20.183556 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:20.183536    2578 scope.go:117] "RemoveContainer" containerID="1de69bee65148a8ef8d66063f4df3670dcfc7b82dc2d95b69ee26408306e35e0"
Sep 11 21:25:20.183870 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:25:20.183848    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de69bee65148a8ef8d66063f4df3670dcfc7b82dc2d95b69ee26408306e35e0\": container with ID starting with 1de69bee65148a8ef8d66063f4df3670dcfc7b82dc2d95b69ee26408306e35e0 not found: ID does not exist" containerID="1de69bee65148a8ef8d66063f4df3670dcfc7b82dc2d95b69ee26408306e35e0"
Sep 11 21:25:20.183947 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:20.183883    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de69bee65148a8ef8d66063f4df3670dcfc7b82dc2d95b69ee26408306e35e0"} err="failed to get container status \"1de69bee65148a8ef8d66063f4df3670dcfc7b82dc2d95b69ee26408306e35e0\": rpc error: code = NotFound desc = could not find container \"1de69bee65148a8ef8d66063f4df3670dcfc7b82dc2d95b69ee26408306e35e0\": container with ID starting with 1de69bee65148a8ef8d66063f4df3670dcfc7b82dc2d95b69ee26408306e35e0 not found: ID does not exist"
Sep 11 21:25:20.195647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:20.195616    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"]
Sep 11 21:25:20.203762 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:20.203735    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-908a1435/e2e-lifecycle-model-908a1435-kserve-7d75d5df4c-stq46"]
Sep 11 21:25:21.927438 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:21.927405    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="217c78d8-f167-4fd0-b118-bcb8d524bef0" path="/var/lib/kubelet/pods/217c78d8-f167-4fd0-b118-bcb8d524bef0/volumes"
Sep 11 21:25:22.924712 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:22.924678    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:25:41.747126 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:41.747089    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:25:59.714297 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:25:59.714263    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:26:17.036057 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:26:17.036020    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:26:32.829863 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:26:32.829773    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:26:34.131605 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:26:34.131579    2578 scope.go:117] "RemoveContainer" containerID="bc0138289003f6373df76c3d533377cf101296b7e41134aa1e1d513bd1591ce1"
Sep 11 21:26:49.032448 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:26:49.032417    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:27:06.824720 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:06.824686    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-354c80-bfd69945c-l2phv"]
Sep 11 21:27:06.825181 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:06.825099    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="055c5601-2aae-442a-932f-ada1ae86ba83" containerName="payload-pre-processing"
Sep 11 21:27:06.825181 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:06.825113    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="055c5601-2aae-442a-932f-ada1ae86ba83" containerName="payload-pre-processing"
Sep 11 21:27:06.825181 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:06.825135    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="217c78d8-f167-4fd0-b118-bcb8d524bef0" containerName="main"
Sep 11 21:27:06.825181 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:06.825141    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="217c78d8-f167-4fd0-b118-bcb8d524bef0" containerName="main"
Sep 11 21:27:06.825314 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:06.825198    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="217c78d8-f167-4fd0-b118-bcb8d524bef0" containerName="main"
Sep 11 21:27:06.825314 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:06.825211    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="055c5601-2aae-442a-932f-ada1ae86ba83" containerName="payload-pre-processing"
Sep 11 21:27:06.828116 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:06.828099    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-354c80-bfd69945c-l2phv"
Sep 11 21:27:06.834208 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:06.834068    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-354c80\""
Sep 11 21:27:06.834208 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:06.834135    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-354c80-dockercfg-4mzd7\""
Sep 11 21:27:06.856240 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:06.856218    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-354c80-bfd69945c-l2phv"]
Sep 11 21:27:06.957234 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:06.957200    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm65q\" (UniqueName: \"kubernetes.io/projected/74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4-kube-api-access-sm65q\") pod \"payload-processing-e2e-api-a-354c80-bfd69945c-l2phv\" (UID: \"74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4\") " pod="openshift-ingress/payload-processing-e2e-api-a-354c80-bfd69945c-l2phv"
Sep 11 21:27:06.957234 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:06.957239    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-354c80-bfd69945c-l2phv\" (UID: \"74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4\") " pod="openshift-ingress/payload-processing-e2e-api-a-354c80-bfd69945c-l2phv"
Sep 11 21:27:07.058129 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:07.058094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm65q\" (UniqueName: \"kubernetes.io/projected/74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4-kube-api-access-sm65q\") pod \"payload-processing-e2e-api-a-354c80-bfd69945c-l2phv\" (UID: \"74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4\") " pod="openshift-ingress/payload-processing-e2e-api-a-354c80-bfd69945c-l2phv"
Sep 11 21:27:07.058129 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:07.058133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-354c80-bfd69945c-l2phv\" (UID: \"74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4\") " pod="openshift-ingress/payload-processing-e2e-api-a-354c80-bfd69945c-l2phv"
Sep 11 21:27:07.058801 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:07.058784    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-354c80-bfd69945c-l2phv\" (UID: \"74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4\") " pod="openshift-ingress/payload-processing-e2e-api-a-354c80-bfd69945c-l2phv"
Sep 11 21:27:07.068003 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:07.067972    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm65q\" (UniqueName: \"kubernetes.io/projected/74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4-kube-api-access-sm65q\") pod \"payload-processing-e2e-api-a-354c80-bfd69945c-l2phv\" (UID: \"74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4\") " pod="openshift-ingress/payload-processing-e2e-api-a-354c80-bfd69945c-l2phv"
Sep 11 21:27:07.137329 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:07.137257    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-354c80-bfd69945c-l2phv"
Sep 11 21:27:07.410628 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:07.410600    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-354c80-bfd69945c-l2phv"]
Sep 11 21:27:07.411974 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:27:07.411946    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74dc0372_3cd9_4e39_bbc2_6853e7d9f4a4.slice/crio-70d6e66c25410314e534477f667d9fb949837ba3c751786b7a2b090c44859f46 WatchSource:0}: Error finding container 70d6e66c25410314e534477f667d9fb949837ba3c751786b7a2b090c44859f46: Status 404 returned error can't find the container with id 70d6e66c25410314e534477f667d9fb949837ba3c751786b7a2b090c44859f46
Sep 11 21:27:07.645562 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:07.645530    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-354c80-bfd69945c-l2phv" event={"ID":"74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4","Type":"ContainerStarted","Data":"d20f0334da52116169e4307a180eef0951409dc17434d754c59125e05f4fb79a"}
Sep 11 21:27:07.645751 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:07.645581    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-354c80-bfd69945c-l2phv" event={"ID":"74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4","Type":"ContainerStarted","Data":"70d6e66c25410314e534477f667d9fb949837ba3c751786b7a2b090c44859f46"}
Sep 11 21:27:07.645751 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:07.645609    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-354c80-bfd69945c-l2phv"
Sep 11 21:27:07.665582 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:07.665487    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-354c80-bfd69945c-l2phv" podStartSLOduration=1.6654720790000002 podStartE2EDuration="1.665472079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:27:07.664105849 +0000 UTC m=+1294.368075204" watchObservedRunningTime="2026-09-11 21:27:07.665472079 +0000 UTC m=+1294.369441367"
Sep 11 21:27:07.842041 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:07.842001    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:27:18.652242 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:18.652207    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-354c80-bfd69945c-l2phv"
Sep 11 21:27:27.443341 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:27.443306    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:27:30.974955 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:30.974921    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7702-f3aac8"]
Sep 11 21:27:30.978222 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:30.978204    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7702-f3aac8"
Sep 11 21:27:30.981916 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:30.981890    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6dwfw\""
Sep 11 21:27:31.000747 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:31.000721    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7702-f3aac8"]
Sep 11 21:27:31.067187 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:31.067151    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs64w\" (UniqueName: \"kubernetes.io/projected/51e208fd-0ab9-46e8-8fd8-17767f0957af-kube-api-access-hs64w\") pod \"test-curl-7702-f3aac8\" (UID: \"51e208fd-0ab9-46e8-8fd8-17767f0957af\") " pod="opendatahub/test-curl-7702-f3aac8"
Sep 11 21:27:31.168700 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:31.168668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs64w\" (UniqueName: \"kubernetes.io/projected/51e208fd-0ab9-46e8-8fd8-17767f0957af-kube-api-access-hs64w\") pod \"test-curl-7702-f3aac8\" (UID: \"51e208fd-0ab9-46e8-8fd8-17767f0957af\") " pod="opendatahub/test-curl-7702-f3aac8"
Sep 11 21:27:31.188187 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:31.188160    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs64w\" (UniqueName: \"kubernetes.io/projected/51e208fd-0ab9-46e8-8fd8-17767f0957af-kube-api-access-hs64w\") pod \"test-curl-7702-f3aac8\" (UID: \"51e208fd-0ab9-46e8-8fd8-17767f0957af\") " pod="opendatahub/test-curl-7702-f3aac8"
Sep 11 21:27:31.288074 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:31.287994    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7702-f3aac8"
Sep 11 21:27:31.563294 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:31.563269    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7702-f3aac8"]
Sep 11 21:27:31.565795 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:27:31.565771    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51e208fd_0ab9_46e8_8fd8_17767f0957af.slice/crio-b7d5baa8f327e7891e376ef4e7082554570de5272b47e70211626064c622d52d WatchSource:0}: Error finding container b7d5baa8f327e7891e376ef4e7082554570de5272b47e70211626064c622d52d: Status 404 returned error can't find the container with id b7d5baa8f327e7891e376ef4e7082554570de5272b47e70211626064c622d52d
Sep 11 21:27:31.757031 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:31.756998    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7702-f3aac8" event={"ID":"51e208fd-0ab9-46e8-8fd8-17767f0957af","Type":"ContainerStarted","Data":"2b863d61443fb0f26a3cfb21f2651ec0932064ff3e8161f2a4b062fd74a1a1b8"}
Sep 11 21:27:31.757222 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:31.757047    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7702-f3aac8" event={"ID":"51e208fd-0ab9-46e8-8fd8-17767f0957af","Type":"ContainerStarted","Data":"b7d5baa8f327e7891e376ef4e7082554570de5272b47e70211626064c622d52d"}
Sep 11 21:27:31.811920 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:31.811845    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7702-f3aac8" podStartSLOduration=1.8118091619999999 podStartE2EDuration="1.811809162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:27:31.809133138 +0000 UTC m=+1318.513102425" watchObservedRunningTime="2026-09-11 21:27:31.811809162 +0000 UTC m=+1318.515778451"
Sep 11 21:27:32.598778 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:32.598693    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7702-f3aac8"]
Sep 11 21:27:32.609883 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:32.609853    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7702-f3aac8"]
Sep 11 21:27:32.761483 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:27:32.761453    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7702-f3aac8" container="test-curl-7702-f3aac8"
Sep 11 21:27:33.765960 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:33.765923    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7702-f3aac8" podUID="51e208fd-0ab9-46e8-8fd8-17767f0957af" containerName="test-curl-7702-f3aac8" containerID="cri-o://2b863d61443fb0f26a3cfb21f2651ec0932064ff3e8161f2a4b062fd74a1a1b8" gracePeriod=2
Sep 11 21:27:33.768682 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:27:33.768657    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7702-f3aac8\" is forbidden: User \"system:node:ip-10-0-142-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-224.ec2.internal' and this object" podUID="51e208fd-0ab9-46e8-8fd8-17767f0957af" pod="opendatahub/test-curl-7702-f3aac8"
Sep 11 21:27:33.927159 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:27:33.927128    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7702-f3aac8\" is forbidden: User \"system:node:ip-10-0-142-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-224.ec2.internal' and this object" podUID="51e208fd-0ab9-46e8-8fd8-17767f0957af" pod="opendatahub/test-curl-7702-f3aac8"
Sep 11 21:27:34.855154 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:34.855121    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"]
Sep 11 21:27:35.908410 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:35.908386    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7702-f3aac8"
Sep 11 21:27:36.008780 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:36.008745    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs64w\" (UniqueName: \"kubernetes.io/projected/51e208fd-0ab9-46e8-8fd8-17767f0957af-kube-api-access-hs64w\") pod \"51e208fd-0ab9-46e8-8fd8-17767f0957af\" (UID: \"51e208fd-0ab9-46e8-8fd8-17767f0957af\") "
Sep 11 21:27:36.010764 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:36.010740    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e208fd-0ab9-46e8-8fd8-17767f0957af-kube-api-access-hs64w" (OuterVolumeSpecName: "kube-api-access-hs64w") pod "51e208fd-0ab9-46e8-8fd8-17767f0957af" (UID: "51e208fd-0ab9-46e8-8fd8-17767f0957af"). InnerVolumeSpecName "kube-api-access-hs64w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:27:36.109717 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:36.109644    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs64w\" (UniqueName: \"kubernetes.io/projected/51e208fd-0ab9-46e8-8fd8-17767f0957af-kube-api-access-hs64w\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:27:36.783797 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:36.783758    2578 generic.go:356] "Generic (PLEG): container finished" podID="51e208fd-0ab9-46e8-8fd8-17767f0957af" containerID="2b863d61443fb0f26a3cfb21f2651ec0932064ff3e8161f2a4b062fd74a1a1b8" exitCode=137
Sep 11 21:27:36.784031 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:36.783800    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7702-f3aac8"
Sep 11 21:27:36.784031 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:36.783853    2578 scope.go:117] "RemoveContainer" containerID="2b863d61443fb0f26a3cfb21f2651ec0932064ff3e8161f2a4b062fd74a1a1b8"
Sep 11 21:27:36.792483 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:36.792463    2578 scope.go:117] "RemoveContainer" containerID="2b863d61443fb0f26a3cfb21f2651ec0932064ff3e8161f2a4b062fd74a1a1b8"
Sep 11 21:27:36.792724 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:27:36.792704    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b863d61443fb0f26a3cfb21f2651ec0932064ff3e8161f2a4b062fd74a1a1b8\": container with ID starting with 2b863d61443fb0f26a3cfb21f2651ec0932064ff3e8161f2a4b062fd74a1a1b8 not found: ID does not exist" containerID="2b863d61443fb0f26a3cfb21f2651ec0932064ff3e8161f2a4b062fd74a1a1b8"
Sep 11 21:27:36.792789 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:36.792730    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b863d61443fb0f26a3cfb21f2651ec0932064ff3e8161f2a4b062fd74a1a1b8"} err="failed to get container status \"2b863d61443fb0f26a3cfb21f2651ec0932064ff3e8161f2a4b062fd74a1a1b8\": rpc error: code = NotFound desc = could not find container \"2b863d61443fb0f26a3cfb21f2651ec0932064ff3e8161f2a4b062fd74a1a1b8\": container with ID starting with 2b863d61443fb0f26a3cfb21f2651ec0932064ff3e8161f2a4b062fd74a1a1b8 not found: ID does not exist"
Sep 11 21:27:37.927198 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:37.927167    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51e208fd-0ab9-46e8-8fd8-17767f0957af" path="/var/lib/kubelet/pods/51e208fd-0ab9-46e8-8fd8-17767f0957af/volumes"
Sep 11 21:27:42.606456 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:42.606420    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp"]
Sep 11 21:27:42.606848 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:42.606740    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51e208fd-0ab9-46e8-8fd8-17767f0957af" containerName="test-curl-7702-f3aac8"
Sep 11 21:27:42.606848 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:42.606750    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="51e208fd-0ab9-46e8-8fd8-17767f0957af" containerName="test-curl-7702-f3aac8"
Sep 11 21:27:42.606848 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:42.606813    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="51e208fd-0ab9-46e8-8fd8-17767f0957af" containerName="test-curl-7702-f3aac8"
Sep 11 21:27:42.611132 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:42.611115    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp"
Sep 11 21:27:42.613901 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:42.613874    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2748c5\""
Sep 11 21:27:42.614236 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:42.614218    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2748c5-dockercfg-7zm46\""
Sep 11 21:27:42.624698 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:42.624677    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp"]
Sep 11 21:27:42.662991 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:42.662958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b2141f-0bab-4ce3-81a3-1b95d539d2fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp\" (UID: \"c1b2141f-0bab-4ce3-81a3-1b95d539d2fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp"
Sep 11 21:27:42.663128 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:42.663020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-665c4\" (UniqueName: \"kubernetes.io/projected/c1b2141f-0bab-4ce3-81a3-1b95d539d2fb-kube-api-access-665c4\") pod \"payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp\" (UID: \"c1b2141f-0bab-4ce3-81a3-1b95d539d2fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp"
Sep 11 21:27:42.703851 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:42.703808    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2748c5-f65968655-trwmp"]
Sep 11 21:27:42.707050 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:42.707036    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2748c5-f65968655-trwmp"
Sep 11 21:27:42.725964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:42.725935    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2748c5-f65968655-trwmp"]
Sep 11 21:27:42.764081 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:42.764054    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qklrx\" (UniqueName: \"kubernetes.io/projected/6cf51d65-3459-42b8-b86a-a253a4ca5d8e-kube-api-access-qklrx\") pod \"payload-processing-e2e-api-b-2748c5-f65968655-trwmp\" (UID: \"6cf51d65-3459-42b8-b86a-a253a4ca5d8e\") " pod="openshift-ingress/payload-processing-e2e-api-b-2748c5-f65968655-trwmp"
Sep 11 21:27:42.764251 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:42.764096    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b2141f-0bab-4ce3-81a3-1b95d539d2fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp\" (UID: \"c1b2141f-0bab-4ce3-81a3-1b95d539d2fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp"
Sep 11 21:27:42.764251 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:42.764182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cf51d65-3459-42b8-b86a-a253a4ca5d8e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2748c5-f65968655-trwmp\" (UID: \"6cf51d65-3459-42b8-b86a-a253a4ca5d8e\") " pod="openshift-ingress/payload-processing-e2e-api-b-2748c5-f65968655-trwmp"
Sep 11 21:27:42.764251 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:42.764236    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-665c4\" (UniqueName: \"kubernetes.io/projected/c1b2141f-0bab-4ce3-81a3-1b95d539d2fb-kube-api-access-665c4\") pod \"payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp\" (UID: \"c1b2141f-0bab-4ce3-81a3-1b95d539d2fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp"
Sep 11 21:27:42.764875 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:42.764860    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b2141f-0bab-4ce3-81a3-1b95d539d2fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp\" (UID: \"c1b2141f-0bab-4ce3-81a3-1b95d539d2fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp"
Sep 11 21:27:42.775363 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:42.775340    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-665c4\" (UniqueName: \"kubernetes.io/projected/c1b2141f-0bab-4ce3-81a3-1b95d539d2fb-kube-api-access-665c4\") pod \"payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp\" (UID: \"c1b2141f-0bab-4ce3-81a3-1b95d539d2fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp"
Sep 11 21:27:42.865334 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:42.865254    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cf51d65-3459-42b8-b86a-a253a4ca5d8e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2748c5-f65968655-trwmp\" (UID: \"6cf51d65-3459-42b8-b86a-a253a4ca5d8e\") " pod="openshift-ingress/payload-processing-e2e-api-b-2748c5-f65968655-trwmp"
Sep 11 21:27:42.865334 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:42.865330    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qklrx\" (UniqueName: \"kubernetes.io/projected/6cf51d65-3459-42b8-b86a-a253a4ca5d8e-kube-api-access-qklrx\") pod \"payload-processing-e2e-api-b-2748c5-f65968655-trwmp\" (UID: \"6cf51d65-3459-42b8-b86a-a253a4ca5d8e\") " pod="openshift-ingress/payload-processing-e2e-api-b-2748c5-f65968655-trwmp"
Sep 11 21:27:42.865970 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:42.865951    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cf51d65-3459-42b8-b86a-a253a4ca5d8e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2748c5-f65968655-trwmp\" (UID: \"6cf51d65-3459-42b8-b86a-a253a4ca5d8e\") " pod="openshift-ingress/payload-processing-e2e-api-b-2748c5-f65968655-trwmp"
Sep 11 21:27:42.900398 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:42.900369    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qklrx\" (UniqueName: \"kubernetes.io/projected/6cf51d65-3459-42b8-b86a-a253a4ca5d8e-kube-api-access-qklrx\") pod \"payload-processing-e2e-api-b-2748c5-f65968655-trwmp\" (UID: \"6cf51d65-3459-42b8-b86a-a253a4ca5d8e\") " pod="openshift-ingress/payload-processing-e2e-api-b-2748c5-f65968655-trwmp"
Sep 11 21:27:42.920735 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:42.920705    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp"
Sep 11 21:27:43.016560 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:43.016529    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2748c5-f65968655-trwmp"
Sep 11 21:27:43.210198 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:43.210166    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp"]
Sep 11 21:27:43.212065 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:27:43.212035    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b2141f_0bab_4ce3_81a3_1b95d539d2fb.slice/crio-5715d474de9b6f7d47a28df0a7063b7f9eb8e7a6210ddb28cd1c7f5ab7c7a5d5 WatchSource:0}: Error finding container 5715d474de9b6f7d47a28df0a7063b7f9eb8e7a6210ddb28cd1c7f5ab7c7a5d5: Status 404 returned error can't find the container with id 5715d474de9b6f7d47a28df0a7063b7f9eb8e7a6210ddb28cd1c7f5ab7c7a5d5
Sep 11 21:27:43.297550 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:43.297521    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2748c5-f65968655-trwmp"]
Sep 11 21:27:43.299945 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:27:43.299918    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cf51d65_3459_42b8_b86a_a253a4ca5d8e.slice/crio-e8f66dc78c0cd7ec890c951c34a1249cc43d99f440c9f0925b849ca75cebd59b WatchSource:0}: Error finding container e8f66dc78c0cd7ec890c951c34a1249cc43d99f440c9f0925b849ca75cebd59b: Status 404 returned error can't find the container with id e8f66dc78c0cd7ec890c951c34a1249cc43d99f440c9f0925b849ca75cebd59b
Sep 11 21:27:43.823068 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:43.823033    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2748c5-f65968655-trwmp" event={"ID":"6cf51d65-3459-42b8-b86a-a253a4ca5d8e","Type":"ContainerStarted","Data":"a81d9e2388e58ecc06d312519f7d066bc1b8e8572a67ee86741096d78f1827f9"}
Sep 11 21:27:43.823526 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:43.823100    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2748c5-f65968655-trwmp" event={"ID":"6cf51d65-3459-42b8-b86a-a253a4ca5d8e","Type":"ContainerStarted","Data":"e8f66dc78c0cd7ec890c951c34a1249cc43d99f440c9f0925b849ca75cebd59b"}
Sep 11 21:27:43.823526 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:43.823135    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2748c5-f65968655-trwmp"
Sep 11 21:27:43.824695 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:43.824666    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp" event={"ID":"c1b2141f-0bab-4ce3-81a3-1b95d539d2fb","Type":"ContainerStarted","Data":"1f2fc693ea8b5aeec03eb63a1bd81faf63f92cce17045689f9fbcc54aedac49c"}
Sep 11 21:27:43.824809 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:43.824716    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp" event={"ID":"c1b2141f-0bab-4ce3-81a3-1b95d539d2fb","Type":"ContainerStarted","Data":"5715d474de9b6f7d47a28df0a7063b7f9eb8e7a6210ddb28cd1c7f5ab7c7a5d5"}
Sep 11 21:27:43.824809 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:43.824782    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp"
Sep 11 21:27:43.851848 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:43.851777    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2748c5-f65968655-trwmp" podStartSLOduration=1.851757186 podStartE2EDuration="1.851757186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:27:43.851219266 +0000 UTC m=+1330.555188554" watchObservedRunningTime="2026-09-11 21:27:43.851757186 +0000 UTC m=+1330.555726478"
Sep 11 21:27:43.877221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:43.877084    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp" podStartSLOduration=1.877063675 podStartE2EDuration="1.877063675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:27:43.87625972 +0000 UTC m=+1330.580229000" watchObservedRunningTime="2026-09-11 21:27:43.877063675 +0000 UTC m=+1330.581032965"
Sep 11 21:27:47.307024 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:47.306993    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"]
Sep 11 21:27:47.309910 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:47.309893    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:27:47.313115 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:47.313100    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2bd772\"/\"sub-test-model-2bd772-kserve-self-signed-certs\""
Sep 11 21:27:47.333108 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:47.333081    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"]
Sep 11 21:27:47.402990 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:47.402953    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a38851a-0e48-4999-827f-f83cb10156d3-home\") pod \"sub-test-model-2bd772-kserve-6fd88449bc-dxr9v\" (UID: \"7a38851a-0e48-4999-827f-f83cb10156d3\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:27:47.402990 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:47.403003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a38851a-0e48-4999-827f-f83cb10156d3-model-cache\") pod \"sub-test-model-2bd772-kserve-6fd88449bc-dxr9v\" (UID: \"7a38851a-0e48-4999-827f-f83cb10156d3\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:27:47.403206 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:47.403029    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a38851a-0e48-4999-827f-f83cb10156d3-dshm\") pod \"sub-test-model-2bd772-kserve-6fd88449bc-dxr9v\" (UID: \"7a38851a-0e48-4999-827f-f83cb10156d3\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:27:47.403206 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:47.403072    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a38851a-0e48-4999-827f-f83cb10156d3-tmp-dir\") pod \"sub-test-model-2bd772-kserve-6fd88449bc-dxr9v\" (UID: \"7a38851a-0e48-4999-827f-f83cb10156d3\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:27:47.403206 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:47.403094    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x45pm\" (UniqueName: \"kubernetes.io/projected/7a38851a-0e48-4999-827f-f83cb10156d3-kube-api-access-x45pm\") pod \"sub-test-model-2bd772-kserve-6fd88449bc-dxr9v\" (UID: \"7a38851a-0e48-4999-827f-f83cb10156d3\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:27:47.403206 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:47.403121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a38851a-0e48-4999-827f-f83cb10156d3-tls-certs\") pod \"sub-test-model-2bd772-kserve-6fd88449bc-dxr9v\" (UID: \"7a38851a-0e48-4999-827f-f83cb10156d3\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:27:47.503860 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:47.503802    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a38851a-0e48-4999-827f-f83cb10156d3-home\") pod \"sub-test-model-2bd772-kserve-6fd88449bc-dxr9v\" (UID: \"7a38851a-0e48-4999-827f-f83cb10156d3\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:27:47.504025 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:47.503883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a38851a-0e48-4999-827f-f83cb10156d3-model-cache\") pod \"sub-test-model-2bd772-kserve-6fd88449bc-dxr9v\" (UID: \"7a38851a-0e48-4999-827f-f83cb10156d3\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:27:47.504025 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:47.503923    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a38851a-0e48-4999-827f-f83cb10156d3-dshm\") pod \"sub-test-model-2bd772-kserve-6fd88449bc-dxr9v\" (UID: \"7a38851a-0e48-4999-827f-f83cb10156d3\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:27:47.504025 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:47.503994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a38851a-0e48-4999-827f-f83cb10156d3-tmp-dir\") pod \"sub-test-model-2bd772-kserve-6fd88449bc-dxr9v\" (UID: \"7a38851a-0e48-4999-827f-f83cb10156d3\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:27:47.504199 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:47.504029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x45pm\" (UniqueName: \"kubernetes.io/projected/7a38851a-0e48-4999-827f-f83cb10156d3-kube-api-access-x45pm\") pod \"sub-test-model-2bd772-kserve-6fd88449bc-dxr9v\" (UID: \"7a38851a-0e48-4999-827f-f83cb10156d3\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:27:47.504199 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:47.504079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a38851a-0e48-4999-827f-f83cb10156d3-tls-certs\") pod \"sub-test-model-2bd772-kserve-6fd88449bc-dxr9v\" (UID: \"7a38851a-0e48-4999-827f-f83cb10156d3\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:27:47.504292 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:47.504275    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a38851a-0e48-4999-827f-f83cb10156d3-home\") pod \"sub-test-model-2bd772-kserve-6fd88449bc-dxr9v\" (UID: \"7a38851a-0e48-4999-827f-f83cb10156d3\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:27:47.504340 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:47.504303    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a38851a-0e48-4999-827f-f83cb10156d3-tmp-dir\") pod \"sub-test-model-2bd772-kserve-6fd88449bc-dxr9v\" (UID: \"7a38851a-0e48-4999-827f-f83cb10156d3\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:27:47.504340 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:47.504329    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a38851a-0e48-4999-827f-f83cb10156d3-model-cache\") pod \"sub-test-model-2bd772-kserve-6fd88449bc-dxr9v\" (UID: \"7a38851a-0e48-4999-827f-f83cb10156d3\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:27:47.506272 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:47.506251    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a38851a-0e48-4999-827f-f83cb10156d3-dshm\") pod \"sub-test-model-2bd772-kserve-6fd88449bc-dxr9v\" (UID: \"7a38851a-0e48-4999-827f-f83cb10156d3\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:27:47.506525 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:47.506505    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a38851a-0e48-4999-827f-f83cb10156d3-tls-certs\") pod \"sub-test-model-2bd772-kserve-6fd88449bc-dxr9v\" (UID: \"7a38851a-0e48-4999-827f-f83cb10156d3\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:27:47.515182 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:47.515161    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x45pm\" (UniqueName: \"kubernetes.io/projected/7a38851a-0e48-4999-827f-f83cb10156d3-kube-api-access-x45pm\") pod \"sub-test-model-2bd772-kserve-6fd88449bc-dxr9v\" (UID: \"7a38851a-0e48-4999-827f-f83cb10156d3\") " pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:27:47.618894 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:47.618784    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:27:47.906497 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:47.906471    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"]
Sep 11 21:27:47.909572 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:27:47.909547    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a38851a_0e48_4999_827f_f83cb10156d3.slice/crio-78d9aaf5f1dbeee35579d341904910e81700df3bdb04e464cb8a955a4e1935ba WatchSource:0}: Error finding container 78d9aaf5f1dbeee35579d341904910e81700df3bdb04e464cb8a955a4e1935ba: Status 404 returned error can't find the container with id 78d9aaf5f1dbeee35579d341904910e81700df3bdb04e464cb8a955a4e1935ba
Sep 11 21:27:48.851047 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:48.851013    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v" event={"ID":"7a38851a-0e48-4999-827f-f83cb10156d3","Type":"ContainerStarted","Data":"2a0052c2f5097c32f7798d442c7c2d15fc5c94c2366ddb0ec82b4f7915dde1fb"}
Sep 11 21:27:48.851047 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:48.851051    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v" event={"ID":"7a38851a-0e48-4999-827f-f83cb10156d3","Type":"ContainerStarted","Data":"78d9aaf5f1dbeee35579d341904910e81700df3bdb04e464cb8a955a4e1935ba"}
Sep 11 21:27:49.157504 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:49.157402    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v" podStartSLOduration=2.157382766 podStartE2EDuration="2.157382766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:27:48.889502809 +0000 UTC m=+1335.593472097" watchObservedRunningTime="2026-09-11 21:27:49.157382766 +0000 UTC m=+1335.861352108"
Sep 11 21:27:49.157667 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:49.157642    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:27:49.873390 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:49.873334    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc" podUID="e34bde6e-b402-4be5-af5d-e0f16a3efabe" containerName="main" containerID="cri-o://48cbad06f9f2a08f1a4a7df064952f0b66c3ed0eb3f4ed361c3e31f493ac6f54" gracePeriod=45
Sep 11 21:27:50.134571 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.134507    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:27:50.227691 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.227665    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e34bde6e-b402-4be5-af5d-e0f16a3efabe-tls-certs\") pod \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\" (UID: \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\") "
Sep 11 21:27:50.227691 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.227702    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xj255\" (UniqueName: \"kubernetes.io/projected/e34bde6e-b402-4be5-af5d-e0f16a3efabe-kube-api-access-xj255\") pod \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\" (UID: \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\") "
Sep 11 21:27:50.227963 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.227733    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e34bde6e-b402-4be5-af5d-e0f16a3efabe-dshm\") pod \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\" (UID: \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\") "
Sep 11 21:27:50.227963 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.227854    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e34bde6e-b402-4be5-af5d-e0f16a3efabe-tmp-dir\") pod \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\" (UID: \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\") "
Sep 11 21:27:50.227963 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.227887    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e34bde6e-b402-4be5-af5d-e0f16a3efabe-model-cache\") pod \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\" (UID: \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\") "
Sep 11 21:27:50.227963 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.227941    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e34bde6e-b402-4be5-af5d-e0f16a3efabe-home\") pod \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\" (UID: \"e34bde6e-b402-4be5-af5d-e0f16a3efabe\") "
Sep 11 21:27:50.228171 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.228112    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34bde6e-b402-4be5-af5d-e0f16a3efabe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e34bde6e-b402-4be5-af5d-e0f16a3efabe" (UID: "e34bde6e-b402-4be5-af5d-e0f16a3efabe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:27:50.228244 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.228194    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34bde6e-b402-4be5-af5d-e0f16a3efabe-model-cache" (OuterVolumeSpecName: "model-cache") pod "e34bde6e-b402-4be5-af5d-e0f16a3efabe" (UID: "e34bde6e-b402-4be5-af5d-e0f16a3efabe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:27:50.228311 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.228238    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34bde6e-b402-4be5-af5d-e0f16a3efabe-home" (OuterVolumeSpecName: "home") pod "e34bde6e-b402-4be5-af5d-e0f16a3efabe" (UID: "e34bde6e-b402-4be5-af5d-e0f16a3efabe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:27:50.228311 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.228264    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e34bde6e-b402-4be5-af5d-e0f16a3efabe-tmp-dir\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:27:50.229975 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.229952    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e34bde6e-b402-4be5-af5d-e0f16a3efabe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e34bde6e-b402-4be5-af5d-e0f16a3efabe" (UID: "e34bde6e-b402-4be5-af5d-e0f16a3efabe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:27:50.229975 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.229964    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e34bde6e-b402-4be5-af5d-e0f16a3efabe-kube-api-access-xj255" (OuterVolumeSpecName: "kube-api-access-xj255") pod "e34bde6e-b402-4be5-af5d-e0f16a3efabe" (UID: "e34bde6e-b402-4be5-af5d-e0f16a3efabe"). InnerVolumeSpecName "kube-api-access-xj255". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:27:50.230125 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.230030    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34bde6e-b402-4be5-af5d-e0f16a3efabe-dshm" (OuterVolumeSpecName: "dshm") pod "e34bde6e-b402-4be5-af5d-e0f16a3efabe" (UID: "e34bde6e-b402-4be5-af5d-e0f16a3efabe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:27:50.335896 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.335863    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e34bde6e-b402-4be5-af5d-e0f16a3efabe-dshm\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:27:50.335896 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.335890    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e34bde6e-b402-4be5-af5d-e0f16a3efabe-model-cache\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:27:50.335896 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.335899    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e34bde6e-b402-4be5-af5d-e0f16a3efabe-home\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:27:50.336120 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.335908    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e34bde6e-b402-4be5-af5d-e0f16a3efabe-tls-certs\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:27:50.336120 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.335916    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj255\" (UniqueName: \"kubernetes.io/projected/e34bde6e-b402-4be5-af5d-e0f16a3efabe-kube-api-access-xj255\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:27:50.862117 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.862080    2578 generic.go:356] "Generic (PLEG): container finished" podID="e34bde6e-b402-4be5-af5d-e0f16a3efabe" containerID="48cbad06f9f2a08f1a4a7df064952f0b66c3ed0eb3f4ed361c3e31f493ac6f54" exitCode=0
Sep 11 21:27:50.862299 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.862138    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"
Sep 11 21:27:50.862299 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.862160    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc" event={"ID":"e34bde6e-b402-4be5-af5d-e0f16a3efabe","Type":"ContainerDied","Data":"48cbad06f9f2a08f1a4a7df064952f0b66c3ed0eb3f4ed361c3e31f493ac6f54"}
Sep 11 21:27:50.862299 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.862204    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc" event={"ID":"e34bde6e-b402-4be5-af5d-e0f16a3efabe","Type":"ContainerDied","Data":"3d9b878e620b854764dc42f523c50e218f9ab576ee19c64b076e5d3738d50542"}
Sep 11 21:27:50.862299 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.862221    2578 scope.go:117] "RemoveContainer" containerID="48cbad06f9f2a08f1a4a7df064952f0b66c3ed0eb3f4ed361c3e31f493ac6f54"
Sep 11 21:27:50.871252 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.871231    2578 scope.go:117] "RemoveContainer" containerID="48cbad06f9f2a08f1a4a7df064952f0b66c3ed0eb3f4ed361c3e31f493ac6f54"
Sep 11 21:27:50.871474 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:27:50.871459    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48cbad06f9f2a08f1a4a7df064952f0b66c3ed0eb3f4ed361c3e31f493ac6f54\": container with ID starting with 48cbad06f9f2a08f1a4a7df064952f0b66c3ed0eb3f4ed361c3e31f493ac6f54 not found: ID does not exist" containerID="48cbad06f9f2a08f1a4a7df064952f0b66c3ed0eb3f4ed361c3e31f493ac6f54"
Sep 11 21:27:50.871522 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.871482    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48cbad06f9f2a08f1a4a7df064952f0b66c3ed0eb3f4ed361c3e31f493ac6f54"} err="failed to get container status \"48cbad06f9f2a08f1a4a7df064952f0b66c3ed0eb3f4ed361c3e31f493ac6f54\": rpc error: code = NotFound desc = could not find container \"48cbad06f9f2a08f1a4a7df064952f0b66c3ed0eb3f4ed361c3e31f493ac6f54\": container with ID starting with 48cbad06f9f2a08f1a4a7df064952f0b66c3ed0eb3f4ed361c3e31f493ac6f54 not found: ID does not exist"
Sep 11 21:27:50.888722 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.888699    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"]
Sep 11 21:27:50.895093 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:50.895073    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2bd772/auth-test-model-2bd772-kserve-c956569b7-wvgwc"]
Sep 11 21:27:51.927959 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:51.927929    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e34bde6e-b402-4be5-af5d-e0f16a3efabe" path="/var/lib/kubelet/pods/e34bde6e-b402-4be5-af5d-e0f16a3efabe/volumes"
Sep 11 21:27:54.831982 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:54.831949    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2748c5-f65968655-trwmp"
Sep 11 21:27:54.832335 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:54.832117    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp"
Sep 11 21:27:57.619948 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:57.619861    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:27:57.619948 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:57.619916    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:27:57.632728 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:57.632704    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:27:57.904438 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:27:57.904368    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:28:08.159232 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:08.159198    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:28:24.541328 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:24.541298    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-354c80-bfd69945c-l2phv"]
Sep 11 21:28:24.541879 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:24.541512    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-354c80-bfd69945c-l2phv" podUID="74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4" containerName="payload-processing" containerID="cri-o://d20f0334da52116169e4307a180eef0951409dc17434d754c59125e05f4fb79a" gracePeriod=30
Sep 11 21:28:24.797445 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:24.797387    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-354c80-bfd69945c-l2phv"
Sep 11 21:28:24.829429 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:24.829406    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4-plugins-config-volume\") pod \"74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4\" (UID: \"74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4\") "
Sep 11 21:28:24.829527 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:24.829445    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sm65q\" (UniqueName: \"kubernetes.io/projected/74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4-kube-api-access-sm65q\") pod \"74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4\" (UID: \"74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4\") "
Sep 11 21:28:24.829744 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:24.829724    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4" (UID: "74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:28:24.831724 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:24.831698    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4-kube-api-access-sm65q" (OuterVolumeSpecName: "kube-api-access-sm65q") pod "74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4" (UID: "74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4"). InnerVolumeSpecName "kube-api-access-sm65q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:28:24.930178 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:24.930144    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4-plugins-config-volume\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:28:24.930178 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:24.930171    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm65q\" (UniqueName: \"kubernetes.io/projected/74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4-kube-api-access-sm65q\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:28:25.013667 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:25.013636    2578 generic.go:356] "Generic (PLEG): container finished" podID="74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4" containerID="d20f0334da52116169e4307a180eef0951409dc17434d754c59125e05f4fb79a" exitCode=0
Sep 11 21:28:25.013842 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:25.013676    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-354c80-bfd69945c-l2phv"
Sep 11 21:28:25.013842 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:25.013715    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-354c80-bfd69945c-l2phv" event={"ID":"74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4","Type":"ContainerDied","Data":"d20f0334da52116169e4307a180eef0951409dc17434d754c59125e05f4fb79a"}
Sep 11 21:28:25.013842 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:25.013765    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-354c80-bfd69945c-l2phv" event={"ID":"74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4","Type":"ContainerDied","Data":"70d6e66c25410314e534477f667d9fb949837ba3c751786b7a2b090c44859f46"}
Sep 11 21:28:25.013842 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:25.013785    2578 scope.go:117] "RemoveContainer" containerID="d20f0334da52116169e4307a180eef0951409dc17434d754c59125e05f4fb79a"
Sep 11 21:28:25.022548 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:25.022533    2578 scope.go:117] "RemoveContainer" containerID="d20f0334da52116169e4307a180eef0951409dc17434d754c59125e05f4fb79a"
Sep 11 21:28:25.022806 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:28:25.022790    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d20f0334da52116169e4307a180eef0951409dc17434d754c59125e05f4fb79a\": container with ID starting with d20f0334da52116169e4307a180eef0951409dc17434d754c59125e05f4fb79a not found: ID does not exist" containerID="d20f0334da52116169e4307a180eef0951409dc17434d754c59125e05f4fb79a"
Sep 11 21:28:25.022889 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:25.022832    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d20f0334da52116169e4307a180eef0951409dc17434d754c59125e05f4fb79a"} err="failed to get container status \"d20f0334da52116169e4307a180eef0951409dc17434d754c59125e05f4fb79a\": rpc error: code = NotFound desc = could not find container \"d20f0334da52116169e4307a180eef0951409dc17434d754c59125e05f4fb79a\": container with ID starting with d20f0334da52116169e4307a180eef0951409dc17434d754c59125e05f4fb79a not found: ID does not exist"
Sep 11 21:28:25.044994 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:25.044956    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-354c80-bfd69945c-l2phv"]
Sep 11 21:28:25.049898 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:25.049848    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-354c80-bfd69945c-l2phv"]
Sep 11 21:28:25.933549 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:25.933513    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4" path="/var/lib/kubelet/pods/74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4/volumes"
Sep 11 21:28:27.558479 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.558447    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"]
Sep 11 21:28:27.558838 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.558770    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e34bde6e-b402-4be5-af5d-e0f16a3efabe" containerName="main"
Sep 11 21:28:27.558838 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.558780    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e34bde6e-b402-4be5-af5d-e0f16a3efabe" containerName="main"
Sep 11 21:28:27.558838 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.558791    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4" containerName="payload-processing"
Sep 11 21:28:27.558838 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.558797    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4" containerName="payload-processing"
Sep 11 21:28:27.558966 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.558866    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e34bde6e-b402-4be5-af5d-e0f16a3efabe" containerName="main"
Sep 11 21:28:27.558966 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.558875    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="74dc0372-3cd9-4e39-bbc2-6853e7d9f4a4" containerName="payload-processing"
Sep 11 21:28:27.563142 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.563123    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:28:27.584409 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.584388    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-713791\"/\"kube-root-ca.crt\""
Sep 11 21:28:27.585378 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.585364    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-713791\"/\"openshift-service-ca.crt\""
Sep 11 21:28:27.585518 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.585500    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-713791\"/\"sub-test-model-713791-kserve-self-signed-certs\""
Sep 11 21:28:27.586221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.586205    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-713791\"/\"default-dockercfg-bhxr5\""
Sep 11 21:28:27.598624 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.598598    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"]
Sep 11 21:28:27.646869 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.646841    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-tls-certs\") pod \"sub-test-model-713791-kserve-7d76d847ff-jmnnz\" (UID: \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\") " pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:28:27.647056 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.646882    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2klv\" (UniqueName: \"kubernetes.io/projected/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-kube-api-access-g2klv\") pod \"sub-test-model-713791-kserve-7d76d847ff-jmnnz\" (UID: \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\") " pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:28:27.647056 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.646989    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-home\") pod \"sub-test-model-713791-kserve-7d76d847ff-jmnnz\" (UID: \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\") " pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:28:27.647056 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.647032    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-model-cache\") pod \"sub-test-model-713791-kserve-7d76d847ff-jmnnz\" (UID: \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\") " pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:28:27.647056 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.647050    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-tmp-dir\") pod \"sub-test-model-713791-kserve-7d76d847ff-jmnnz\" (UID: \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\") " pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:28:27.647280 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.647070    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-dshm\") pod \"sub-test-model-713791-kserve-7d76d847ff-jmnnz\" (UID: \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\") " pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:28:27.747870 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.747844    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-tls-certs\") pod \"sub-test-model-713791-kserve-7d76d847ff-jmnnz\" (UID: \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\") " pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:28:27.748031 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.747876    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2klv\" (UniqueName: \"kubernetes.io/projected/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-kube-api-access-g2klv\") pod \"sub-test-model-713791-kserve-7d76d847ff-jmnnz\" (UID: \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\") " pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:28:27.748031 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.747905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-home\") pod \"sub-test-model-713791-kserve-7d76d847ff-jmnnz\" (UID: \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\") " pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:28:27.748031 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.747938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-model-cache\") pod \"sub-test-model-713791-kserve-7d76d847ff-jmnnz\" (UID: \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\") " pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:28:27.748031 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.747954    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-tmp-dir\") pod \"sub-test-model-713791-kserve-7d76d847ff-jmnnz\" (UID: \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\") " pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:28:27.748031 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.747971    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-dshm\") pod \"sub-test-model-713791-kserve-7d76d847ff-jmnnz\" (UID: \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\") " pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:28:27.748364 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.748342    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-tmp-dir\") pod \"sub-test-model-713791-kserve-7d76d847ff-jmnnz\" (UID: \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\") " pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:28:27.748442 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.748355    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-home\") pod \"sub-test-model-713791-kserve-7d76d847ff-jmnnz\" (UID: \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\") " pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:28:27.748442 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.748402    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-model-cache\") pod \"sub-test-model-713791-kserve-7d76d847ff-jmnnz\" (UID: \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\") " pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:28:27.750138 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.750122    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-dshm\") pod \"sub-test-model-713791-kserve-7d76d847ff-jmnnz\" (UID: \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\") " pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:28:27.750243 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.750227    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-tls-certs\") pod \"sub-test-model-713791-kserve-7d76d847ff-jmnnz\" (UID: \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\") " pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:28:27.771123 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.771099    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2klv\" (UniqueName: \"kubernetes.io/projected/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-kube-api-access-g2klv\") pod \"sub-test-model-713791-kserve-7d76d847ff-jmnnz\" (UID: \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\") " pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:28:27.871909 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:27.871847    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:28:28.157283 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:28.157249    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"]
Sep 11 21:28:28.158691 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:28:28.158670    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e9b5001_ff00_4bce_8a7a_0a58d689aea8.slice/crio-c706aa188f1069d1db812f6ebcd3f1dd57633a344b6645e20c79f53a0f8483f2 WatchSource:0}: Error finding container c706aa188f1069d1db812f6ebcd3f1dd57633a344b6645e20c79f53a0f8483f2: Status 404 returned error can't find the container with id c706aa188f1069d1db812f6ebcd3f1dd57633a344b6645e20c79f53a0f8483f2
Sep 11 21:28:29.035783 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:29.035749    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz" event={"ID":"8e9b5001-ff00-4bce-8a7a-0a58d689aea8","Type":"ContainerStarted","Data":"7b50f93eb8f1689d9822ae8ed178ec49179d520f5092ca9b1c8c62fef21461cc"}
Sep 11 21:28:29.035783 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:29.035786    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz" event={"ID":"8e9b5001-ff00-4bce-8a7a-0a58d689aea8","Type":"ContainerStarted","Data":"c706aa188f1069d1db812f6ebcd3f1dd57633a344b6645e20c79f53a0f8483f2"}
Sep 11 21:28:31.923157 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:31.923105    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz" podStartSLOduration=4.923087807 podStartE2EDuration="4.923087807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:28:29.075193732 +0000 UTC m=+1375.779163022" watchObservedRunningTime="2026-09-11 21:28:31.923087807 +0000 UTC m=+1378.627057137"
Sep 11 21:28:31.928017 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:31.927990    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:28:37.872347 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:37.872315    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:28:37.872777 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:37.872360    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:28:37.884668 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:37.884645    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:28:38.085181 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:38.085154    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:28:50.890288 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:50.890250    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2748c5-f65968655-trwmp"]
Sep 11 21:28:50.890889 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:50.890494    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2748c5-f65968655-trwmp" podUID="6cf51d65-3459-42b8-b86a-a253a4ca5d8e" containerName="payload-processing" containerID="cri-o://a81d9e2388e58ecc06d312519f7d066bc1b8e8572a67ee86741096d78f1827f9" gracePeriod=30
Sep 11 21:28:50.899314 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:50.899282    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp"]
Sep 11 21:28:50.899539 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:50.899503    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp" podUID="c1b2141f-0bab-4ce3-81a3-1b95d539d2fb" containerName="payload-pre-processing" containerID="cri-o://1f2fc693ea8b5aeec03eb63a1bd81faf63f92cce17045689f9fbcc54aedac49c" gracePeriod=30
Sep 11 21:28:51.134876 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:51.134844    2578 generic.go:356] "Generic (PLEG): container finished" podID="6cf51d65-3459-42b8-b86a-a253a4ca5d8e" containerID="a81d9e2388e58ecc06d312519f7d066bc1b8e8572a67ee86741096d78f1827f9" exitCode=0
Sep 11 21:28:51.135031 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:51.134918    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2748c5-f65968655-trwmp" event={"ID":"6cf51d65-3459-42b8-b86a-a253a4ca5d8e","Type":"ContainerDied","Data":"a81d9e2388e58ecc06d312519f7d066bc1b8e8572a67ee86741096d78f1827f9"}
Sep 11 21:28:51.136742 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:51.136713    2578 generic.go:356] "Generic (PLEG): container finished" podID="c1b2141f-0bab-4ce3-81a3-1b95d539d2fb" containerID="1f2fc693ea8b5aeec03eb63a1bd81faf63f92cce17045689f9fbcc54aedac49c" exitCode=0
Sep 11 21:28:51.136855 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:51.136785    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp" event={"ID":"c1b2141f-0bab-4ce3-81a3-1b95d539d2fb","Type":"ContainerDied","Data":"1f2fc693ea8b5aeec03eb63a1bd81faf63f92cce17045689f9fbcc54aedac49c"}
Sep 11 21:28:51.155567 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:51.155550    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2748c5-f65968655-trwmp"
Sep 11 21:28:51.158356 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:51.158339    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp"
Sep 11 21:28:51.241052 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:51.241022    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qklrx\" (UniqueName: \"kubernetes.io/projected/6cf51d65-3459-42b8-b86a-a253a4ca5d8e-kube-api-access-qklrx\") pod \"6cf51d65-3459-42b8-b86a-a253a4ca5d8e\" (UID: \"6cf51d65-3459-42b8-b86a-a253a4ca5d8e\") "
Sep 11 21:28:51.241202 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:51.241070    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-665c4\" (UniqueName: \"kubernetes.io/projected/c1b2141f-0bab-4ce3-81a3-1b95d539d2fb-kube-api-access-665c4\") pod \"c1b2141f-0bab-4ce3-81a3-1b95d539d2fb\" (UID: \"c1b2141f-0bab-4ce3-81a3-1b95d539d2fb\") "
Sep 11 21:28:51.241202 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:51.241117    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b2141f-0bab-4ce3-81a3-1b95d539d2fb-plugins-config-volume\") pod \"c1b2141f-0bab-4ce3-81a3-1b95d539d2fb\" (UID: \"c1b2141f-0bab-4ce3-81a3-1b95d539d2fb\") "
Sep 11 21:28:51.241202 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:51.241139    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cf51d65-3459-42b8-b86a-a253a4ca5d8e-plugins-config-volume\") pod \"6cf51d65-3459-42b8-b86a-a253a4ca5d8e\" (UID: \"6cf51d65-3459-42b8-b86a-a253a4ca5d8e\") "
Sep 11 21:28:51.241542 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:51.241514    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b2141f-0bab-4ce3-81a3-1b95d539d2fb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c1b2141f-0bab-4ce3-81a3-1b95d539d2fb" (UID: "c1b2141f-0bab-4ce3-81a3-1b95d539d2fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:28:51.241661 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:51.241578    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cf51d65-3459-42b8-b86a-a253a4ca5d8e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6cf51d65-3459-42b8-b86a-a253a4ca5d8e" (UID: "6cf51d65-3459-42b8-b86a-a253a4ca5d8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:28:51.243168 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:51.243139    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cf51d65-3459-42b8-b86a-a253a4ca5d8e-kube-api-access-qklrx" (OuterVolumeSpecName: "kube-api-access-qklrx") pod "6cf51d65-3459-42b8-b86a-a253a4ca5d8e" (UID: "6cf51d65-3459-42b8-b86a-a253a4ca5d8e"). InnerVolumeSpecName "kube-api-access-qklrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:28:51.243263 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:51.243209    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b2141f-0bab-4ce3-81a3-1b95d539d2fb-kube-api-access-665c4" (OuterVolumeSpecName: "kube-api-access-665c4") pod "c1b2141f-0bab-4ce3-81a3-1b95d539d2fb" (UID: "c1b2141f-0bab-4ce3-81a3-1b95d539d2fb"). InnerVolumeSpecName "kube-api-access-665c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:28:51.342337 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:51.342316    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-665c4\" (UniqueName: \"kubernetes.io/projected/c1b2141f-0bab-4ce3-81a3-1b95d539d2fb-kube-api-access-665c4\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:28:51.342337 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:51.342340    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b2141f-0bab-4ce3-81a3-1b95d539d2fb-plugins-config-volume\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:28:51.342480 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:51.342348    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cf51d65-3459-42b8-b86a-a253a4ca5d8e-plugins-config-volume\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:28:51.342480 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:51.342357    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qklrx\" (UniqueName: \"kubernetes.io/projected/6cf51d65-3459-42b8-b86a-a253a4ca5d8e-kube-api-access-qklrx\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:28:52.142711 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:52.142676    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2748c5-f65968655-trwmp"
Sep 11 21:28:52.143183 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:52.142725    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2748c5-f65968655-trwmp" event={"ID":"6cf51d65-3459-42b8-b86a-a253a4ca5d8e","Type":"ContainerDied","Data":"e8f66dc78c0cd7ec890c951c34a1249cc43d99f440c9f0925b849ca75cebd59b"}
Sep 11 21:28:52.143183 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:52.142785    2578 scope.go:117] "RemoveContainer" containerID="a81d9e2388e58ecc06d312519f7d066bc1b8e8572a67ee86741096d78f1827f9"
Sep 11 21:28:52.144187 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:52.144170    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp"
Sep 11 21:28:52.144265 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:52.144171    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp" event={"ID":"c1b2141f-0bab-4ce3-81a3-1b95d539d2fb","Type":"ContainerDied","Data":"5715d474de9b6f7d47a28df0a7063b7f9eb8e7a6210ddb28cd1c7f5ab7c7a5d5"}
Sep 11 21:28:52.150981 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:52.150963    2578 scope.go:117] "RemoveContainer" containerID="1f2fc693ea8b5aeec03eb63a1bd81faf63f92cce17045689f9fbcc54aedac49c"
Sep 11 21:28:52.169187 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:52.169164    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp"]
Sep 11 21:28:52.171578 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:52.171558    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2748c5-8445f79c66-z7zqp"]
Sep 11 21:28:52.187609 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:52.187565    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2748c5-f65968655-trwmp"]
Sep 11 21:28:52.198918 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:52.198899    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2748c5-f65968655-trwmp"]
Sep 11 21:28:53.926997 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:53.926963    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cf51d65-3459-42b8-b86a-a253a4ca5d8e" path="/var/lib/kubelet/pods/6cf51d65-3459-42b8-b86a-a253a4ca5d8e/volumes"
Sep 11 21:28:53.927342 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:53.927302    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1b2141f-0bab-4ce3-81a3-1b95d539d2fb" path="/var/lib/kubelet/pods/c1b2141f-0bab-4ce3-81a3-1b95d539d2fb/volumes"
Sep 11 21:28:56.490059 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:28:56.490025    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:29:11.279226 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:11.279194    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5"]
Sep 11 21:29:11.279836 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:11.279440    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5" podUID="c3178682-f664-4782-8d89-f683b3a69d2f" containerName="payload-processing" containerID="cri-o://b003a1d88edcd4b99bbf308e4c48e791b5a6b44942eed558be9091f3d4349183" gracePeriod=30
Sep 11 21:29:11.293203 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:11.293176    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw"]
Sep 11 21:29:11.293406 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:11.293380    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw" podUID="46c779ff-b35c-4251-b874-7bdb2821cee7" containerName="payload-pre-processing" containerID="cri-o://bbd98e1f654764bffc479a379c7f1546820ae4fbc5b76311456d7783deca5f1e" gracePeriod=30
Sep 11 21:29:11.565106 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:11.565082    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5"
Sep 11 21:29:11.567628 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:11.567613    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw"
Sep 11 21:29:11.726733 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:11.726701    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3178682-f664-4782-8d89-f683b3a69d2f-plugins-config-volume\") pod \"c3178682-f664-4782-8d89-f683b3a69d2f\" (UID: \"c3178682-f664-4782-8d89-f683b3a69d2f\") "
Sep 11 21:29:11.726733 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:11.726737    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9b6d\" (UniqueName: \"kubernetes.io/projected/c3178682-f664-4782-8d89-f683b3a69d2f-kube-api-access-c9b6d\") pod \"c3178682-f664-4782-8d89-f683b3a69d2f\" (UID: \"c3178682-f664-4782-8d89-f683b3a69d2f\") "
Sep 11 21:29:11.726998 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:11.726758    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5d9j\" (UniqueName: \"kubernetes.io/projected/46c779ff-b35c-4251-b874-7bdb2821cee7-kube-api-access-x5d9j\") pod \"46c779ff-b35c-4251-b874-7bdb2821cee7\" (UID: \"46c779ff-b35c-4251-b874-7bdb2821cee7\") "
Sep 11 21:29:11.726998 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:11.726804    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c779ff-b35c-4251-b874-7bdb2821cee7-plugins-config-volume\") pod \"46c779ff-b35c-4251-b874-7bdb2821cee7\" (UID: \"46c779ff-b35c-4251-b874-7bdb2821cee7\") "
Sep 11 21:29:11.727235 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:11.727202    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3178682-f664-4782-8d89-f683b3a69d2f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c3178682-f664-4782-8d89-f683b3a69d2f" (UID: "c3178682-f664-4782-8d89-f683b3a69d2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:29:11.727338 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:11.727271    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46c779ff-b35c-4251-b874-7bdb2821cee7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "46c779ff-b35c-4251-b874-7bdb2821cee7" (UID: "46c779ff-b35c-4251-b874-7bdb2821cee7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:29:11.728921 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:11.728898    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3178682-f664-4782-8d89-f683b3a69d2f-kube-api-access-c9b6d" (OuterVolumeSpecName: "kube-api-access-c9b6d") pod "c3178682-f664-4782-8d89-f683b3a69d2f" (UID: "c3178682-f664-4782-8d89-f683b3a69d2f"). InnerVolumeSpecName "kube-api-access-c9b6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:29:11.729009 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:11.728940    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c779ff-b35c-4251-b874-7bdb2821cee7-kube-api-access-x5d9j" (OuterVolumeSpecName: "kube-api-access-x5d9j") pod "46c779ff-b35c-4251-b874-7bdb2821cee7" (UID: "46c779ff-b35c-4251-b874-7bdb2821cee7"). InnerVolumeSpecName "kube-api-access-x5d9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:29:11.827747 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:11.827644    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3178682-f664-4782-8d89-f683b3a69d2f-plugins-config-volume\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:29:11.827747 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:11.827689    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9b6d\" (UniqueName: \"kubernetes.io/projected/c3178682-f664-4782-8d89-f683b3a69d2f-kube-api-access-c9b6d\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:29:11.827747 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:11.827699    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5d9j\" (UniqueName: \"kubernetes.io/projected/46c779ff-b35c-4251-b874-7bdb2821cee7-kube-api-access-x5d9j\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:29:11.827747 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:11.827709    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c779ff-b35c-4251-b874-7bdb2821cee7-plugins-config-volume\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:29:12.237299 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:12.237265    2578 generic.go:356] "Generic (PLEG): container finished" podID="46c779ff-b35c-4251-b874-7bdb2821cee7" containerID="bbd98e1f654764bffc479a379c7f1546820ae4fbc5b76311456d7783deca5f1e" exitCode=0
Sep 11 21:29:12.237565 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:12.237323    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw"
Sep 11 21:29:12.237565 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:12.237355    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw" event={"ID":"46c779ff-b35c-4251-b874-7bdb2821cee7","Type":"ContainerDied","Data":"bbd98e1f654764bffc479a379c7f1546820ae4fbc5b76311456d7783deca5f1e"}
Sep 11 21:29:12.237565 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:12.237437    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw" event={"ID":"46c779ff-b35c-4251-b874-7bdb2821cee7","Type":"ContainerDied","Data":"437083a8f1fccf464dd60c9af981cda17531e77bb351efd8ae6094ebf7d5a253"}
Sep 11 21:29:12.237565 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:12.237463    2578 scope.go:117] "RemoveContainer" containerID="bbd98e1f654764bffc479a379c7f1546820ae4fbc5b76311456d7783deca5f1e"
Sep 11 21:29:12.238837 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:12.238801    2578 generic.go:356] "Generic (PLEG): container finished" podID="c3178682-f664-4782-8d89-f683b3a69d2f" containerID="b003a1d88edcd4b99bbf308e4c48e791b5a6b44942eed558be9091f3d4349183" exitCode=0
Sep 11 21:29:12.238937 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:12.238857    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5" event={"ID":"c3178682-f664-4782-8d89-f683b3a69d2f","Type":"ContainerDied","Data":"b003a1d88edcd4b99bbf308e4c48e791b5a6b44942eed558be9091f3d4349183"}
Sep 11 21:29:12.238937 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:12.238878    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5"
Sep 11 21:29:12.238937 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:12.238914    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5" event={"ID":"c3178682-f664-4782-8d89-f683b3a69d2f","Type":"ContainerDied","Data":"a7e7a8352874678a9485548044d4beafc5d2fda250c11e40f14004381dc3b4f3"}
Sep 11 21:29:12.246279 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:12.246260    2578 scope.go:117] "RemoveContainer" containerID="bbd98e1f654764bffc479a379c7f1546820ae4fbc5b76311456d7783deca5f1e"
Sep 11 21:29:12.246615 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:29:12.246579    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbd98e1f654764bffc479a379c7f1546820ae4fbc5b76311456d7783deca5f1e\": container with ID starting with bbd98e1f654764bffc479a379c7f1546820ae4fbc5b76311456d7783deca5f1e not found: ID does not exist" containerID="bbd98e1f654764bffc479a379c7f1546820ae4fbc5b76311456d7783deca5f1e"
Sep 11 21:29:12.246807 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:12.246618    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbd98e1f654764bffc479a379c7f1546820ae4fbc5b76311456d7783deca5f1e"} err="failed to get container status \"bbd98e1f654764bffc479a379c7f1546820ae4fbc5b76311456d7783deca5f1e\": rpc error: code = NotFound desc = could not find container \"bbd98e1f654764bffc479a379c7f1546820ae4fbc5b76311456d7783deca5f1e\": container with ID starting with bbd98e1f654764bffc479a379c7f1546820ae4fbc5b76311456d7783deca5f1e not found: ID does not exist"
Sep 11 21:29:12.246807 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:12.246643    2578 scope.go:117] "RemoveContainer" containerID="b003a1d88edcd4b99bbf308e4c48e791b5a6b44942eed558be9091f3d4349183"
Sep 11 21:29:12.254020 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:12.254004    2578 scope.go:117] "RemoveContainer" containerID="b003a1d88edcd4b99bbf308e4c48e791b5a6b44942eed558be9091f3d4349183"
Sep 11 21:29:12.254244 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:29:12.254228    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b003a1d88edcd4b99bbf308e4c48e791b5a6b44942eed558be9091f3d4349183\": container with ID starting with b003a1d88edcd4b99bbf308e4c48e791b5a6b44942eed558be9091f3d4349183 not found: ID does not exist" containerID="b003a1d88edcd4b99bbf308e4c48e791b5a6b44942eed558be9091f3d4349183"
Sep 11 21:29:12.254304 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:12.254254    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b003a1d88edcd4b99bbf308e4c48e791b5a6b44942eed558be9091f3d4349183"} err="failed to get container status \"b003a1d88edcd4b99bbf308e4c48e791b5a6b44942eed558be9091f3d4349183\": rpc error: code = NotFound desc = could not find container \"b003a1d88edcd4b99bbf308e4c48e791b5a6b44942eed558be9091f3d4349183\": container with ID starting with b003a1d88edcd4b99bbf308e4c48e791b5a6b44942eed558be9091f3d4349183 not found: ID does not exist"
Sep 11 21:29:12.278492 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:12.278474    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5"]
Sep 11 21:29:12.292705 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:12.292684    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1117b2-5bbf88f677-svsp5"]
Sep 11 21:29:12.328644 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:12.328622    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw"]
Sep 11 21:29:12.333111 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:12.333091    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1117b2-754687b5f4-k9mxw"]
Sep 11 21:29:13.927213 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:13.927181    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46c779ff-b35c-4251-b874-7bdb2821cee7" path="/var/lib/kubelet/pods/46c779ff-b35c-4251-b874-7bdb2821cee7/volumes"
Sep 11 21:29:13.927552 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:13.927514    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3178682-f664-4782-8d89-f683b3a69d2f" path="/var/lib/kubelet/pods/c3178682-f664-4782-8d89-f683b3a69d2f/volumes"
Sep 11 21:29:14.092550 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:14.092519    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:29:26.277561 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:26.277491    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7702-185f32"]
Sep 11 21:29:26.278052 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:26.277803    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46c779ff-b35c-4251-b874-7bdb2821cee7" containerName="payload-pre-processing"
Sep 11 21:29:26.278052 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:26.277813    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="46c779ff-b35c-4251-b874-7bdb2821cee7" containerName="payload-pre-processing"
Sep 11 21:29:26.278052 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:26.277832    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b2141f-0bab-4ce3-81a3-1b95d539d2fb" containerName="payload-pre-processing"
Sep 11 21:29:26.278052 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:26.277838    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b2141f-0bab-4ce3-81a3-1b95d539d2fb" containerName="payload-pre-processing"
Sep 11 21:29:26.278052 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:26.277849    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3178682-f664-4782-8d89-f683b3a69d2f" containerName="payload-processing"
Sep 11 21:29:26.278052 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:26.277858    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3178682-f664-4782-8d89-f683b3a69d2f" containerName="payload-processing"
Sep 11 21:29:26.278052 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:26.277876    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cf51d65-3459-42b8-b86a-a253a4ca5d8e" containerName="payload-processing"
Sep 11 21:29:26.278052 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:26.277882    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cf51d65-3459-42b8-b86a-a253a4ca5d8e" containerName="payload-processing"
Sep 11 21:29:26.278052 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:26.277926    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="46c779ff-b35c-4251-b874-7bdb2821cee7" containerName="payload-pre-processing"
Sep 11 21:29:26.278052 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:26.277934    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1b2141f-0bab-4ce3-81a3-1b95d539d2fb" containerName="payload-pre-processing"
Sep 11 21:29:26.278052 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:26.277941    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cf51d65-3459-42b8-b86a-a253a4ca5d8e" containerName="payload-processing"
Sep 11 21:29:26.278052 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:26.277948    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3178682-f664-4782-8d89-f683b3a69d2f" containerName="payload-processing"
Sep 11 21:29:26.282136 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:26.282114    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7702-185f32"
Sep 11 21:29:26.285177 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:26.285155    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6dwfw\""
Sep 11 21:29:26.298119 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:26.298097    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7702-185f32"]
Sep 11 21:29:26.341654 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:26.341621    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn6k9\" (UniqueName: \"kubernetes.io/projected/e4104316-9b91-45b1-860e-eed9a5348dfb-kube-api-access-bn6k9\") pod \"test-curl-7702-185f32\" (UID: \"e4104316-9b91-45b1-860e-eed9a5348dfb\") " pod="opendatahub/test-curl-7702-185f32"
Sep 11 21:29:26.442316 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:26.442282    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn6k9\" (UniqueName: \"kubernetes.io/projected/e4104316-9b91-45b1-860e-eed9a5348dfb-kube-api-access-bn6k9\") pod \"test-curl-7702-185f32\" (UID: \"e4104316-9b91-45b1-860e-eed9a5348dfb\") " pod="opendatahub/test-curl-7702-185f32"
Sep 11 21:29:26.458941 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:26.458913    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn6k9\" (UniqueName: \"kubernetes.io/projected/e4104316-9b91-45b1-860e-eed9a5348dfb-kube-api-access-bn6k9\") pod \"test-curl-7702-185f32\" (UID: \"e4104316-9b91-45b1-860e-eed9a5348dfb\") " pod="opendatahub/test-curl-7702-185f32"
Sep 11 21:29:26.590988 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:26.590909    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7702-185f32"
Sep 11 21:29:26.883967 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:26.883881    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7702-185f32"]
Sep 11 21:29:26.888855 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:29:26.888810    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4104316_9b91_45b1_860e_eed9a5348dfb.slice/crio-37fa262112859c5f78d47dbe3e9c3c7564368d33031055809561002c98506171 WatchSource:0}: Error finding container 37fa262112859c5f78d47dbe3e9c3c7564368d33031055809561002c98506171: Status 404 returned error can't find the container with id 37fa262112859c5f78d47dbe3e9c3c7564368d33031055809561002c98506171
Sep 11 21:29:27.307910 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:27.307872    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7702-185f32" event={"ID":"e4104316-9b91-45b1-860e-eed9a5348dfb","Type":"ContainerStarted","Data":"a8d596d885b597a6448410749021f17df1f7eb32d49eb17185726018a1a575ce"}
Sep 11 21:29:27.307910 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:27.307915    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7702-185f32" event={"ID":"e4104316-9b91-45b1-860e-eed9a5348dfb","Type":"ContainerStarted","Data":"37fa262112859c5f78d47dbe3e9c3c7564368d33031055809561002c98506171"}
Sep 11 21:29:27.344672 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:27.344617    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7702-185f32" podStartSLOduration=1.344600384 podStartE2EDuration="1.344600384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:29:27.343261276 +0000 UTC m=+1434.047230565" watchObservedRunningTime="2026-09-11 21:29:27.344600384 +0000 UTC m=+1434.048569673"
Sep 11 21:29:28.118781 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:28.118704    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7702-185f32"]
Sep 11 21:29:28.127918 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:28.127883    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7702-185f32"]
Sep 11 21:29:28.257242 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:28.257198    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7702-7f8620"]
Sep 11 21:29:28.257591 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:28.257572    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4104316-9b91-45b1-860e-eed9a5348dfb" containerName="test-curl-7702-185f32"
Sep 11 21:29:28.257677 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:28.257611    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4104316-9b91-45b1-860e-eed9a5348dfb" containerName="test-curl-7702-185f32"
Sep 11 21:29:28.257735 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:28.257699    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4104316-9b91-45b1-860e-eed9a5348dfb" containerName="test-curl-7702-185f32"
Sep 11 21:29:28.260680 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:28.260662    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7702-7f8620"
Sep 11 21:29:28.276708 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:28.276684    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7702-7f8620"]
Sep 11 21:29:28.312510 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:29:28.312483    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7702-185f32" container="test-curl-7702-185f32"
Sep 11 21:29:28.356886 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:28.356853    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fjv8\" (UniqueName: \"kubernetes.io/projected/3c528012-b068-4867-9a7b-b163828cb54c-kube-api-access-4fjv8\") pod \"test-curl-7702-7f8620\" (UID: \"3c528012-b068-4867-9a7b-b163828cb54c\") " pod="opendatahub/test-curl-7702-7f8620"
Sep 11 21:29:28.457243 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:28.457216    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fjv8\" (UniqueName: \"kubernetes.io/projected/3c528012-b068-4867-9a7b-b163828cb54c-kube-api-access-4fjv8\") pod \"test-curl-7702-7f8620\" (UID: \"3c528012-b068-4867-9a7b-b163828cb54c\") " pod="opendatahub/test-curl-7702-7f8620"
Sep 11 21:29:28.468747 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:28.468722    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fjv8\" (UniqueName: \"kubernetes.io/projected/3c528012-b068-4867-9a7b-b163828cb54c-kube-api-access-4fjv8\") pod \"test-curl-7702-7f8620\" (UID: \"3c528012-b068-4867-9a7b-b163828cb54c\") " pod="opendatahub/test-curl-7702-7f8620"
Sep 11 21:29:28.568496 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:28.568465    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7702-7f8620"
Sep 11 21:29:28.854888 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:28.854785    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7702-7f8620"]
Sep 11 21:29:28.857464 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:29:28.857435    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c528012_b068_4867_9a7b_b163828cb54c.slice/crio-13f0d950468bb4962c55761363639ca57aa8e93b512377b9a9049706cfbf9377 WatchSource:0}: Error finding container 13f0d950468bb4962c55761363639ca57aa8e93b512377b9a9049706cfbf9377: Status 404 returned error can't find the container with id 13f0d950468bb4962c55761363639ca57aa8e93b512377b9a9049706cfbf9377
Sep 11 21:29:29.317902 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:29.317863    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7702-7f8620" event={"ID":"3c528012-b068-4867-9a7b-b163828cb54c","Type":"ContainerStarted","Data":"d4dad797fa723d94dde797e01784d03b8344c658a6d19b0f05bbaf7d05dd0377"}
Sep 11 21:29:29.317902 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:29.317909    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7702-7f8620" event={"ID":"3c528012-b068-4867-9a7b-b163828cb54c","Type":"ContainerStarted","Data":"13f0d950468bb4962c55761363639ca57aa8e93b512377b9a9049706cfbf9377"}
Sep 11 21:29:29.318409 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:29.317949    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7702-185f32" podUID="e4104316-9b91-45b1-860e-eed9a5348dfb" containerName="test-curl-7702-185f32" containerID="cri-o://a8d596d885b597a6448410749021f17df1f7eb32d49eb17185726018a1a575ce" gracePeriod=2
Sep 11 21:29:29.322582 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:29:29.322558    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7702-185f32\" is forbidden: User \"system:node:ip-10-0-142-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-224.ec2.internal' and this object" podUID="e4104316-9b91-45b1-860e-eed9a5348dfb" pod="opendatahub/test-curl-7702-185f32"
Sep 11 21:29:29.361487 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:29.361445    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7702-7f8620" podStartSLOduration=1.361430676 podStartE2EDuration="1.361430676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:29:29.360640212 +0000 UTC m=+1436.064609500" watchObservedRunningTime="2026-09-11 21:29:29.361430676 +0000 UTC m=+1436.065399965"
Sep 11 21:29:29.362898 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:29:29.362877    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7702-185f32\" is forbidden: User \"system:node:ip-10-0-142-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-224.ec2.internal' and this object" podUID="e4104316-9b91-45b1-860e-eed9a5348dfb" pod="opendatahub/test-curl-7702-185f32"
Sep 11 21:29:29.985436 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:29.985407    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:29:30.122523 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:30.122444    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7702-7f8620"]
Sep 11 21:29:30.130036 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:30.130006    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7702-7f8620"]
Sep 11 21:29:30.321784 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:29:30.321757    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7702-7f8620" container="test-curl-7702-7f8620"
Sep 11 21:29:30.750060 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:30.750029    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f"]
Sep 11 21:29:30.750300 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:30.750271    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f" podUID="2333549d-7185-4098-94f6-262d7e8dcb39" containerName="payload-pre-processing" containerID="cri-o://7a5e4bd90ffe721d9ce653df13a980cfd6fa9fc45258f3107338791097dd7eee" gracePeriod=30
Sep 11 21:29:30.992154 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:30.992129    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f"
Sep 11 21:29:31.080722 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:31.080619    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rz5g\" (UniqueName: \"kubernetes.io/projected/2333549d-7185-4098-94f6-262d7e8dcb39-kube-api-access-4rz5g\") pod \"2333549d-7185-4098-94f6-262d7e8dcb39\" (UID: \"2333549d-7185-4098-94f6-262d7e8dcb39\") "
Sep 11 21:29:31.080722 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:31.080706    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2333549d-7185-4098-94f6-262d7e8dcb39-plugins-config-volume\") pod \"2333549d-7185-4098-94f6-262d7e8dcb39\" (UID: \"2333549d-7185-4098-94f6-262d7e8dcb39\") "
Sep 11 21:29:31.081111 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:31.081084    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2333549d-7185-4098-94f6-262d7e8dcb39-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2333549d-7185-4098-94f6-262d7e8dcb39" (UID: "2333549d-7185-4098-94f6-262d7e8dcb39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:29:31.082732 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:31.082705    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2333549d-7185-4098-94f6-262d7e8dcb39-kube-api-access-4rz5g" (OuterVolumeSpecName: "kube-api-access-4rz5g") pod "2333549d-7185-4098-94f6-262d7e8dcb39" (UID: "2333549d-7185-4098-94f6-262d7e8dcb39"). InnerVolumeSpecName "kube-api-access-4rz5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:29:31.181921 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:31.181893    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2333549d-7185-4098-94f6-262d7e8dcb39-plugins-config-volume\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:29:31.181921 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:31.181915    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rz5g\" (UniqueName: \"kubernetes.io/projected/2333549d-7185-4098-94f6-262d7e8dcb39-kube-api-access-4rz5g\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:29:31.328504 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:31.328475    2578 generic.go:356] "Generic (PLEG): container finished" podID="2333549d-7185-4098-94f6-262d7e8dcb39" containerID="7a5e4bd90ffe721d9ce653df13a980cfd6fa9fc45258f3107338791097dd7eee" exitCode=0
Sep 11 21:29:31.328904 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:31.328533    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f"
Sep 11 21:29:31.328904 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:31.328552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f" event={"ID":"2333549d-7185-4098-94f6-262d7e8dcb39","Type":"ContainerDied","Data":"7a5e4bd90ffe721d9ce653df13a980cfd6fa9fc45258f3107338791097dd7eee"}
Sep 11 21:29:31.328904 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:31.328609    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f" event={"ID":"2333549d-7185-4098-94f6-262d7e8dcb39","Type":"ContainerDied","Data":"5784a82fdaac2357cdb403a97eb42d898c4e8c226837590da6bf7bfe001296c8"}
Sep 11 21:29:31.328904 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:31.328632    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7702-7f8620" podUID="3c528012-b068-4867-9a7b-b163828cb54c" containerName="test-curl-7702-7f8620" containerID="cri-o://d4dad797fa723d94dde797e01784d03b8344c658a6d19b0f05bbaf7d05dd0377" gracePeriod=2
Sep 11 21:29:31.328904 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:31.328641    2578 scope.go:117] "RemoveContainer" containerID="7a5e4bd90ffe721d9ce653df13a980cfd6fa9fc45258f3107338791097dd7eee"
Sep 11 21:29:31.341166 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:29:31.341133    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7702-7f8620\" is forbidden: User \"system:node:ip-10-0-142-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-224.ec2.internal' and this object" podUID="3c528012-b068-4867-9a7b-b163828cb54c" pod="opendatahub/test-curl-7702-7f8620"
Sep 11 21:29:31.344177 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:29:31.344154    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7702-7f8620\" is forbidden: User \"system:node:ip-10-0-142-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-224.ec2.internal' and this object" podUID="3c528012-b068-4867-9a7b-b163828cb54c" pod="opendatahub/test-curl-7702-7f8620"
Sep 11 21:29:31.374105 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:29:31.374071    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7702-7f8620\" is forbidden: User \"system:node:ip-10-0-142-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-224.ec2.internal' and this object" podUID="3c528012-b068-4867-9a7b-b163828cb54c" pod="opendatahub/test-curl-7702-7f8620"
Sep 11 21:29:31.374332 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:31.374301    2578 scope.go:117] "RemoveContainer" containerID="7a5e4bd90ffe721d9ce653df13a980cfd6fa9fc45258f3107338791097dd7eee"
Sep 11 21:29:31.374683 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:29:31.374657    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a5e4bd90ffe721d9ce653df13a980cfd6fa9fc45258f3107338791097dd7eee\": container with ID starting with 7a5e4bd90ffe721d9ce653df13a980cfd6fa9fc45258f3107338791097dd7eee not found: ID does not exist" containerID="7a5e4bd90ffe721d9ce653df13a980cfd6fa9fc45258f3107338791097dd7eee"
Sep 11 21:29:31.374781 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:31.374688    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a5e4bd90ffe721d9ce653df13a980cfd6fa9fc45258f3107338791097dd7eee"} err="failed to get container status \"7a5e4bd90ffe721d9ce653df13a980cfd6fa9fc45258f3107338791097dd7eee\": rpc error: code = NotFound desc = could not find container \"7a5e4bd90ffe721d9ce653df13a980cfd6fa9fc45258f3107338791097dd7eee\": container with ID starting with 7a5e4bd90ffe721d9ce653df13a980cfd6fa9fc45258f3107338791097dd7eee not found: ID does not exist"
Sep 11 21:29:31.392529 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:31.392305    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f"]
Sep 11 21:29:31.400421 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:31.400399    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-55fb09-66bcf94998-8q22f"]
Sep 11 21:29:31.445330 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:31.445307    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7702-185f32"
Sep 11 21:29:31.447548 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:29:31.447528    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7702-7f8620\" is forbidden: User \"system:node:ip-10-0-142-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-224.ec2.internal' and this object" podUID="3c528012-b068-4867-9a7b-b163828cb54c" pod="opendatahub/test-curl-7702-7f8620"
Sep 11 21:29:31.483852 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:31.483807    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bn6k9\" (UniqueName: \"kubernetes.io/projected/e4104316-9b91-45b1-860e-eed9a5348dfb-kube-api-access-bn6k9\") pod \"e4104316-9b91-45b1-860e-eed9a5348dfb\" (UID: \"e4104316-9b91-45b1-860e-eed9a5348dfb\") "
Sep 11 21:29:31.485800 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:31.485772    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4104316-9b91-45b1-860e-eed9a5348dfb-kube-api-access-bn6k9" (OuterVolumeSpecName: "kube-api-access-bn6k9") pod "e4104316-9b91-45b1-860e-eed9a5348dfb" (UID: "e4104316-9b91-45b1-860e-eed9a5348dfb"). InnerVolumeSpecName "kube-api-access-bn6k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:29:31.585419 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:31.585335    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn6k9\" (UniqueName: \"kubernetes.io/projected/e4104316-9b91-45b1-860e-eed9a5348dfb-kube-api-access-bn6k9\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:29:31.927451 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:31.927420    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2333549d-7185-4098-94f6-262d7e8dcb39" path="/var/lib/kubelet/pods/2333549d-7185-4098-94f6-262d7e8dcb39/volumes"
Sep 11 21:29:31.927738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:31.927725    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4104316-9b91-45b1-860e-eed9a5348dfb" path="/var/lib/kubelet/pods/e4104316-9b91-45b1-860e-eed9a5348dfb/volumes"
Sep 11 21:29:32.335372 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:32.335278    2578 generic.go:356] "Generic (PLEG): container finished" podID="e4104316-9b91-45b1-860e-eed9a5348dfb" containerID="a8d596d885b597a6448410749021f17df1f7eb32d49eb17185726018a1a575ce" exitCode=137
Sep 11 21:29:32.335372 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:32.335327    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7702-185f32"
Sep 11 21:29:32.335372 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:32.335353    2578 scope.go:117] "RemoveContainer" containerID="a8d596d885b597a6448410749021f17df1f7eb32d49eb17185726018a1a575ce"
Sep 11 21:29:32.343169 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:32.343154    2578 scope.go:117] "RemoveContainer" containerID="a8d596d885b597a6448410749021f17df1f7eb32d49eb17185726018a1a575ce"
Sep 11 21:29:32.343395 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:29:32.343376    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d596d885b597a6448410749021f17df1f7eb32d49eb17185726018a1a575ce\": container with ID starting with a8d596d885b597a6448410749021f17df1f7eb32d49eb17185726018a1a575ce not found: ID does not exist" containerID="a8d596d885b597a6448410749021f17df1f7eb32d49eb17185726018a1a575ce"
Sep 11 21:29:32.343455 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:32.343401    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d596d885b597a6448410749021f17df1f7eb32d49eb17185726018a1a575ce"} err="failed to get container status \"a8d596d885b597a6448410749021f17df1f7eb32d49eb17185726018a1a575ce\": rpc error: code = NotFound desc = could not find container \"a8d596d885b597a6448410749021f17df1f7eb32d49eb17185726018a1a575ce\": container with ID starting with a8d596d885b597a6448410749021f17df1f7eb32d49eb17185726018a1a575ce not found: ID does not exist"
Sep 11 21:29:32.630322 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:32.630232    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"]
Sep 11 21:29:33.498816 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:33.498793    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7702-7f8620"
Sep 11 21:29:33.602193 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:33.602110    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fjv8\" (UniqueName: \"kubernetes.io/projected/3c528012-b068-4867-9a7b-b163828cb54c-kube-api-access-4fjv8\") pod \"3c528012-b068-4867-9a7b-b163828cb54c\" (UID: \"3c528012-b068-4867-9a7b-b163828cb54c\") "
Sep 11 21:29:33.604087 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:33.604065    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c528012-b068-4867-9a7b-b163828cb54c-kube-api-access-4fjv8" (OuterVolumeSpecName: "kube-api-access-4fjv8") pod "3c528012-b068-4867-9a7b-b163828cb54c" (UID: "3c528012-b068-4867-9a7b-b163828cb54c"). InnerVolumeSpecName "kube-api-access-4fjv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:29:33.703534 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:33.703498    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fjv8\" (UniqueName: \"kubernetes.io/projected/3c528012-b068-4867-9a7b-b163828cb54c-kube-api-access-4fjv8\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:29:33.781868 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:33.781832    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"]
Sep 11 21:29:33.926812 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:33.926781    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c528012-b068-4867-9a7b-b163828cb54c" path="/var/lib/kubelet/pods/3c528012-b068-4867-9a7b-b163828cb54c/volumes"
Sep 11 21:29:34.175998 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:34.175972    2578 scope.go:117] "RemoveContainer" containerID="ae306c442fef39b03e3b8d6142d23f2bd57072da41de1e7056657fb4df8bb2aa"
Sep 11 21:29:34.346689 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:34.346660    2578 generic.go:356] "Generic (PLEG): container finished" podID="3c528012-b068-4867-9a7b-b163828cb54c" containerID="d4dad797fa723d94dde797e01784d03b8344c658a6d19b0f05bbaf7d05dd0377" exitCode=137
Sep 11 21:29:34.346689 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:34.346700    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7702-7f8620"
Sep 11 21:29:34.346973 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:34.346740    2578 scope.go:117] "RemoveContainer" containerID="d4dad797fa723d94dde797e01784d03b8344c658a6d19b0f05bbaf7d05dd0377"
Sep 11 21:29:34.354620 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:34.354603    2578 scope.go:117] "RemoveContainer" containerID="d4dad797fa723d94dde797e01784d03b8344c658a6d19b0f05bbaf7d05dd0377"
Sep 11 21:29:34.354904 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:29:34.354882    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4dad797fa723d94dde797e01784d03b8344c658a6d19b0f05bbaf7d05dd0377\": container with ID starting with d4dad797fa723d94dde797e01784d03b8344c658a6d19b0f05bbaf7d05dd0377 not found: ID does not exist" containerID="d4dad797fa723d94dde797e01784d03b8344c658a6d19b0f05bbaf7d05dd0377"
Sep 11 21:29:34.355045 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:34.354911    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4dad797fa723d94dde797e01784d03b8344c658a6d19b0f05bbaf7d05dd0377"} err="failed to get container status \"d4dad797fa723d94dde797e01784d03b8344c658a6d19b0f05bbaf7d05dd0377\": rpc error: code = NotFound desc = could not find container \"d4dad797fa723d94dde797e01784d03b8344c658a6d19b0f05bbaf7d05dd0377\": container with ID starting with d4dad797fa723d94dde797e01784d03b8344c658a6d19b0f05bbaf7d05dd0377 not found: ID does not exist"
Sep 11 21:29:43.401385 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:43.401342    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:29:47.648222 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:47.648159    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v" podUID="7a38851a-0e48-4999-827f-f83cb10156d3" containerName="main" containerID="cri-o://2a0052c2f5097c32f7798d442c7c2d15fc5c94c2366ddb0ec82b4f7915dde1fb" gracePeriod=45
Sep 11 21:29:47.907315 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:47.907254    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:29:48.029676 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.029649    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x45pm\" (UniqueName: \"kubernetes.io/projected/7a38851a-0e48-4999-827f-f83cb10156d3-kube-api-access-x45pm\") pod \"7a38851a-0e48-4999-827f-f83cb10156d3\" (UID: \"7a38851a-0e48-4999-827f-f83cb10156d3\") "
Sep 11 21:29:48.029877 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.029716    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a38851a-0e48-4999-827f-f83cb10156d3-home\") pod \"7a38851a-0e48-4999-827f-f83cb10156d3\" (UID: \"7a38851a-0e48-4999-827f-f83cb10156d3\") "
Sep 11 21:29:48.029877 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.029742    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a38851a-0e48-4999-827f-f83cb10156d3-tmp-dir\") pod \"7a38851a-0e48-4999-827f-f83cb10156d3\" (UID: \"7a38851a-0e48-4999-827f-f83cb10156d3\") "
Sep 11 21:29:48.029877 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.029764    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a38851a-0e48-4999-827f-f83cb10156d3-tls-certs\") pod \"7a38851a-0e48-4999-827f-f83cb10156d3\" (UID: \"7a38851a-0e48-4999-827f-f83cb10156d3\") "
Sep 11 21:29:48.029877 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.029816    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a38851a-0e48-4999-827f-f83cb10156d3-model-cache\") pod \"7a38851a-0e48-4999-827f-f83cb10156d3\" (UID: \"7a38851a-0e48-4999-827f-f83cb10156d3\") "
Sep 11 21:29:48.029877 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.029861    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a38851a-0e48-4999-827f-f83cb10156d3-dshm\") pod \"7a38851a-0e48-4999-827f-f83cb10156d3\" (UID: \"7a38851a-0e48-4999-827f-f83cb10156d3\") "
Sep 11 21:29:48.030157 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.029989    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a38851a-0e48-4999-827f-f83cb10156d3-home" (OuterVolumeSpecName: "home") pod "7a38851a-0e48-4999-827f-f83cb10156d3" (UID: "7a38851a-0e48-4999-827f-f83cb10156d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:29:48.030157 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.030057    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a38851a-0e48-4999-827f-f83cb10156d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7a38851a-0e48-4999-827f-f83cb10156d3" (UID: "7a38851a-0e48-4999-827f-f83cb10156d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:29:48.030157 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.030074    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a38851a-0e48-4999-827f-f83cb10156d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a38851a-0e48-4999-827f-f83cb10156d3" (UID: "7a38851a-0e48-4999-827f-f83cb10156d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:29:48.030157 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.030108    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a38851a-0e48-4999-827f-f83cb10156d3-home\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:29:48.030157 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.030119    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a38851a-0e48-4999-827f-f83cb10156d3-tmp-dir\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:29:48.031914 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.031887    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a38851a-0e48-4999-827f-f83cb10156d3-kube-api-access-x45pm" (OuterVolumeSpecName: "kube-api-access-x45pm") pod "7a38851a-0e48-4999-827f-f83cb10156d3" (UID: "7a38851a-0e48-4999-827f-f83cb10156d3"). InnerVolumeSpecName "kube-api-access-x45pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:29:48.032148 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.032120    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a38851a-0e48-4999-827f-f83cb10156d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a38851a-0e48-4999-827f-f83cb10156d3" (UID: "7a38851a-0e48-4999-827f-f83cb10156d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:29:48.032148 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.032127    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a38851a-0e48-4999-827f-f83cb10156d3-dshm" (OuterVolumeSpecName: "dshm") pod "7a38851a-0e48-4999-827f-f83cb10156d3" (UID: "7a38851a-0e48-4999-827f-f83cb10156d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:29:48.131198 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.131167    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a38851a-0e48-4999-827f-f83cb10156d3-tls-certs\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:29:48.131198 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.131192    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a38851a-0e48-4999-827f-f83cb10156d3-model-cache\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:29:48.131198 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.131200    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a38851a-0e48-4999-827f-f83cb10156d3-dshm\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:29:48.131198 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.131208    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x45pm\" (UniqueName: \"kubernetes.io/projected/7a38851a-0e48-4999-827f-f83cb10156d3-kube-api-access-x45pm\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:29:48.409718 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.409681    2578 generic.go:356] "Generic (PLEG): container finished" podID="7a38851a-0e48-4999-827f-f83cb10156d3" containerID="2a0052c2f5097c32f7798d442c7c2d15fc5c94c2366ddb0ec82b4f7915dde1fb" exitCode=0
Sep 11 21:29:48.409914 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.409722    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"
Sep 11 21:29:48.409914 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.409760    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v" event={"ID":"7a38851a-0e48-4999-827f-f83cb10156d3","Type":"ContainerDied","Data":"2a0052c2f5097c32f7798d442c7c2d15fc5c94c2366ddb0ec82b4f7915dde1fb"}
Sep 11 21:29:48.409914 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.409802    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v" event={"ID":"7a38851a-0e48-4999-827f-f83cb10156d3","Type":"ContainerDied","Data":"78d9aaf5f1dbeee35579d341904910e81700df3bdb04e464cb8a955a4e1935ba"}
Sep 11 21:29:48.409914 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.409831    2578 scope.go:117] "RemoveContainer" containerID="2a0052c2f5097c32f7798d442c7c2d15fc5c94c2366ddb0ec82b4f7915dde1fb"
Sep 11 21:29:48.418014 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.417999    2578 scope.go:117] "RemoveContainer" containerID="2a0052c2f5097c32f7798d442c7c2d15fc5c94c2366ddb0ec82b4f7915dde1fb"
Sep 11 21:29:48.418262 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:29:48.418243    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a0052c2f5097c32f7798d442c7c2d15fc5c94c2366ddb0ec82b4f7915dde1fb\": container with ID starting with 2a0052c2f5097c32f7798d442c7c2d15fc5c94c2366ddb0ec82b4f7915dde1fb not found: ID does not exist" containerID="2a0052c2f5097c32f7798d442c7c2d15fc5c94c2366ddb0ec82b4f7915dde1fb"
Sep 11 21:29:48.418323 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.418269    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a0052c2f5097c32f7798d442c7c2d15fc5c94c2366ddb0ec82b4f7915dde1fb"} err="failed to get container status \"2a0052c2f5097c32f7798d442c7c2d15fc5c94c2366ddb0ec82b4f7915dde1fb\": rpc error: code = NotFound desc = could not find container \"2a0052c2f5097c32f7798d442c7c2d15fc5c94c2366ddb0ec82b4f7915dde1fb\": container with ID starting with 2a0052c2f5097c32f7798d442c7c2d15fc5c94c2366ddb0ec82b4f7915dde1fb not found: ID does not exist"
Sep 11 21:29:48.444473 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.444446    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"]
Sep 11 21:29:48.454723 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.454699    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v"]
Sep 11 21:29:48.799733 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:48.799668    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz" podUID="8e9b5001-ff00-4bce-8a7a-0a58d689aea8" containerName="main" containerID="cri-o://7b50f93eb8f1689d9822ae8ed178ec49179d520f5092ca9b1c8c62fef21461cc" gracePeriod=45
Sep 11 21:29:49.044873 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.044851    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:29:49.139966 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.139878    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-tls-certs\") pod \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\" (UID: \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\") "
Sep 11 21:29:49.139966 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.139913    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-tmp-dir\") pod \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\" (UID: \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\") "
Sep 11 21:29:49.140200 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.139988    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2klv\" (UniqueName: \"kubernetes.io/projected/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-kube-api-access-g2klv\") pod \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\" (UID: \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\") "
Sep 11 21:29:49.140200 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.140002    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-dshm\") pod \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\" (UID: \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\") "
Sep 11 21:29:49.140200 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.140031    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-model-cache\") pod \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\" (UID: \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\") "
Sep 11 21:29:49.140200 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.140050    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-home\") pod \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\" (UID: \"8e9b5001-ff00-4bce-8a7a-0a58d689aea8\") "
Sep 11 21:29:49.140405 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.140260    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8e9b5001-ff00-4bce-8a7a-0a58d689aea8" (UID: "8e9b5001-ff00-4bce-8a7a-0a58d689aea8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:29:49.140405 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.140302    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-model-cache" (OuterVolumeSpecName: "model-cache") pod "8e9b5001-ff00-4bce-8a7a-0a58d689aea8" (UID: "8e9b5001-ff00-4bce-8a7a-0a58d689aea8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:29:49.140405 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.140366    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-home" (OuterVolumeSpecName: "home") pod "8e9b5001-ff00-4bce-8a7a-0a58d689aea8" (UID: "8e9b5001-ff00-4bce-8a7a-0a58d689aea8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:29:49.142168 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.142136    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-dshm" (OuterVolumeSpecName: "dshm") pod "8e9b5001-ff00-4bce-8a7a-0a58d689aea8" (UID: "8e9b5001-ff00-4bce-8a7a-0a58d689aea8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:29:49.142268 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.142193    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e9b5001-ff00-4bce-8a7a-0a58d689aea8" (UID: "8e9b5001-ff00-4bce-8a7a-0a58d689aea8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:29:49.142268 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.142233    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-kube-api-access-g2klv" (OuterVolumeSpecName: "kube-api-access-g2klv") pod "8e9b5001-ff00-4bce-8a7a-0a58d689aea8" (UID: "8e9b5001-ff00-4bce-8a7a-0a58d689aea8"). InnerVolumeSpecName "kube-api-access-g2klv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:29:49.241499 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.241458    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2klv\" (UniqueName: \"kubernetes.io/projected/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-kube-api-access-g2klv\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:29:49.241499 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.241496    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-dshm\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:29:49.241499 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.241506    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-model-cache\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:29:49.241499 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.241514    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-home\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:29:49.241753 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.241522    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-tls-certs\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:29:49.241753 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.241529    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e9b5001-ff00-4bce-8a7a-0a58d689aea8-tmp-dir\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:29:49.415764 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.415682    2578 generic.go:356] "Generic (PLEG): container finished" podID="8e9b5001-ff00-4bce-8a7a-0a58d689aea8" containerID="7b50f93eb8f1689d9822ae8ed178ec49179d520f5092ca9b1c8c62fef21461cc" exitCode=0
Sep 11 21:29:49.415764 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.415727    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"
Sep 11 21:29:49.415764 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.415739    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz" event={"ID":"8e9b5001-ff00-4bce-8a7a-0a58d689aea8","Type":"ContainerDied","Data":"7b50f93eb8f1689d9822ae8ed178ec49179d520f5092ca9b1c8c62fef21461cc"}
Sep 11 21:29:49.416030 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.415775    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz" event={"ID":"8e9b5001-ff00-4bce-8a7a-0a58d689aea8","Type":"ContainerDied","Data":"c706aa188f1069d1db812f6ebcd3f1dd57633a344b6645e20c79f53a0f8483f2"}
Sep 11 21:29:49.416030 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.415795    2578 scope.go:117] "RemoveContainer" containerID="7b50f93eb8f1689d9822ae8ed178ec49179d520f5092ca9b1c8c62fef21461cc"
Sep 11 21:29:49.425640 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.425623    2578 scope.go:117] "RemoveContainer" containerID="7b50f93eb8f1689d9822ae8ed178ec49179d520f5092ca9b1c8c62fef21461cc"
Sep 11 21:29:49.425950 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:29:49.425925    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b50f93eb8f1689d9822ae8ed178ec49179d520f5092ca9b1c8c62fef21461cc\": container with ID starting with 7b50f93eb8f1689d9822ae8ed178ec49179d520f5092ca9b1c8c62fef21461cc not found: ID does not exist" containerID="7b50f93eb8f1689d9822ae8ed178ec49179d520f5092ca9b1c8c62fef21461cc"
Sep 11 21:29:49.426045 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.425955    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b50f93eb8f1689d9822ae8ed178ec49179d520f5092ca9b1c8c62fef21461cc"} err="failed to get container status \"7b50f93eb8f1689d9822ae8ed178ec49179d520f5092ca9b1c8c62fef21461cc\": rpc error: code = NotFound desc = could not find container \"7b50f93eb8f1689d9822ae8ed178ec49179d520f5092ca9b1c8c62fef21461cc\": container with ID starting with 7b50f93eb8f1689d9822ae8ed178ec49179d520f5092ca9b1c8c62fef21461cc not found: ID does not exist"
Sep 11 21:29:49.441684 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.441659    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"]
Sep 11 21:29:49.445109 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.445089    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-713791/sub-test-model-713791-kserve-7d76d847ff-jmnnz"]
Sep 11 21:29:49.928146 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.928116    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a38851a-0e48-4999-827f-f83cb10156d3" path="/var/lib/kubelet/pods/7a38851a-0e48-4999-827f-f83cb10156d3/volumes"
Sep 11 21:29:49.928511 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:49.928501    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e9b5001-ff00-4bce-8a7a-0a58d689aea8" path="/var/lib/kubelet/pods/8e9b5001-ff00-4bce-8a7a-0a58d689aea8/volumes"
Sep 11 21:29:50.804788 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:50.804756    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:29:52.893679 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:29:52.893633    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-2bd772/sub-test-model-2bd772-kserve-6fd88449bc-dxr9v" podUID="7a38851a-0e48-4999-827f-f83cb10156d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Sep 11 21:30:00.165163 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:00.165123    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819370-bvjnh"]
Sep 11 21:30:00.165615 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:00.165412    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2333549d-7185-4098-94f6-262d7e8dcb39" containerName="payload-pre-processing"
Sep 11 21:30:00.165615 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:00.165422    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2333549d-7185-4098-94f6-262d7e8dcb39" containerName="payload-pre-processing"
Sep 11 21:30:00.165615 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:00.165435    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c528012-b068-4867-9a7b-b163828cb54c" containerName="test-curl-7702-7f8620"
Sep 11 21:30:00.165615 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:00.165441    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c528012-b068-4867-9a7b-b163828cb54c" containerName="test-curl-7702-7f8620"
Sep 11 21:30:00.165615 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:00.165454    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e9b5001-ff00-4bce-8a7a-0a58d689aea8" containerName="main"
Sep 11 21:30:00.165615 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:00.165460    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e9b5001-ff00-4bce-8a7a-0a58d689aea8" containerName="main"
Sep 11 21:30:00.165615 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:00.165468    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a38851a-0e48-4999-827f-f83cb10156d3" containerName="main"
Sep 11 21:30:00.165615 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:00.165473    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a38851a-0e48-4999-827f-f83cb10156d3" containerName="main"
Sep 11 21:30:00.165615 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:00.165519    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a38851a-0e48-4999-827f-f83cb10156d3" containerName="main"
Sep 11 21:30:00.165615 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:00.165527    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e9b5001-ff00-4bce-8a7a-0a58d689aea8" containerName="main"
Sep 11 21:30:00.165615 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:00.165534    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c528012-b068-4867-9a7b-b163828cb54c" containerName="test-curl-7702-7f8620"
Sep 11 21:30:00.165615 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:00.165540    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2333549d-7185-4098-94f6-262d7e8dcb39" containerName="payload-pre-processing"
Sep 11 21:30:00.169542 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:00.169525    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819370-bvjnh"
Sep 11 21:30:00.172723 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:00.172701    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfm9g\""
Sep 11 21:30:00.178104 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:00.178089    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819370-bvjnh"
Sep 11 21:30:00.209571 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:00.209524    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819370-bvjnh"]
Sep 11 21:30:00.456259 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:00.456224    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819370-bvjnh"]
Sep 11 21:30:00.459090 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:30:00.459047    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e85ed78_f7a5_436b_ae76_941ad037f7b4.slice/crio-49fd0ea585e6ea7e95d21e6b90bc2b194aca23193a57e7d854d323da486e9561 WatchSource:0}: Error finding container 49fd0ea585e6ea7e95d21e6b90bc2b194aca23193a57e7d854d323da486e9561: Status 404 returned error can't find the container with id 49fd0ea585e6ea7e95d21e6b90bc2b194aca23193a57e7d854d323da486e9561
Sep 11 21:30:01.468854 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:01.468803    2578 generic.go:356] "Generic (PLEG): container finished" podID="9e85ed78-f7a5-436b-ae76-941ad037f7b4" containerID="50598e7a4bd18b724c524978aec528e34541c857800530fc718f8f2c69b8554e" exitCode=0
Sep 11 21:30:01.469216 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:01.468886    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819370-bvjnh" event={"ID":"9e85ed78-f7a5-436b-ae76-941ad037f7b4","Type":"ContainerDied","Data":"50598e7a4bd18b724c524978aec528e34541c857800530fc718f8f2c69b8554e"}
Sep 11 21:30:01.469216 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:01.468936    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819370-bvjnh" event={"ID":"9e85ed78-f7a5-436b-ae76-941ad037f7b4","Type":"ContainerStarted","Data":"49fd0ea585e6ea7e95d21e6b90bc2b194aca23193a57e7d854d323da486e9561"}
Sep 11 21:30:02.598138 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:02.598113    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819370-bvjnh"
Sep 11 21:30:03.478080 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:03.478056    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819370-bvjnh"
Sep 11 21:30:03.478281 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:03.478050    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819370-bvjnh" event={"ID":"9e85ed78-f7a5-436b-ae76-941ad037f7b4","Type":"ContainerDied","Data":"49fd0ea585e6ea7e95d21e6b90bc2b194aca23193a57e7d854d323da486e9561"}
Sep 11 21:30:03.478281 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:03.478175    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49fd0ea585e6ea7e95d21e6b90bc2b194aca23193a57e7d854d323da486e9561"
Sep 11 21:30:07.257562 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.257532    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"]
Sep 11 21:30:07.257966 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.257857    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e85ed78-f7a5-436b-ae76-941ad037f7b4" containerName="cleanup"
Sep 11 21:30:07.257966 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.257870    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e85ed78-f7a5-436b-ae76-941ad037f7b4" containerName="cleanup"
Sep 11 21:30:07.257966 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.257934    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e85ed78-f7a5-436b-ae76-941ad037f7b4" containerName="cleanup"
Sep 11 21:30:07.260706 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.260688    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:30:07.263324 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.263304    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-713791\"/\"kube-root-ca.crt\""
Sep 11 21:30:07.264131 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.264115    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-713791\"/\"openshift-service-ca.crt\""
Sep 11 21:30:07.264184 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.264170    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-713791\"/\"default-dockercfg-bhxr5\""
Sep 11 21:30:07.264310 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.264124    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-713791\"/\"rate-test-model-713791-kserve-self-signed-certs\""
Sep 11 21:30:07.274590 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.274568    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"]
Sep 11 21:30:07.393683 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.393645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/044c4b48-f659-4ba3-8461-09efb47d0bb2-tmp-dir\") pod \"rate-test-model-713791-kserve-68b8987fd7-mfkfg\" (UID: \"044c4b48-f659-4ba3-8461-09efb47d0bb2\") " pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:30:07.393683 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.393686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/044c4b48-f659-4ba3-8461-09efb47d0bb2-tls-certs\") pod \"rate-test-model-713791-kserve-68b8987fd7-mfkfg\" (UID: \"044c4b48-f659-4ba3-8461-09efb47d0bb2\") " pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:30:07.393879 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.393748    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/044c4b48-f659-4ba3-8461-09efb47d0bb2-dshm\") pod \"rate-test-model-713791-kserve-68b8987fd7-mfkfg\" (UID: \"044c4b48-f659-4ba3-8461-09efb47d0bb2\") " pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:30:07.393879 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.393769    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj79q\" (UniqueName: \"kubernetes.io/projected/044c4b48-f659-4ba3-8461-09efb47d0bb2-kube-api-access-kj79q\") pod \"rate-test-model-713791-kserve-68b8987fd7-mfkfg\" (UID: \"044c4b48-f659-4ba3-8461-09efb47d0bb2\") " pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:30:07.393879 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.393814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/044c4b48-f659-4ba3-8461-09efb47d0bb2-home\") pod \"rate-test-model-713791-kserve-68b8987fd7-mfkfg\" (UID: \"044c4b48-f659-4ba3-8461-09efb47d0bb2\") " pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:30:07.393996 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.393900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/044c4b48-f659-4ba3-8461-09efb47d0bb2-model-cache\") pod \"rate-test-model-713791-kserve-68b8987fd7-mfkfg\" (UID: \"044c4b48-f659-4ba3-8461-09efb47d0bb2\") " pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:30:07.494523 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.494488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/044c4b48-f659-4ba3-8461-09efb47d0bb2-model-cache\") pod \"rate-test-model-713791-kserve-68b8987fd7-mfkfg\" (UID: \"044c4b48-f659-4ba3-8461-09efb47d0bb2\") " pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:30:07.494701 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.494538    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/044c4b48-f659-4ba3-8461-09efb47d0bb2-tmp-dir\") pod \"rate-test-model-713791-kserve-68b8987fd7-mfkfg\" (UID: \"044c4b48-f659-4ba3-8461-09efb47d0bb2\") " pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:30:07.494701 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.494644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/044c4b48-f659-4ba3-8461-09efb47d0bb2-tls-certs\") pod \"rate-test-model-713791-kserve-68b8987fd7-mfkfg\" (UID: \"044c4b48-f659-4ba3-8461-09efb47d0bb2\") " pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:30:07.494789 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.494728    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/044c4b48-f659-4ba3-8461-09efb47d0bb2-dshm\") pod \"rate-test-model-713791-kserve-68b8987fd7-mfkfg\" (UID: \"044c4b48-f659-4ba3-8461-09efb47d0bb2\") " pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:30:07.494789 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.494761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj79q\" (UniqueName: \"kubernetes.io/projected/044c4b48-f659-4ba3-8461-09efb47d0bb2-kube-api-access-kj79q\") pod \"rate-test-model-713791-kserve-68b8987fd7-mfkfg\" (UID: \"044c4b48-f659-4ba3-8461-09efb47d0bb2\") " pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:30:07.494934 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.494788    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/044c4b48-f659-4ba3-8461-09efb47d0bb2-home\") pod \"rate-test-model-713791-kserve-68b8987fd7-mfkfg\" (UID: \"044c4b48-f659-4ba3-8461-09efb47d0bb2\") " pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:30:07.494993 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.494962    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/044c4b48-f659-4ba3-8461-09efb47d0bb2-model-cache\") pod \"rate-test-model-713791-kserve-68b8987fd7-mfkfg\" (UID: \"044c4b48-f659-4ba3-8461-09efb47d0bb2\") " pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:30:07.495043 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.495000    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/044c4b48-f659-4ba3-8461-09efb47d0bb2-tmp-dir\") pod \"rate-test-model-713791-kserve-68b8987fd7-mfkfg\" (UID: \"044c4b48-f659-4ba3-8461-09efb47d0bb2\") " pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:30:07.495113 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.495096    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/044c4b48-f659-4ba3-8461-09efb47d0bb2-home\") pod \"rate-test-model-713791-kserve-68b8987fd7-mfkfg\" (UID: \"044c4b48-f659-4ba3-8461-09efb47d0bb2\") " pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:30:07.496961 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.496935    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/044c4b48-f659-4ba3-8461-09efb47d0bb2-dshm\") pod \"rate-test-model-713791-kserve-68b8987fd7-mfkfg\" (UID: \"044c4b48-f659-4ba3-8461-09efb47d0bb2\") " pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:30:07.497195 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.497180    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/044c4b48-f659-4ba3-8461-09efb47d0bb2-tls-certs\") pod \"rate-test-model-713791-kserve-68b8987fd7-mfkfg\" (UID: \"044c4b48-f659-4ba3-8461-09efb47d0bb2\") " pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:30:07.503454 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.503430    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj79q\" (UniqueName: \"kubernetes.io/projected/044c4b48-f659-4ba3-8461-09efb47d0bb2-kube-api-access-kj79q\") pod \"rate-test-model-713791-kserve-68b8987fd7-mfkfg\" (UID: \"044c4b48-f659-4ba3-8461-09efb47d0bb2\") " pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:30:07.569260 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.569196    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:30:07.865468 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:07.865386    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"]
Sep 11 21:30:07.868613 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:30:07.868579    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod044c4b48_f659_4ba3_8461_09efb47d0bb2.slice/crio-d12308a667af9dea191f002b4f9badcfa378e6e1d7fc868cff201679312c2352 WatchSource:0}: Error finding container d12308a667af9dea191f002b4f9badcfa378e6e1d7fc868cff201679312c2352: Status 404 returned error can't find the container with id d12308a667af9dea191f002b4f9badcfa378e6e1d7fc868cff201679312c2352
Sep 11 21:30:08.500642 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:08.500602    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg" event={"ID":"044c4b48-f659-4ba3-8461-09efb47d0bb2","Type":"ContainerStarted","Data":"3e44903e26187f6d699a61b1ee51f121d0bd09d8c723bef8113b84207dbe7fda"}
Sep 11 21:30:08.501009 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:08.500654    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg" event={"ID":"044c4b48-f659-4ba3-8461-09efb47d0bb2","Type":"ContainerStarted","Data":"d12308a667af9dea191f002b4f9badcfa378e6e1d7fc868cff201679312c2352"}
Sep 11 21:30:08.522546 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:08.522504    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg" podStartSLOduration=1.522489529 podStartE2EDuration="1.522489529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:30:08.520302885 +0000 UTC m=+1475.224272174" watchObservedRunningTime="2026-09-11 21:30:08.522489529 +0000 UTC m=+1475.226458817"
Sep 11 21:30:11.149524 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:11.149489    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:30:17.569907 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:17.569872    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:30:17.570286 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:17.569949    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:30:17.582073 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:17.582043    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:30:18.553262 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:18.553232    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:30:31.542614 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:31.542578    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:30:47.134240 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:47.134208    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:30:51.821933 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:51.821903    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"]
Sep 11 21:30:58.430379 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:30:58.430345    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:31:06.839622 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:06.839563    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg" podUID="044c4b48-f659-4ba3-8461-09efb47d0bb2" containerName="main" containerID="cri-o://3e44903e26187f6d699a61b1ee51f121d0bd09d8c723bef8113b84207dbe7fda" gracePeriod=45
Sep 11 21:31:07.076274 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.076251    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:31:07.101813 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.101750    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/044c4b48-f659-4ba3-8461-09efb47d0bb2-tmp-dir\") pod \"044c4b48-f659-4ba3-8461-09efb47d0bb2\" (UID: \"044c4b48-f659-4ba3-8461-09efb47d0bb2\") "
Sep 11 21:31:07.101813 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.101786    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/044c4b48-f659-4ba3-8461-09efb47d0bb2-model-cache\") pod \"044c4b48-f659-4ba3-8461-09efb47d0bb2\" (UID: \"044c4b48-f659-4ba3-8461-09efb47d0bb2\") "
Sep 11 21:31:07.102035 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.101853    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kj79q\" (UniqueName: \"kubernetes.io/projected/044c4b48-f659-4ba3-8461-09efb47d0bb2-kube-api-access-kj79q\") pod \"044c4b48-f659-4ba3-8461-09efb47d0bb2\" (UID: \"044c4b48-f659-4ba3-8461-09efb47d0bb2\") "
Sep 11 21:31:07.102035 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.101894    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/044c4b48-f659-4ba3-8461-09efb47d0bb2-home\") pod \"044c4b48-f659-4ba3-8461-09efb47d0bb2\" (UID: \"044c4b48-f659-4ba3-8461-09efb47d0bb2\") "
Sep 11 21:31:07.102035 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.101927    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/044c4b48-f659-4ba3-8461-09efb47d0bb2-dshm\") pod \"044c4b48-f659-4ba3-8461-09efb47d0bb2\" (UID: \"044c4b48-f659-4ba3-8461-09efb47d0bb2\") "
Sep 11 21:31:07.102035 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.101958    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/044c4b48-f659-4ba3-8461-09efb47d0bb2-tls-certs\") pod \"044c4b48-f659-4ba3-8461-09efb47d0bb2\" (UID: \"044c4b48-f659-4ba3-8461-09efb47d0bb2\") "
Sep 11 21:31:07.102221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.102002    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044c4b48-f659-4ba3-8461-09efb47d0bb2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "044c4b48-f659-4ba3-8461-09efb47d0bb2" (UID: "044c4b48-f659-4ba3-8461-09efb47d0bb2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:31:07.102221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.102134    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044c4b48-f659-4ba3-8461-09efb47d0bb2-model-cache" (OuterVolumeSpecName: "model-cache") pod "044c4b48-f659-4ba3-8461-09efb47d0bb2" (UID: "044c4b48-f659-4ba3-8461-09efb47d0bb2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:31:07.102221 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.102155    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044c4b48-f659-4ba3-8461-09efb47d0bb2-home" (OuterVolumeSpecName: "home") pod "044c4b48-f659-4ba3-8461-09efb47d0bb2" (UID: "044c4b48-f659-4ba3-8461-09efb47d0bb2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:31:07.104055 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.104030    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044c4b48-f659-4ba3-8461-09efb47d0bb2-dshm" (OuterVolumeSpecName: "dshm") pod "044c4b48-f659-4ba3-8461-09efb47d0bb2" (UID: "044c4b48-f659-4ba3-8461-09efb47d0bb2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:31:07.104140 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.104053    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/044c4b48-f659-4ba3-8461-09efb47d0bb2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "044c4b48-f659-4ba3-8461-09efb47d0bb2" (UID: "044c4b48-f659-4ba3-8461-09efb47d0bb2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:31:07.104140 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.104126    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044c4b48-f659-4ba3-8461-09efb47d0bb2-kube-api-access-kj79q" (OuterVolumeSpecName: "kube-api-access-kj79q") pod "044c4b48-f659-4ba3-8461-09efb47d0bb2" (UID: "044c4b48-f659-4ba3-8461-09efb47d0bb2"). InnerVolumeSpecName "kube-api-access-kj79q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:31:07.203003 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.202975    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/044c4b48-f659-4ba3-8461-09efb47d0bb2-dshm\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:31:07.203003 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.203000    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/044c4b48-f659-4ba3-8461-09efb47d0bb2-tls-certs\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:31:07.203003 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.203010    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/044c4b48-f659-4ba3-8461-09efb47d0bb2-tmp-dir\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:31:07.203230 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.203019    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/044c4b48-f659-4ba3-8461-09efb47d0bb2-model-cache\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:31:07.203230 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.203028    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj79q\" (UniqueName: \"kubernetes.io/projected/044c4b48-f659-4ba3-8461-09efb47d0bb2-kube-api-access-kj79q\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:31:07.203230 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.203036    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/044c4b48-f659-4ba3-8461-09efb47d0bb2-home\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:31:07.758301 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.758270    2578 generic.go:356] "Generic (PLEG): container finished" podID="044c4b48-f659-4ba3-8461-09efb47d0bb2" containerID="3e44903e26187f6d699a61b1ee51f121d0bd09d8c723bef8113b84207dbe7fda" exitCode=0
Sep 11 21:31:07.758506 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.758309    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"
Sep 11 21:31:07.758506 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.758352    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg" event={"ID":"044c4b48-f659-4ba3-8461-09efb47d0bb2","Type":"ContainerDied","Data":"3e44903e26187f6d699a61b1ee51f121d0bd09d8c723bef8113b84207dbe7fda"}
Sep 11 21:31:07.758506 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.758395    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg" event={"ID":"044c4b48-f659-4ba3-8461-09efb47d0bb2","Type":"ContainerDied","Data":"d12308a667af9dea191f002b4f9badcfa378e6e1d7fc868cff201679312c2352"}
Sep 11 21:31:07.758506 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.758413    2578 scope.go:117] "RemoveContainer" containerID="3e44903e26187f6d699a61b1ee51f121d0bd09d8c723bef8113b84207dbe7fda"
Sep 11 21:31:07.766999 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.766982    2578 scope.go:117] "RemoveContainer" containerID="3e44903e26187f6d699a61b1ee51f121d0bd09d8c723bef8113b84207dbe7fda"
Sep 11 21:31:07.767268 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:31:07.767251    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e44903e26187f6d699a61b1ee51f121d0bd09d8c723bef8113b84207dbe7fda\": container with ID starting with 3e44903e26187f6d699a61b1ee51f121d0bd09d8c723bef8113b84207dbe7fda not found: ID does not exist" containerID="3e44903e26187f6d699a61b1ee51f121d0bd09d8c723bef8113b84207dbe7fda"
Sep 11 21:31:07.767316 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.767278    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e44903e26187f6d699a61b1ee51f121d0bd09d8c723bef8113b84207dbe7fda"} err="failed to get container status \"3e44903e26187f6d699a61b1ee51f121d0bd09d8c723bef8113b84207dbe7fda\": rpc error: code = NotFound desc = could not find container \"3e44903e26187f6d699a61b1ee51f121d0bd09d8c723bef8113b84207dbe7fda\": container with ID starting with 3e44903e26187f6d699a61b1ee51f121d0bd09d8c723bef8113b84207dbe7fda not found: ID does not exist"
Sep 11 21:31:07.781721 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.781698    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"]
Sep 11 21:31:07.784797 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.784776    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-713791/rate-test-model-713791-kserve-68b8987fd7-mfkfg"]
Sep 11 21:31:07.927419 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:07.927386    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="044c4b48-f659-4ba3-8461-09efb47d0bb2" path="/var/lib/kubelet/pods/044c4b48-f659-4ba3-8461-09efb47d0bb2/volumes"
Sep 11 21:31:11.028118 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:11.028082    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:31:27.481832 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.481796    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"]
Sep 11 21:31:27.482270 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.482110    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="044c4b48-f659-4ba3-8461-09efb47d0bb2" containerName="main"
Sep 11 21:31:27.482270 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.482122    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="044c4b48-f659-4ba3-8461-09efb47d0bb2" containerName="main"
Sep 11 21:31:27.482270 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.482174    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="044c4b48-f659-4ba3-8461-09efb47d0bb2" containerName="main"
Sep 11 21:31:27.486240 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.486219    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.489269 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.489246    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-e9c558-openshift-default-dockercfg-vs8k5\""
Sep 11 21:31:27.517994 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.517969    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"]
Sep 11 21:31:27.565617 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.565585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istio-token\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.565617 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.565616    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istiod-ca-cert\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.565853 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.565635    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jwlg\" (UniqueName: \"kubernetes.io/projected/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-kube-api-access-9jwlg\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.565853 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.565665    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istio-data\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.565853 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.565713    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-workload-certs\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.565853 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.565751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-workload-socket\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.565853 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.565764    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istio-envoy\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.565853 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.565802    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istio-podinfo\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.565853 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.565841    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-credential-socket\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.666387 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.666356    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-workload-certs\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.666533 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.666409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-workload-socket\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.666533 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.666430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istio-envoy\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.666533 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.666473    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istio-podinfo\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.666533 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.666496    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-credential-socket\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.666533 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.666528    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istio-token\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.666784 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.666632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istiod-ca-cert\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.666784 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.666682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jwlg\" (UniqueName: \"kubernetes.io/projected/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-kube-api-access-9jwlg\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.666784 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.666714    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istio-data\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.666784 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.666776    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-workload-socket\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.667035 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.666912    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-credential-socket\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.667095 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.667033    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istio-data\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.667151 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.667098    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-workload-certs\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.667402 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.667382    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istiod-ca-cert\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.668748 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.668731    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istio-envoy\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.668884 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.668865    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istio-podinfo\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.681927 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.681904    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istio-token\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.682186 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.682165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jwlg\" (UniqueName: \"kubernetes.io/projected/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-kube-api-access-9jwlg\") pod \"e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") " pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:27.796122 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:27.796041    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:28.097904 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:28.097805    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"]
Sep 11 21:31:28.101340 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:31:28.101312    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d3677f_0ae8_4756_9ee2_dcc7b5875e9f.slice/crio-a059cbc32a8086f1048c97a16cfe3a62a8856b294be80837c813a6b98ee002d2 WatchSource:0}: Error finding container a059cbc32a8086f1048c97a16cfe3a62a8856b294be80837c813a6b98ee002d2: Status 404 returned error can't find the container with id a059cbc32a8086f1048c97a16cfe3a62a8856b294be80837c813a6b98ee002d2
Sep 11 21:31:28.103405 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:28.103374    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:31:28.103468 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:28.103437    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:31:28.103504 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:28.103469    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:31:28.844633 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:28.844599    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7" event={"ID":"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f","Type":"ContainerStarted","Data":"d7f77835cbb3f4145304da36556d11f669ed96fe3ee645450a073a3e403d60a3"}
Sep 11 21:31:28.845127 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:28.844645    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7" event={"ID":"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f","Type":"ContainerStarted","Data":"a059cbc32a8086f1048c97a16cfe3a62a8856b294be80837c813a6b98ee002d2"}
Sep 11 21:31:29.796737 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:29.796699    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:29.801623 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:29.801601    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:29.834052 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:29.833992    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7" podStartSLOduration=2.833978719 podStartE2EDuration="2.833978719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:31:28.882672357 +0000 UTC m=+1555.586641647" watchObservedRunningTime="2026-09-11 21:31:29.833978719 +0000 UTC m=+1556.537948008"
Sep 11 21:31:29.851497 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:29.851471    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:29.852249 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:29.852231    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:40.122476 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:40.122442    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:31:49.935751 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:49.935711    2578 generic.go:356] "Generic (PLEG): container finished" podID="b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f" containerID="d7f77835cbb3f4145304da36556d11f669ed96fe3ee645450a073a3e403d60a3" exitCode=0
Sep 11 21:31:49.936348 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:49.935767    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7" event={"ID":"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f","Type":"ContainerDied","Data":"d7f77835cbb3f4145304da36556d11f669ed96fe3ee645450a073a3e403d60a3"}
Sep 11 21:31:49.936348 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:49.936085    2578 scope.go:117] "RemoveContainer" containerID="d7f77835cbb3f4145304da36556d11f669ed96fe3ee645450a073a3e403d60a3"
Sep 11 21:31:49.937124 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:49.937096    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:31:49.937191 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:49.937168    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:31:49.937226 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:49.937203    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 21:31:50.946008 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:50.945950    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7" event={"ID":"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f","Type":"ContainerStarted","Data":"ee95aa3052240b5a6a8c178863653c2992767302872767a0e791462c2b93ffeb"}
Sep 11 21:31:51.796474 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:51.796437    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:51.796679 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:51.796657    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Sep 11 21:31:51.796725 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:51.796707    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7" podUID="b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Sep 11 21:31:52.797068 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:52.797033    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Sep 11 21:31:52.797439 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:52.797087    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7" podUID="b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Sep 11 21:31:53.797139 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:53.797102    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Sep 11 21:31:53.797533 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:53.797157    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7" podUID="b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Sep 11 21:31:54.797331 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:54.797295    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Sep 11 21:31:54.797705 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:54.797349    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7" podUID="b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Sep 11 21:31:55.800778 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:55.800746    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:55.801156 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:55.801037    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:31:55.801624 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:31:55.801608    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:32:15.024927 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.024892    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"]
Sep 11 21:32:15.028016 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.027998    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:32:15.030948 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.030926    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-19ce3e\"/\"openshift-service-ca.crt\""
Sep 11 21:32:15.031079 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.030964    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-19ce3e\"/\"ipp-route-19ce3e-kserve-self-signed-certs\""
Sep 11 21:32:15.031079 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.030930    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-19ce3e\"/\"default-dockercfg-8bbsc\""
Sep 11 21:32:15.031079 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.030977    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-19ce3e\"/\"kube-root-ca.crt\""
Sep 11 21:32:15.037647 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.037623    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"]
Sep 11 21:32:15.048587 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.048568    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-tls-certs\") pod \"ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8\" (UID: \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\") " pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:32:15.048669 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.048606    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbjmc\" (UniqueName: \"kubernetes.io/projected/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-kube-api-access-lbjmc\") pod \"ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8\" (UID: \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\") " pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:32:15.048669 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.048625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-home\") pod \"ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8\" (UID: \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\") " pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:32:15.048747 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.048705    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-model-cache\") pod \"ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8\" (UID: \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\") " pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:32:15.048747 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.048720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-tmp-dir\") pod \"ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8\" (UID: \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\") " pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:32:15.048812 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.048737    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-dshm\") pod \"ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8\" (UID: \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\") " pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:32:15.149846 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.149798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbjmc\" (UniqueName: \"kubernetes.io/projected/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-kube-api-access-lbjmc\") pod \"ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8\" (UID: \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\") " pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:32:15.149846 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.149850    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-home\") pod \"ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8\" (UID: \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\") " pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:32:15.150034 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.149906    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-model-cache\") pod \"ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8\" (UID: \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\") " pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:32:15.150034 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.149921    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-tmp-dir\") pod \"ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8\" (UID: \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\") " pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:32:15.150101 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.150033    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-dshm\") pod \"ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8\" (UID: \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\") " pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:32:15.150101 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.150078    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-tls-certs\") pod \"ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8\" (UID: \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\") " pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:32:15.150233 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.150216    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-home\") pod \"ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8\" (UID: \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\") " pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:32:15.150294 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.150275    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-model-cache\") pod \"ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8\" (UID: \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\") " pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:32:15.150349 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.150335    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-tmp-dir\") pod \"ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8\" (UID: \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\") " pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:32:15.152235 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.152207    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-dshm\") pod \"ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8\" (UID: \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\") " pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:32:15.152389 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.152375    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-tls-certs\") pod \"ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8\" (UID: \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\") " pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:32:15.159527 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.159505    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbjmc\" (UniqueName: \"kubernetes.io/projected/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-kube-api-access-lbjmc\") pod \"ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8\" (UID: \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\") " pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:32:15.337460 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.337386    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:32:15.610609 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:15.610542    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"]
Sep 11 21:32:15.613688 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:32:15.613659    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaeb2e54d_81f9_4b21_9e79_5a7ff2732040.slice/crio-e43e6c3bd18a1244ef0a1b3450a69618718ff05fae8b183bff1554a4702dbbd3 WatchSource:0}: Error finding container e43e6c3bd18a1244ef0a1b3450a69618718ff05fae8b183bff1554a4702dbbd3: Status 404 returned error can't find the container with id e43e6c3bd18a1244ef0a1b3450a69618718ff05fae8b183bff1554a4702dbbd3
Sep 11 21:32:16.066494 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:16.066461    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8" event={"ID":"aeb2e54d-81f9-4b21-9e79-5a7ff2732040","Type":"ContainerStarted","Data":"a8e9f30cca3632c0c143b9b2713b4fcf250adcd501d223a87d324c2e5c8f8ed6"}
Sep 11 21:32:16.066494 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:16.066499    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8" event={"ID":"aeb2e54d-81f9-4b21-9e79-5a7ff2732040","Type":"ContainerStarted","Data":"e43e6c3bd18a1244ef0a1b3450a69618718ff05fae8b183bff1554a4702dbbd3"}
Sep 11 21:32:16.083736 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:16.083694    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8" podStartSLOduration=1.08367904 podStartE2EDuration="1.08367904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:16.08326694 +0000 UTC m=+1602.787236229" watchObservedRunningTime="2026-09-11 21:32:16.08367904 +0000 UTC m=+1602.787648329"
Sep 11 21:32:19.819095 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:19.819061    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:32:25.337592 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:25.337564    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:32:25.338025 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:25.337607    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:32:25.349686 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:25.349665    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:32:26.121314 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:26.121235    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:32:31.415943 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:31.415910    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:32:44.578354 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:44.578321    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj"]
Sep 11 21:32:44.581544 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:44.581528    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj"
Sep 11 21:32:44.584100 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:44.584076    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-8a57ce\""
Sep 11 21:32:44.584200 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:44.584178    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-8a57ce-dockercfg-mj95v\""
Sep 11 21:32:44.597895 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:44.597874    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj"]
Sep 11 21:32:44.623563 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:44.623543    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q"]
Sep 11 21:32:44.626695 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:44.626677    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q"
Sep 11 21:32:44.646255 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:44.646237    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q"]
Sep 11 21:32:44.659557 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:44.659530    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b2720bc-3da9-4f16-8633-837da70d883d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj\" (UID: \"6b2720bc-3da9-4f16-8633-837da70d883d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj"
Sep 11 21:32:44.659669 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:44.659611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64tsm\" (UniqueName: \"kubernetes.io/projected/6b2720bc-3da9-4f16-8633-837da70d883d-kube-api-access-64tsm\") pod \"payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj\" (UID: \"6b2720bc-3da9-4f16-8633-837da70d883d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj"
Sep 11 21:32:44.760134 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:44.760102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2336f0a2-8525-4875-b3cd-e21df7d0da4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q\" (UID: \"2336f0a2-8525-4875-b3cd-e21df7d0da4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q"
Sep 11 21:32:44.760311 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:44.760146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64tsm\" (UniqueName: \"kubernetes.io/projected/6b2720bc-3da9-4f16-8633-837da70d883d-kube-api-access-64tsm\") pod \"payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj\" (UID: \"6b2720bc-3da9-4f16-8633-837da70d883d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj"
Sep 11 21:32:44.760311 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:44.760232    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq2kl\" (UniqueName: \"kubernetes.io/projected/2336f0a2-8525-4875-b3cd-e21df7d0da4f-kube-api-access-hq2kl\") pod \"payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q\" (UID: \"2336f0a2-8525-4875-b3cd-e21df7d0da4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q"
Sep 11 21:32:44.760311 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:44.760290    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b2720bc-3da9-4f16-8633-837da70d883d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj\" (UID: \"6b2720bc-3da9-4f16-8633-837da70d883d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj"
Sep 11 21:32:44.760934 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:44.760918    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b2720bc-3da9-4f16-8633-837da70d883d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj\" (UID: \"6b2720bc-3da9-4f16-8633-837da70d883d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj"
Sep 11 21:32:44.768549 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:44.768532    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64tsm\" (UniqueName: \"kubernetes.io/projected/6b2720bc-3da9-4f16-8633-837da70d883d-kube-api-access-64tsm\") pod \"payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj\" (UID: \"6b2720bc-3da9-4f16-8633-837da70d883d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj"
Sep 11 21:32:44.861422 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:44.861364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq2kl\" (UniqueName: \"kubernetes.io/projected/2336f0a2-8525-4875-b3cd-e21df7d0da4f-kube-api-access-hq2kl\") pod \"payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q\" (UID: \"2336f0a2-8525-4875-b3cd-e21df7d0da4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q"
Sep 11 21:32:44.861422 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:44.861408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2336f0a2-8525-4875-b3cd-e21df7d0da4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q\" (UID: \"2336f0a2-8525-4875-b3cd-e21df7d0da4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q"
Sep 11 21:32:44.862021 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:44.862007    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2336f0a2-8525-4875-b3cd-e21df7d0da4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q\" (UID: \"2336f0a2-8525-4875-b3cd-e21df7d0da4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q"
Sep 11 21:32:44.868878 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:44.868853    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq2kl\" (UniqueName: \"kubernetes.io/projected/2336f0a2-8525-4875-b3cd-e21df7d0da4f-kube-api-access-hq2kl\") pod \"payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q\" (UID: \"2336f0a2-8525-4875-b3cd-e21df7d0da4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q"
Sep 11 21:32:44.891696 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:44.891673    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj"
Sep 11 21:32:44.935731 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:44.935698    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q"
Sep 11 21:32:45.174793 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:45.174764    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj"]
Sep 11 21:32:45.178911 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:32:45.178886    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2720bc_3da9_4f16_8633_837da70d883d.slice/crio-08bdddd679c4f22b1fd549ab621ed81efc2385fba1bd70c463415180b543068d WatchSource:0}: Error finding container 08bdddd679c4f22b1fd549ab621ed81efc2385fba1bd70c463415180b543068d: Status 404 returned error can't find the container with id 08bdddd679c4f22b1fd549ab621ed81efc2385fba1bd70c463415180b543068d
Sep 11 21:32:45.189979 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:45.189955    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj" event={"ID":"6b2720bc-3da9-4f16-8633-837da70d883d","Type":"ContainerStarted","Data":"08bdddd679c4f22b1fd549ab621ed81efc2385fba1bd70c463415180b543068d"}
Sep 11 21:32:45.223348 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:45.223329    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q"]
Sep 11 21:32:45.225298 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:32:45.225271    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2336f0a2_8525_4875_b3cd_e21df7d0da4f.slice/crio-e93f730710136e670a48aaa51977c0e980303558c6780b0be6cb72f1dbc3572f WatchSource:0}: Error finding container e93f730710136e670a48aaa51977c0e980303558c6780b0be6cb72f1dbc3572f: Status 404 returned error can't find the container with id e93f730710136e670a48aaa51977c0e980303558c6780b0be6cb72f1dbc3572f
Sep 11 21:32:46.195779 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:46.195741    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj" event={"ID":"6b2720bc-3da9-4f16-8633-837da70d883d","Type":"ContainerStarted","Data":"d1a215c64d517050d2c69e8e937d1e4c2473c592d9b5dc1ec7e859c33fb250c9"}
Sep 11 21:32:46.196377 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:46.195906    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj"
Sep 11 21:32:46.197365 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:46.197337    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q" event={"ID":"2336f0a2-8525-4875-b3cd-e21df7d0da4f","Type":"ContainerStarted","Data":"9cd95ebf3e9915bd7c3ac8719d233e452d826dfe3f1fac41b9ff7a8fdcbf7f65"}
Sep 11 21:32:46.197489 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:46.197389    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q"
Sep 11 21:32:46.197489 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:46.197411    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q" event={"ID":"2336f0a2-8525-4875-b3cd-e21df7d0da4f","Type":"ContainerStarted","Data":"e93f730710136e670a48aaa51977c0e980303558c6780b0be6cb72f1dbc3572f"}
Sep 11 21:32:46.215409 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:46.215352    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj" podStartSLOduration=2.215337731 podStartE2EDuration="2.215337731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:46.213783313 +0000 UTC m=+1632.917752595" watchObservedRunningTime="2026-09-11 21:32:46.215337731 +0000 UTC m=+1632.919307020"
Sep 11 21:32:46.233079 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:46.233013    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q" podStartSLOduration=2.232996494 podStartE2EDuration="2.232996494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:46.231410944 +0000 UTC m=+1632.935380233" watchObservedRunningTime="2026-09-11 21:32:46.232996494 +0000 UTC m=+1632.936965784"
Sep 11 21:32:49.626947 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:49.626909    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:32:57.203945 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:57.203908    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj"
Sep 11 21:32:57.204324 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:32:57.203985    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q"
Sep 11 21:33:22.938001 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:22.937967    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj"]
Sep 11 21:33:22.938537 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:22.938225    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj" podUID="6b2720bc-3da9-4f16-8633-837da70d883d" containerName="payload-processing" containerID="cri-o://d1a215c64d517050d2c69e8e937d1e4c2473c592d9b5dc1ec7e859c33fb250c9" gracePeriod=30
Sep 11 21:33:22.955456 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:22.955400    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q"]
Sep 11 21:33:22.955673 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:22.955632    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q" podUID="2336f0a2-8525-4875-b3cd-e21df7d0da4f" containerName="payload-pre-processing" containerID="cri-o://9cd95ebf3e9915bd7c3ac8719d233e452d826dfe3f1fac41b9ff7a8fdcbf7f65" gracePeriod=30
Sep 11 21:33:23.216380 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.216357    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj"
Sep 11 21:33:23.218962 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.218945    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q"
Sep 11 21:33:23.258913 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.258878    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hq2kl\" (UniqueName: \"kubernetes.io/projected/2336f0a2-8525-4875-b3cd-e21df7d0da4f-kube-api-access-hq2kl\") pod \"2336f0a2-8525-4875-b3cd-e21df7d0da4f\" (UID: \"2336f0a2-8525-4875-b3cd-e21df7d0da4f\") "
Sep 11 21:33:23.259103 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.258938    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2336f0a2-8525-4875-b3cd-e21df7d0da4f-plugins-config-volume\") pod \"2336f0a2-8525-4875-b3cd-e21df7d0da4f\" (UID: \"2336f0a2-8525-4875-b3cd-e21df7d0da4f\") "
Sep 11 21:33:23.259103 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.258976    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b2720bc-3da9-4f16-8633-837da70d883d-plugins-config-volume\") pod \"6b2720bc-3da9-4f16-8633-837da70d883d\" (UID: \"6b2720bc-3da9-4f16-8633-837da70d883d\") "
Sep 11 21:33:23.259103 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.259057    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64tsm\" (UniqueName: \"kubernetes.io/projected/6b2720bc-3da9-4f16-8633-837da70d883d-kube-api-access-64tsm\") pod \"6b2720bc-3da9-4f16-8633-837da70d883d\" (UID: \"6b2720bc-3da9-4f16-8633-837da70d883d\") "
Sep 11 21:33:23.259719 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.259671    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2720bc-3da9-4f16-8633-837da70d883d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b2720bc-3da9-4f16-8633-837da70d883d" (UID: "6b2720bc-3da9-4f16-8633-837da70d883d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:33:23.259719 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.259670    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2336f0a2-8525-4875-b3cd-e21df7d0da4f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2336f0a2-8525-4875-b3cd-e21df7d0da4f" (UID: "2336f0a2-8525-4875-b3cd-e21df7d0da4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:33:23.261921 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.261892    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2336f0a2-8525-4875-b3cd-e21df7d0da4f-kube-api-access-hq2kl" (OuterVolumeSpecName: "kube-api-access-hq2kl") pod "2336f0a2-8525-4875-b3cd-e21df7d0da4f" (UID: "2336f0a2-8525-4875-b3cd-e21df7d0da4f"). InnerVolumeSpecName "kube-api-access-hq2kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:33:23.262094 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.262075    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2720bc-3da9-4f16-8633-837da70d883d-kube-api-access-64tsm" (OuterVolumeSpecName: "kube-api-access-64tsm") pod "6b2720bc-3da9-4f16-8633-837da70d883d" (UID: "6b2720bc-3da9-4f16-8633-837da70d883d"). InnerVolumeSpecName "kube-api-access-64tsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:33:23.357887 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.357851    2578 generic.go:356] "Generic (PLEG): container finished" podID="2336f0a2-8525-4875-b3cd-e21df7d0da4f" containerID="9cd95ebf3e9915bd7c3ac8719d233e452d826dfe3f1fac41b9ff7a8fdcbf7f65" exitCode=0
Sep 11 21:33:23.358064 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.357890    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q"
Sep 11 21:33:23.358064 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.357911    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q" event={"ID":"2336f0a2-8525-4875-b3cd-e21df7d0da4f","Type":"ContainerDied","Data":"9cd95ebf3e9915bd7c3ac8719d233e452d826dfe3f1fac41b9ff7a8fdcbf7f65"}
Sep 11 21:33:23.358064 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.357970    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q" event={"ID":"2336f0a2-8525-4875-b3cd-e21df7d0da4f","Type":"ContainerDied","Data":"e93f730710136e670a48aaa51977c0e980303558c6780b0be6cb72f1dbc3572f"}
Sep 11 21:33:23.358064 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.357997    2578 scope.go:117] "RemoveContainer" containerID="9cd95ebf3e9915bd7c3ac8719d233e452d826dfe3f1fac41b9ff7a8fdcbf7f65"
Sep 11 21:33:23.359315 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.359293    2578 generic.go:356] "Generic (PLEG): container finished" podID="6b2720bc-3da9-4f16-8633-837da70d883d" containerID="d1a215c64d517050d2c69e8e937d1e4c2473c592d9b5dc1ec7e859c33fb250c9" exitCode=0
Sep 11 21:33:23.359431 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.359327    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj" event={"ID":"6b2720bc-3da9-4f16-8633-837da70d883d","Type":"ContainerDied","Data":"d1a215c64d517050d2c69e8e937d1e4c2473c592d9b5dc1ec7e859c33fb250c9"}
Sep 11 21:33:23.359431 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.359336    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj"
Sep 11 21:33:23.359431 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.359365    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj" event={"ID":"6b2720bc-3da9-4f16-8633-837da70d883d","Type":"ContainerDied","Data":"08bdddd679c4f22b1fd549ab621ed81efc2385fba1bd70c463415180b543068d"}
Sep 11 21:33:23.359762 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.359743    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq2kl\" (UniqueName: \"kubernetes.io/projected/2336f0a2-8525-4875-b3cd-e21df7d0da4f-kube-api-access-hq2kl\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:33:23.359858 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.359767    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2336f0a2-8525-4875-b3cd-e21df7d0da4f-plugins-config-volume\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:33:23.359858 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.359780    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b2720bc-3da9-4f16-8633-837da70d883d-plugins-config-volume\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:33:23.359858 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.359794    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64tsm\" (UniqueName: \"kubernetes.io/projected/6b2720bc-3da9-4f16-8633-837da70d883d-kube-api-access-64tsm\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:33:23.367790 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.367776    2578 scope.go:117] "RemoveContainer" containerID="9cd95ebf3e9915bd7c3ac8719d233e452d826dfe3f1fac41b9ff7a8fdcbf7f65"
Sep 11 21:33:23.368080 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:33:23.368061    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cd95ebf3e9915bd7c3ac8719d233e452d826dfe3f1fac41b9ff7a8fdcbf7f65\": container with ID starting with 9cd95ebf3e9915bd7c3ac8719d233e452d826dfe3f1fac41b9ff7a8fdcbf7f65 not found: ID does not exist" containerID="9cd95ebf3e9915bd7c3ac8719d233e452d826dfe3f1fac41b9ff7a8fdcbf7f65"
Sep 11 21:33:23.368148 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.368091    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cd95ebf3e9915bd7c3ac8719d233e452d826dfe3f1fac41b9ff7a8fdcbf7f65"} err="failed to get container status \"9cd95ebf3e9915bd7c3ac8719d233e452d826dfe3f1fac41b9ff7a8fdcbf7f65\": rpc error: code = NotFound desc = could not find container \"9cd95ebf3e9915bd7c3ac8719d233e452d826dfe3f1fac41b9ff7a8fdcbf7f65\": container with ID starting with 9cd95ebf3e9915bd7c3ac8719d233e452d826dfe3f1fac41b9ff7a8fdcbf7f65 not found: ID does not exist"
Sep 11 21:33:23.368148 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.368113    2578 scope.go:117] "RemoveContainer" containerID="d1a215c64d517050d2c69e8e937d1e4c2473c592d9b5dc1ec7e859c33fb250c9"
Sep 11 21:33:23.375156 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.375138    2578 scope.go:117] "RemoveContainer" containerID="d1a215c64d517050d2c69e8e937d1e4c2473c592d9b5dc1ec7e859c33fb250c9"
Sep 11 21:33:23.375394 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:33:23.375378    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a215c64d517050d2c69e8e937d1e4c2473c592d9b5dc1ec7e859c33fb250c9\": container with ID starting with d1a215c64d517050d2c69e8e937d1e4c2473c592d9b5dc1ec7e859c33fb250c9 not found: ID does not exist" containerID="d1a215c64d517050d2c69e8e937d1e4c2473c592d9b5dc1ec7e859c33fb250c9"
Sep 11 21:33:23.375441 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.375400    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a215c64d517050d2c69e8e937d1e4c2473c592d9b5dc1ec7e859c33fb250c9"} err="failed to get container status \"d1a215c64d517050d2c69e8e937d1e4c2473c592d9b5dc1ec7e859c33fb250c9\": rpc error: code = NotFound desc = could not find container \"d1a215c64d517050d2c69e8e937d1e4c2473c592d9b5dc1ec7e859c33fb250c9\": container with ID starting with d1a215c64d517050d2c69e8e937d1e4c2473c592d9b5dc1ec7e859c33fb250c9 not found: ID does not exist"
Sep 11 21:33:23.379638 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.379620    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q"]
Sep 11 21:33:23.383755 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.383736    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a57ce-689594dfd-l8z9q"]
Sep 11 21:33:23.394812 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.394790    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj"]
Sep 11 21:33:23.403385 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.399348    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8a57ce-5ccc6c59d5-5cvtj"]
Sep 11 21:33:23.927489 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.927457    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2336f0a2-8525-4875-b3cd-e21df7d0da4f" path="/var/lib/kubelet/pods/2336f0a2-8525-4875-b3cd-e21df7d0da4f/volumes"
Sep 11 21:33:23.927790 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:23.927777    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b2720bc-3da9-4f16-8633-837da70d883d" path="/var/lib/kubelet/pods/6b2720bc-3da9-4f16-8633-837da70d883d/volumes"
Sep 11 21:33:28.223696 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:28.223666    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:33:33.276975 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:33.276937    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ce3e-q8cqk"]
Sep 11 21:33:33.277435 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:33.277371    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b2720bc-3da9-4f16-8633-837da70d883d" containerName="payload-processing"
Sep 11 21:33:33.277435 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:33.277388    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b2720bc-3da9-4f16-8633-837da70d883d" containerName="payload-processing"
Sep 11 21:33:33.277435 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:33.277421    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2336f0a2-8525-4875-b3cd-e21df7d0da4f" containerName="payload-pre-processing"
Sep 11 21:33:33.277435 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:33.277428    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2336f0a2-8525-4875-b3cd-e21df7d0da4f" containerName="payload-pre-processing"
Sep 11 21:33:33.277648 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:33.277502    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2336f0a2-8525-4875-b3cd-e21df7d0da4f" containerName="payload-pre-processing"
Sep 11 21:33:33.277648 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:33.277514    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b2720bc-3da9-4f16-8633-837da70d883d" containerName="payload-processing"
Sep 11 21:33:33.281965 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:33.281944    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ce3e-q8cqk"
Sep 11 21:33:33.284013 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:33.283993    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:33:33.284108 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:33.284092    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfm9g\""
Sep 11 21:33:33.286917 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:33.286890    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ce3e-q8cqk"]
Sep 11 21:33:33.330577 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:33.330555    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f8627a-d485-4b63-8886-04ce148f43a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-19ce3e-q8cqk\" (UID: \"83f8627a-d485-4b63-8886-04ce148f43a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ce3e-q8cqk"
Sep 11 21:33:33.431305 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:33.431278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f8627a-d485-4b63-8886-04ce148f43a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-19ce3e-q8cqk\" (UID: \"83f8627a-d485-4b63-8886-04ce148f43a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ce3e-q8cqk"
Sep 11 21:33:33.431839 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:33.431801    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f8627a-d485-4b63-8886-04ce148f43a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-19ce3e-q8cqk\" (UID: \"83f8627a-d485-4b63-8886-04ce148f43a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ce3e-q8cqk"
Sep 11 21:33:33.592879 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:33.592789    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ce3e-q8cqk"
Sep 11 21:33:33.860894 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:33.860810    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ce3e-q8cqk"]
Sep 11 21:33:33.863999 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:33:33.863974    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f8627a_d485_4b63_8886_04ce148f43a3.slice/crio-323a76c09ec7837ef3f17d6911cdd86d7e029242476e419d27ea973d6c57f5cf WatchSource:0}: Error finding container 323a76c09ec7837ef3f17d6911cdd86d7e029242476e419d27ea973d6c57f5cf: Status 404 returned error can't find the container with id 323a76c09ec7837ef3f17d6911cdd86d7e029242476e419d27ea973d6c57f5cf
Sep 11 21:33:34.414271 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:34.414188    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ce3e-q8cqk" event={"ID":"83f8627a-d485-4b63-8886-04ce148f43a3","Type":"ContainerStarted","Data":"0bf4b534d0553256b7c52218ab7ad0587ee0d24c1e7f92e590531c8b452ee505"}
Sep 11 21:33:34.414271 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:34.414266    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ce3e-q8cqk" event={"ID":"83f8627a-d485-4b63-8886-04ce148f43a3","Type":"ContainerStarted","Data":"323a76c09ec7837ef3f17d6911cdd86d7e029242476e419d27ea973d6c57f5cf"}
Sep 11 21:33:34.431878 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:34.431814    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ce3e-q8cqk" podStartSLOduration=1.431800467 podStartE2EDuration="1.431800467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:33:34.429923158 +0000 UTC m=+1681.133892458" watchObservedRunningTime="2026-09-11 21:33:34.431800467 +0000 UTC m=+1681.135769755"
Sep 11 21:33:35.419728 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:35.419695    2578 generic.go:356] "Generic (PLEG): container finished" podID="83f8627a-d485-4b63-8886-04ce148f43a3" containerID="0bf4b534d0553256b7c52218ab7ad0587ee0d24c1e7f92e590531c8b452ee505" exitCode=0
Sep 11 21:33:35.420104 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:35.419774    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ce3e-q8cqk" event={"ID":"83f8627a-d485-4b63-8886-04ce148f43a3","Type":"ContainerDied","Data":"0bf4b534d0553256b7c52218ab7ad0587ee0d24c1e7f92e590531c8b452ee505"}
Sep 11 21:33:36.544243 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:36.544221    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ce3e-q8cqk"
Sep 11 21:33:36.659204 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:36.659171    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f8627a-d485-4b63-8886-04ce148f43a3-maas-api-service-ca\") pod \"83f8627a-d485-4b63-8886-04ce148f43a3\" (UID: \"83f8627a-d485-4b63-8886-04ce148f43a3\") "
Sep 11 21:33:36.659513 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:36.659492    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f8627a-d485-4b63-8886-04ce148f43a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83f8627a-d485-4b63-8886-04ce148f43a3" (UID: "83f8627a-d485-4b63-8886-04ce148f43a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:33:36.760606 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:36.760538    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f8627a-d485-4b63-8886-04ce148f43a3-maas-api-service-ca\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:33:37.429120 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:37.429083    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ce3e-q8cqk" event={"ID":"83f8627a-d485-4b63-8886-04ce148f43a3","Type":"ContainerDied","Data":"323a76c09ec7837ef3f17d6911cdd86d7e029242476e419d27ea973d6c57f5cf"}
Sep 11 21:33:37.429120 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:37.429106    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ce3e-q8cqk"
Sep 11 21:33:37.429334 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:37.429143    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="323a76c09ec7837ef3f17d6911cdd86d7e029242476e419d27ea973d6c57f5cf"
Sep 11 21:33:46.817865 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:46.817832    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:33:49.204299 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:49.204269    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ce3e-q8cqk"]
Sep 11 21:33:49.209007 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:49.208984    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19ce3e-q8cqk"]
Sep 11 21:33:49.927268 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:49.927233    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83f8627a-d485-4b63-8886-04ce148f43a3" path="/var/lib/kubelet/pods/83f8627a-d485-4b63-8886-04ce148f43a3/volumes"
Sep 11 21:33:54.559198 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:54.559168    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"]
Sep 11 21:33:59.721797 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:33:59.721763    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:34:09.577323 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:09.577267    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8" podUID="aeb2e54d-81f9-4b21-9e79-5a7ff2732040" containerName="main" containerID="cri-o://a8e9f30cca3632c0c143b9b2713b4fcf250adcd501d223a87d324c2e5c8f8ed6" gracePeriod=45
Sep 11 21:34:09.827257 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:09.827233    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:34:09.922307 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:09.922215    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-tls-certs\") pod \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\" (UID: \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\") "
Sep 11 21:34:09.922307 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:09.922250    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-home\") pod \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\" (UID: \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\") "
Sep 11 21:34:09.922307 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:09.922278    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-tmp-dir\") pod \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\" (UID: \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\") "
Sep 11 21:34:09.922596 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:09.922313    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-model-cache\") pod \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\" (UID: \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\") "
Sep 11 21:34:09.922596 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:09.922389    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbjmc\" (UniqueName: \"kubernetes.io/projected/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-kube-api-access-lbjmc\") pod \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\" (UID: \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\") "
Sep 11 21:34:09.922596 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:09.922423    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-dshm\") pod \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\" (UID: \"aeb2e54d-81f9-4b21-9e79-5a7ff2732040\") "
Sep 11 21:34:09.922596 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:09.922486    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-home" (OuterVolumeSpecName: "home") pod "aeb2e54d-81f9-4b21-9e79-5a7ff2732040" (UID: "aeb2e54d-81f9-4b21-9e79-5a7ff2732040"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:34:09.922596 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:09.922550    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aeb2e54d-81f9-4b21-9e79-5a7ff2732040" (UID: "aeb2e54d-81f9-4b21-9e79-5a7ff2732040"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:34:09.922794 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:09.922699    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-home\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:34:09.922794 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:09.922698    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-model-cache" (OuterVolumeSpecName: "model-cache") pod "aeb2e54d-81f9-4b21-9e79-5a7ff2732040" (UID: "aeb2e54d-81f9-4b21-9e79-5a7ff2732040"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:34:09.922794 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:09.922718    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-tmp-dir\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:34:09.924456 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:09.924421    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aeb2e54d-81f9-4b21-9e79-5a7ff2732040" (UID: "aeb2e54d-81f9-4b21-9e79-5a7ff2732040"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:34:09.924630 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:09.924609    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-dshm" (OuterVolumeSpecName: "dshm") pod "aeb2e54d-81f9-4b21-9e79-5a7ff2732040" (UID: "aeb2e54d-81f9-4b21-9e79-5a7ff2732040"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:34:09.924697 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:09.924621    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-kube-api-access-lbjmc" (OuterVolumeSpecName: "kube-api-access-lbjmc") pod "aeb2e54d-81f9-4b21-9e79-5a7ff2732040" (UID: "aeb2e54d-81f9-4b21-9e79-5a7ff2732040"). InnerVolumeSpecName "kube-api-access-lbjmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:34:10.024311 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:10.024276    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-tls-certs\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:34:10.024311 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:10.024311    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-model-cache\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:34:10.024553 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:10.024333    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbjmc\" (UniqueName: \"kubernetes.io/projected/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-kube-api-access-lbjmc\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:34:10.024553 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:10.024344    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeb2e54d-81f9-4b21-9e79-5a7ff2732040-dshm\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:34:10.576775 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:10.576742    2578 generic.go:356] "Generic (PLEG): container finished" podID="aeb2e54d-81f9-4b21-9e79-5a7ff2732040" containerID="a8e9f30cca3632c0c143b9b2713b4fcf250adcd501d223a87d324c2e5c8f8ed6" exitCode=0
Sep 11 21:34:10.576968 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:10.576784    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"
Sep 11 21:34:10.576968 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:10.576861    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8" event={"ID":"aeb2e54d-81f9-4b21-9e79-5a7ff2732040","Type":"ContainerDied","Data":"a8e9f30cca3632c0c143b9b2713b4fcf250adcd501d223a87d324c2e5c8f8ed6"}
Sep 11 21:34:10.576968 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:10.576900    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8" event={"ID":"aeb2e54d-81f9-4b21-9e79-5a7ff2732040","Type":"ContainerDied","Data":"e43e6c3bd18a1244ef0a1b3450a69618718ff05fae8b183bff1554a4702dbbd3"}
Sep 11 21:34:10.576968 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:10.576920    2578 scope.go:117] "RemoveContainer" containerID="a8e9f30cca3632c0c143b9b2713b4fcf250adcd501d223a87d324c2e5c8f8ed6"
Sep 11 21:34:10.584894 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:10.584875    2578 scope.go:117] "RemoveContainer" containerID="a8e9f30cca3632c0c143b9b2713b4fcf250adcd501d223a87d324c2e5c8f8ed6"
Sep 11 21:34:10.585170 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:34:10.585131    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e9f30cca3632c0c143b9b2713b4fcf250adcd501d223a87d324c2e5c8f8ed6\": container with ID starting with a8e9f30cca3632c0c143b9b2713b4fcf250adcd501d223a87d324c2e5c8f8ed6 not found: ID does not exist" containerID="a8e9f30cca3632c0c143b9b2713b4fcf250adcd501d223a87d324c2e5c8f8ed6"
Sep 11 21:34:10.585170 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:10.585162    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e9f30cca3632c0c143b9b2713b4fcf250adcd501d223a87d324c2e5c8f8ed6"} err="failed to get container status \"a8e9f30cca3632c0c143b9b2713b4fcf250adcd501d223a87d324c2e5c8f8ed6\": rpc error: code = NotFound desc = could not find container \"a8e9f30cca3632c0c143b9b2713b4fcf250adcd501d223a87d324c2e5c8f8ed6\": container with ID starting with a8e9f30cca3632c0c143b9b2713b4fcf250adcd501d223a87d324c2e5c8f8ed6 not found: ID does not exist"
Sep 11 21:34:10.592369 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:10.592349    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"]
Sep 11 21:34:10.596492 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:10.596471    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-19ce3e/ipp-route-19ce3e-kserve-6b8ddb9b8c-wgwp8"]
Sep 11 21:34:11.926575 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:11.926537    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeb2e54d-81f9-4b21-9e79-5a7ff2732040" path="/var/lib/kubelet/pods/aeb2e54d-81f9-4b21-9e79-5a7ff2732040/volumes"
Sep 11 21:34:38.373500 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:38.373462    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9c558-kq8q8"]
Sep 11 21:34:38.373928 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:38.373760    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeb2e54d-81f9-4b21-9e79-5a7ff2732040" containerName="main"
Sep 11 21:34:38.373928 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:38.373771    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeb2e54d-81f9-4b21-9e79-5a7ff2732040" containerName="main"
Sep 11 21:34:38.373928 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:38.373778    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83f8627a-d485-4b63-8886-04ce148f43a3" containerName="revoke-keys"
Sep 11 21:34:38.373928 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:38.373785    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="83f8627a-d485-4b63-8886-04ce148f43a3" containerName="revoke-keys"
Sep 11 21:34:38.373928 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:38.373859    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="83f8627a-d485-4b63-8886-04ce148f43a3" containerName="revoke-keys"
Sep 11 21:34:38.373928 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:38.373872    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeb2e54d-81f9-4b21-9e79-5a7ff2732040" containerName="main"
Sep 11 21:34:38.376707 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:38.376690    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9c558-kq8q8"
Sep 11 21:34:38.379478 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:38.379458    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:34:38.379786 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:38.379764    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfm9g\""
Sep 11 21:34:38.389287 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:38.389262    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9c558-kq8q8"]
Sep 11 21:34:38.448166 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:38.448131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd9b797-4677-4f8e-8c9a-4b98744f07ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e9c558-kq8q8\" (UID: \"ddd9b797-4677-4f8e-8c9a-4b98744f07ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9c558-kq8q8"
Sep 11 21:34:38.548888 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:38.548851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd9b797-4677-4f8e-8c9a-4b98744f07ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e9c558-kq8q8\" (UID: \"ddd9b797-4677-4f8e-8c9a-4b98744f07ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9c558-kq8q8"
Sep 11 21:34:38.549464 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:38.549437    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd9b797-4677-4f8e-8c9a-4b98744f07ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e9c558-kq8q8\" (UID: \"ddd9b797-4677-4f8e-8c9a-4b98744f07ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9c558-kq8q8"
Sep 11 21:34:38.686024 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:38.685990    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9c558-kq8q8"
Sep 11 21:34:38.956288 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:38.956214    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9c558-kq8q8"]
Sep 11 21:34:38.959275 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:34:38.959250    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd9b797_4677_4f8e_8c9a_4b98744f07ce.slice/crio-5049a61d17082660701a4ec3df058a6bf677ab96b3ac1bdafa65c3e57d0d5cf8 WatchSource:0}: Error finding container 5049a61d17082660701a4ec3df058a6bf677ab96b3ac1bdafa65c3e57d0d5cf8: Status 404 returned error can't find the container with id 5049a61d17082660701a4ec3df058a6bf677ab96b3ac1bdafa65c3e57d0d5cf8
Sep 11 21:34:39.712396 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:39.712359    2578 generic.go:356] "Generic (PLEG): container finished" podID="ddd9b797-4677-4f8e-8c9a-4b98744f07ce" containerID="c56688d33dde36008ad9da06165bd6594681b9ee05727d7e3a79ec7a7e7694ae" exitCode=0
Sep 11 21:34:39.712762 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:39.712443    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9c558-kq8q8" event={"ID":"ddd9b797-4677-4f8e-8c9a-4b98744f07ce","Type":"ContainerDied","Data":"c56688d33dde36008ad9da06165bd6594681b9ee05727d7e3a79ec7a7e7694ae"}
Sep 11 21:34:39.712762 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:39.712520    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9c558-kq8q8" event={"ID":"ddd9b797-4677-4f8e-8c9a-4b98744f07ce","Type":"ContainerStarted","Data":"5049a61d17082660701a4ec3df058a6bf677ab96b3ac1bdafa65c3e57d0d5cf8"}
Sep 11 21:34:40.839285 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:40.839259    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9c558-kq8q8"
Sep 11 21:34:40.865894 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:40.865869    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd9b797-4677-4f8e-8c9a-4b98744f07ce-maas-api-service-ca\") pod \"ddd9b797-4677-4f8e-8c9a-4b98744f07ce\" (UID: \"ddd9b797-4677-4f8e-8c9a-4b98744f07ce\") "
Sep 11 21:34:40.866197 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:40.866175    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd9b797-4677-4f8e-8c9a-4b98744f07ce-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ddd9b797-4677-4f8e-8c9a-4b98744f07ce" (UID: "ddd9b797-4677-4f8e-8c9a-4b98744f07ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:34:40.966498 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:40.966469    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd9b797-4677-4f8e-8c9a-4b98744f07ce-maas-api-service-ca\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:34:41.722166 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:41.722136    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9c558-kq8q8"
Sep 11 21:34:41.722342 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:41.722136    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9c558-kq8q8" event={"ID":"ddd9b797-4677-4f8e-8c9a-4b98744f07ce","Type":"ContainerDied","Data":"5049a61d17082660701a4ec3df058a6bf677ab96b3ac1bdafa65c3e57d0d5cf8"}
Sep 11 21:34:41.722342 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:41.722257    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5049a61d17082660701a4ec3df058a6bf677ab96b3ac1bdafa65c3e57d0d5cf8"
Sep 11 21:34:49.140276 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:49.140242    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9c558-kq8q8"]
Sep 11 21:34:49.145564 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:49.145541    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e9c558-kq8q8"]
Sep 11 21:34:49.927044 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:49.927012    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddd9b797-4677-4f8e-8c9a-4b98744f07ce" path="/var/lib/kubelet/pods/ddd9b797-4677-4f8e-8c9a-4b98744f07ce/volumes"
Sep 11 21:34:52.917763 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:52.917722    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:34:55.638065 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:55.638032    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"]
Sep 11 21:34:55.638520 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:55.638297    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7" podUID="b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f" containerName="istio-proxy" containerID="cri-o://ee95aa3052240b5a6a8c178863653c2992767302872767a0e791462c2b93ffeb" gracePeriod=30
Sep 11 21:34:55.803619 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:55.803588    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 21:34:55.803777 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:34:55.803650    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7" podUID="b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 21:35:00.806484 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:00.806453    2578 generic.go:356] "Generic (PLEG): container finished" podID="b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f" containerID="ee95aa3052240b5a6a8c178863653c2992767302872767a0e791462c2b93ffeb" exitCode=0
Sep 11 21:35:00.806794 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:00.806530    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7" event={"ID":"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f","Type":"ContainerDied","Data":"ee95aa3052240b5a6a8c178863653c2992767302872767a0e791462c2b93ffeb"}
Sep 11 21:35:00.806794 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:00.806607    2578 scope.go:117] "RemoveContainer" containerID="d7f77835cbb3f4145304da36556d11f669ed96fe3ee645450a073a3e403d60a3"
Sep 11 21:35:00.885874 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:00.885852    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:35:01.035951 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.035914    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jwlg\" (UniqueName: \"kubernetes.io/projected/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-kube-api-access-9jwlg\") pod \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") "
Sep 11 21:35:01.036105 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.035963    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istio-podinfo\") pod \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") "
Sep 11 21:35:01.036105 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.035982    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istiod-ca-cert\") pod \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") "
Sep 11 21:35:01.036105 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.035998    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istio-data\") pod \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") "
Sep 11 21:35:01.036105 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.036014    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-workload-socket\") pod \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") "
Sep 11 21:35:01.036105 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.036053    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-workload-certs\") pod \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") "
Sep 11 21:35:01.036105 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.036088    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istio-token\") pod \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") "
Sep 11 21:35:01.036414 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.036110    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-credential-socket\") pod \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") "
Sep 11 21:35:01.036414 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.036251    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istio-envoy\") pod \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\" (UID: \"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f\") "
Sep 11 21:35:01.036414 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.036304    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istio-data" (OuterVolumeSpecName: "istio-data") pod "b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f" (UID: "b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:01.036414 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.036324    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f" (UID: "b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:01.036414 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.036335    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f" (UID: "b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:01.036628 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.036426    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f" (UID: "b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:35:01.036628 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.036442    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f" (UID: "b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:01.036628 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.036581    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istiod-ca-cert\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:01.036628 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.036598    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istio-data\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:01.036628 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.036610    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-workload-socket\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:01.036628 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.036621    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-workload-certs\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:01.036628 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.036630    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-credential-socket\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:01.038503 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.038476    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f" (UID: "b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:35:01.038503 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.038482    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istio-token" (OuterVolumeSpecName: "istio-token") pod "b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f" (UID: "b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:35:01.038653 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.038480    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-kube-api-access-9jwlg" (OuterVolumeSpecName: "kube-api-access-9jwlg") pod "b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f" (UID: "b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f"). InnerVolumeSpecName "kube-api-access-9jwlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:35:01.038653 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.038506    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f" (UID: "b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:01.137776 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.137727    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jwlg\" (UniqueName: \"kubernetes.io/projected/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-kube-api-access-9jwlg\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:01.137776 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.137770    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istio-podinfo\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:01.137776 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.137779    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istio-token\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:01.137776 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.137787    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f-istio-envoy\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:01.812422 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.812392    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"
Sep 11 21:35:01.812927 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.812442    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7" event={"ID":"b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f","Type":"ContainerDied","Data":"a059cbc32a8086f1048c97a16cfe3a62a8856b294be80837c813a6b98ee002d2"}
Sep 11 21:35:01.812927 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.812501    2578 scope.go:117] "RemoveContainer" containerID="ee95aa3052240b5a6a8c178863653c2992767302872767a0e791462c2b93ffeb"
Sep 11 21:35:01.834109 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.834084    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"]
Sep 11 21:35:01.837504 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.837483    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e9c558-openshift-default-5548bf6f78-9mqd7"]
Sep 11 21:35:01.927446 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:01.927404    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f" path="/var/lib/kubelet/pods/b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f/volumes"
Sep 11 21:35:06.631473 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:06.631436    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd"]
Sep 11 21:35:06.632086 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:06.631709    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd" podUID="43b41491-60a0-4dbf-8aa7-32e9344f0497" containerName="payload-processing" containerID="cri-o://2069b3c4f78148d5c69eb4b4253c3d5a5df9271441784801db30cf745cb98409" gracePeriod=30
Sep 11 21:35:06.636602 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:06.636573    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb"]
Sep 11 21:35:06.636869 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:06.636797    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb" podUID="b9d7fc44-e0c4-4933-8079-f0cd21d243a3" containerName="payload-pre-processing" containerID="cri-o://e2650615b7bd185105f757b5ddacde347cca55c20aa0250245480e76f4f4966a" gracePeriod=30
Sep 11 21:35:06.837680 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:06.837648    2578 generic.go:356] "Generic (PLEG): container finished" podID="43b41491-60a0-4dbf-8aa7-32e9344f0497" containerID="2069b3c4f78148d5c69eb4b4253c3d5a5df9271441784801db30cf745cb98409" exitCode=0
Sep 11 21:35:06.837805 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:06.837711    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd" event={"ID":"43b41491-60a0-4dbf-8aa7-32e9344f0497","Type":"ContainerDied","Data":"2069b3c4f78148d5c69eb4b4253c3d5a5df9271441784801db30cf745cb98409"}
Sep 11 21:35:06.840236 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:06.840202    2578 generic.go:356] "Generic (PLEG): container finished" podID="b9d7fc44-e0c4-4933-8079-f0cd21d243a3" containerID="e2650615b7bd185105f757b5ddacde347cca55c20aa0250245480e76f4f4966a" exitCode=0
Sep 11 21:35:06.840236 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:06.840235    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb" event={"ID":"b9d7fc44-e0c4-4933-8079-f0cd21d243a3","Type":"ContainerDied","Data":"e2650615b7bd185105f757b5ddacde347cca55c20aa0250245480e76f4f4966a"}
Sep 11 21:35:06.898061 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:06.898033    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd"
Sep 11 21:35:06.900864 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:06.900845    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb"
Sep 11 21:35:06.981390 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:06.981363    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdj7b\" (UniqueName: \"kubernetes.io/projected/43b41491-60a0-4dbf-8aa7-32e9344f0497-kube-api-access-qdj7b\") pod \"43b41491-60a0-4dbf-8aa7-32e9344f0497\" (UID: \"43b41491-60a0-4dbf-8aa7-32e9344f0497\") "
Sep 11 21:35:06.981552 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:06.981422    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9d7fc44-e0c4-4933-8079-f0cd21d243a3-plugins-config-volume\") pod \"b9d7fc44-e0c4-4933-8079-f0cd21d243a3\" (UID: \"b9d7fc44-e0c4-4933-8079-f0cd21d243a3\") "
Sep 11 21:35:06.981552 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:06.981458    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hr5l7\" (UniqueName: \"kubernetes.io/projected/b9d7fc44-e0c4-4933-8079-f0cd21d243a3-kube-api-access-hr5l7\") pod \"b9d7fc44-e0c4-4933-8079-f0cd21d243a3\" (UID: \"b9d7fc44-e0c4-4933-8079-f0cd21d243a3\") "
Sep 11 21:35:06.981552 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:06.981526    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b41491-60a0-4dbf-8aa7-32e9344f0497-plugins-config-volume\") pod \"43b41491-60a0-4dbf-8aa7-32e9344f0497\" (UID: \"43b41491-60a0-4dbf-8aa7-32e9344f0497\") "
Sep 11 21:35:06.981845 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:06.981804    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d7fc44-e0c4-4933-8079-f0cd21d243a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b9d7fc44-e0c4-4933-8079-f0cd21d243a3" (UID: "b9d7fc44-e0c4-4933-8079-f0cd21d243a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:35:06.981924 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:06.981882    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b41491-60a0-4dbf-8aa7-32e9344f0497-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "43b41491-60a0-4dbf-8aa7-32e9344f0497" (UID: "43b41491-60a0-4dbf-8aa7-32e9344f0497"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:35:06.984117 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:06.984088    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b41491-60a0-4dbf-8aa7-32e9344f0497-kube-api-access-qdj7b" (OuterVolumeSpecName: "kube-api-access-qdj7b") pod "43b41491-60a0-4dbf-8aa7-32e9344f0497" (UID: "43b41491-60a0-4dbf-8aa7-32e9344f0497"). InnerVolumeSpecName "kube-api-access-qdj7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:35:06.984227 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:06.984145    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d7fc44-e0c4-4933-8079-f0cd21d243a3-kube-api-access-hr5l7" (OuterVolumeSpecName: "kube-api-access-hr5l7") pod "b9d7fc44-e0c4-4933-8079-f0cd21d243a3" (UID: "b9d7fc44-e0c4-4933-8079-f0cd21d243a3"). InnerVolumeSpecName "kube-api-access-hr5l7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:35:07.082580 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:07.082555    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdj7b\" (UniqueName: \"kubernetes.io/projected/43b41491-60a0-4dbf-8aa7-32e9344f0497-kube-api-access-qdj7b\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:07.082580 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:07.082574    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9d7fc44-e0c4-4933-8079-f0cd21d243a3-plugins-config-volume\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:07.082580 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:07.082584    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr5l7\" (UniqueName: \"kubernetes.io/projected/b9d7fc44-e0c4-4933-8079-f0cd21d243a3-kube-api-access-hr5l7\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:07.082580 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:07.082593    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b41491-60a0-4dbf-8aa7-32e9344f0497-plugins-config-volume\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:07.845786 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:07.845750    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd"
Sep 11 21:35:07.845786 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:07.845780    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd" event={"ID":"43b41491-60a0-4dbf-8aa7-32e9344f0497","Type":"ContainerDied","Data":"d6bfc7807894b28f8e3c2aa369d2f32ceeb3ebf21348c62afb1487d0f01e3337"}
Sep 11 21:35:07.846276 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:07.845853    2578 scope.go:117] "RemoveContainer" containerID="2069b3c4f78148d5c69eb4b4253c3d5a5df9271441784801db30cf745cb98409"
Sep 11 21:35:07.847104 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:07.847093    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb"
Sep 11 21:35:07.847181 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:07.847091    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb" event={"ID":"b9d7fc44-e0c4-4933-8079-f0cd21d243a3","Type":"ContainerDied","Data":"683dc98b11a3255bc51fe7d3926c844ff96bc8831c9b84b45804796700c0e13c"}
Sep 11 21:35:07.853812 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:07.853791    2578 scope.go:117] "RemoveContainer" containerID="e2650615b7bd185105f757b5ddacde347cca55c20aa0250245480e76f4f4966a"
Sep 11 21:35:07.869767 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:07.869745    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd"]
Sep 11 21:35:07.873919 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:07.873899    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2bd772-7fc67c976c-nkbsd"]
Sep 11 21:35:07.885700 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:07.885678    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb"]
Sep 11 21:35:07.889253 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:07.889231    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2bd772-7ccf4bfdf8-fmsbb"]
Sep 11 21:35:07.928174 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:07.928149    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43b41491-60a0-4dbf-8aa7-32e9344f0497" path="/var/lib/kubelet/pods/43b41491-60a0-4dbf-8aa7-32e9344f0497/volumes"
Sep 11 21:35:07.928558 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:07.928539    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9d7fc44-e0c4-4933-8079-f0cd21d243a3" path="/var/lib/kubelet/pods/b9d7fc44-e0c4-4933-8079-f0cd21d243a3/volumes"
Sep 11 21:35:09.522990 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:09.522955    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:35:13.543350 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:13.543320    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"]
Sep 11 21:35:13.543744 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:13.543501    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd" podUID="d37d20ec-69f2-49ce-a03f-5970c06d7177" containerName="istio-proxy" containerID="cri-o://484dac02c7842e069e115bdc141f917c6efea6641a2fdb8c0a0ecdb69d881b29" gracePeriod=30
Sep 11 21:35:13.902402 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:13.902323    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-713791-rvv4k"]
Sep 11 21:35:13.902610 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:13.902598    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d7fc44-e0c4-4933-8079-f0cd21d243a3" containerName="payload-pre-processing"
Sep 11 21:35:13.902662 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:13.902611    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d7fc44-e0c4-4933-8079-f0cd21d243a3" containerName="payload-pre-processing"
Sep 11 21:35:13.902662 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:13.902623    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddd9b797-4677-4f8e-8c9a-4b98744f07ce" containerName="revoke-keys"
Sep 11 21:35:13.902662 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:13.902629    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddd9b797-4677-4f8e-8c9a-4b98744f07ce" containerName="revoke-keys"
Sep 11 21:35:13.902662 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:13.902639    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f" containerName="istio-proxy"
Sep 11 21:35:13.902662 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:13.902645    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f" containerName="istio-proxy"
Sep 11 21:35:13.902662 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:13.902654    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f" containerName="istio-proxy"
Sep 11 21:35:13.902662 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:13.902659    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f" containerName="istio-proxy"
Sep 11 21:35:13.902921 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:13.902666    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43b41491-60a0-4dbf-8aa7-32e9344f0497" containerName="payload-processing"
Sep 11 21:35:13.902921 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:13.902671    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="43b41491-60a0-4dbf-8aa7-32e9344f0497" containerName="payload-processing"
Sep 11 21:35:13.902921 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:13.902721    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="43b41491-60a0-4dbf-8aa7-32e9344f0497" containerName="payload-processing"
Sep 11 21:35:13.902921 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:13.902729    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9d7fc44-e0c4-4933-8079-f0cd21d243a3" containerName="payload-pre-processing"
Sep 11 21:35:13.902921 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:13.902734    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f" containerName="istio-proxy"
Sep 11 21:35:13.902921 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:13.902741    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddd9b797-4677-4f8e-8c9a-4b98744f07ce" containerName="revoke-keys"
Sep 11 21:35:13.907036 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:13.907019    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-713791-rvv4k"
Sep 11 21:35:13.909167 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:13.909140    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:35:13.909280 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:13.909189    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfm9g\""
Sep 11 21:35:13.913504 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:13.913481    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-713791-rvv4k"]
Sep 11 21:35:14.033432 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:14.033393    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9611c70c-bc9e-4106-a27e-4b377231944d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-713791-rvv4k\" (UID: \"9611c70c-bc9e-4106-a27e-4b377231944d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-713791-rvv4k"
Sep 11 21:35:14.134225 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:14.134180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9611c70c-bc9e-4106-a27e-4b377231944d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-713791-rvv4k\" (UID: \"9611c70c-bc9e-4106-a27e-4b377231944d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-713791-rvv4k"
Sep 11 21:35:14.134751 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:14.134732    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9611c70c-bc9e-4106-a27e-4b377231944d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-713791-rvv4k\" (UID: \"9611c70c-bc9e-4106-a27e-4b377231944d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-713791-rvv4k"
Sep 11 21:35:14.217967 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:14.217941    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-713791-rvv4k"
Sep 11 21:35:14.485693 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:14.485603    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-713791-rvv4k"]
Sep 11 21:35:14.489506 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:35:14.489479    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9611c70c_bc9e_4106_a27e_4b377231944d.slice/crio-8a5378836cc17dfb2a96e4a4b409f32150fc139c5ba03d15ff044f6a08d06eb0 WatchSource:0}: Error finding container 8a5378836cc17dfb2a96e4a4b409f32150fc139c5ba03d15ff044f6a08d06eb0: Status 404 returned error can't find the container with id 8a5378836cc17dfb2a96e4a4b409f32150fc139c5ba03d15ff044f6a08d06eb0
Sep 11 21:35:14.881855 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:14.881810    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-713791-rvv4k" event={"ID":"9611c70c-bc9e-4106-a27e-4b377231944d","Type":"ContainerStarted","Data":"01bad8100da0fe22567bac1f117b1d5e5ddd2bfa4417c269fe0719cf61959127"}
Sep 11 21:35:14.882198 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:14.881873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-713791-rvv4k" event={"ID":"9611c70c-bc9e-4106-a27e-4b377231944d","Type":"ContainerStarted","Data":"8a5378836cc17dfb2a96e4a4b409f32150fc139c5ba03d15ff044f6a08d06eb0"}
Sep 11 21:35:14.897815 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:14.897764    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-713791-rvv4k" podStartSLOduration=1.8977482129999999 podStartE2EDuration="1.897748213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:35:14.897704842 +0000 UTC m=+1781.601674140" watchObservedRunningTime="2026-09-11 21:35:14.897748213 +0000 UTC m=+1781.601717504"
Sep 11 21:35:15.887401 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:15.887370    2578 generic.go:356] "Generic (PLEG): container finished" podID="9611c70c-bc9e-4106-a27e-4b377231944d" containerID="01bad8100da0fe22567bac1f117b1d5e5ddd2bfa4417c269fe0719cf61959127" exitCode=0
Sep 11 21:35:15.887769 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:15.887418    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-713791-rvv4k" event={"ID":"9611c70c-bc9e-4106-a27e-4b377231944d","Type":"ContainerDied","Data":"01bad8100da0fe22567bac1f117b1d5e5ddd2bfa4417c269fe0719cf61959127"}
Sep 11 21:35:17.010505 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:17.010481    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-713791-rvv4k"
Sep 11 21:35:17.159464 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:17.159389    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9611c70c-bc9e-4106-a27e-4b377231944d-maas-api-service-ca\") pod \"9611c70c-bc9e-4106-a27e-4b377231944d\" (UID: \"9611c70c-bc9e-4106-a27e-4b377231944d\") "
Sep 11 21:35:17.159696 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:17.159676    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9611c70c-bc9e-4106-a27e-4b377231944d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9611c70c-bc9e-4106-a27e-4b377231944d" (UID: "9611c70c-bc9e-4106-a27e-4b377231944d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:35:17.260798 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:17.260760    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9611c70c-bc9e-4106-a27e-4b377231944d-maas-api-service-ca\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:17.897176 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:17.897142    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-713791-rvv4k" event={"ID":"9611c70c-bc9e-4106-a27e-4b377231944d","Type":"ContainerDied","Data":"8a5378836cc17dfb2a96e4a4b409f32150fc139c5ba03d15ff044f6a08d06eb0"}
Sep 11 21:35:17.897346 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:17.897191    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a5378836cc17dfb2a96e4a4b409f32150fc139c5ba03d15ff044f6a08d06eb0"
Sep 11 21:35:17.897346 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:17.897158    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-713791-rvv4k"
Sep 11 21:35:17.987372 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:17.987341    2578 patch_prober.go:28] interesting pod/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 21:35:17.987526 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:17.987395    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd" podUID="d37d20ec-69f2-49ce-a03f-5970c06d7177" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 21:35:18.783539 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.783516    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:35:18.873223 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.873156    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-istio-envoy\") pod \"d37d20ec-69f2-49ce-a03f-5970c06d7177\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") "
Sep 11 21:35:18.873223 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.873190    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-workload-certs\") pod \"d37d20ec-69f2-49ce-a03f-5970c06d7177\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") "
Sep 11 21:35:18.873223 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.873216    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-workload-socket\") pod \"d37d20ec-69f2-49ce-a03f-5970c06d7177\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") "
Sep 11 21:35:18.873457 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.873251    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-credential-socket\") pod \"d37d20ec-69f2-49ce-a03f-5970c06d7177\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") "
Sep 11 21:35:18.873457 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.873288    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp7rl\" (UniqueName: \"kubernetes.io/projected/d37d20ec-69f2-49ce-a03f-5970c06d7177-kube-api-access-mp7rl\") pod \"d37d20ec-69f2-49ce-a03f-5970c06d7177\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") "
Sep 11 21:35:18.873457 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.873311    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d37d20ec-69f2-49ce-a03f-5970c06d7177-istiod-ca-cert\") pod \"d37d20ec-69f2-49ce-a03f-5970c06d7177\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") "
Sep 11 21:35:18.873457 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.873336    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d37d20ec-69f2-49ce-a03f-5970c06d7177-istio-token\") pod \"d37d20ec-69f2-49ce-a03f-5970c06d7177\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") "
Sep 11 21:35:18.873457 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.873364    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d37d20ec-69f2-49ce-a03f-5970c06d7177-istio-podinfo\") pod \"d37d20ec-69f2-49ce-a03f-5970c06d7177\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") "
Sep 11 21:35:18.873457 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.873415    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-istio-data\") pod \"d37d20ec-69f2-49ce-a03f-5970c06d7177\" (UID: \"d37d20ec-69f2-49ce-a03f-5970c06d7177\") "
Sep 11 21:35:18.873745 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.873452    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d37d20ec-69f2-49ce-a03f-5970c06d7177" (UID: "d37d20ec-69f2-49ce-a03f-5970c06d7177"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:18.873745 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.873532    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d37d20ec-69f2-49ce-a03f-5970c06d7177" (UID: "d37d20ec-69f2-49ce-a03f-5970c06d7177"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:18.873745 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.873565    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d37d20ec-69f2-49ce-a03f-5970c06d7177" (UID: "d37d20ec-69f2-49ce-a03f-5970c06d7177"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:18.873933 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.873744    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-workload-certs\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:18.873933 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.873750    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-istio-data" (OuterVolumeSpecName: "istio-data") pod "d37d20ec-69f2-49ce-a03f-5970c06d7177" (UID: "d37d20ec-69f2-49ce-a03f-5970c06d7177"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:18.873933 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.873762    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-workload-socket\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:18.873933 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.873774    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-credential-socket\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:18.874137 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.873984    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d37d20ec-69f2-49ce-a03f-5970c06d7177-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d37d20ec-69f2-49ce-a03f-5970c06d7177" (UID: "d37d20ec-69f2-49ce-a03f-5970c06d7177"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:35:18.875447 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.875419    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37d20ec-69f2-49ce-a03f-5970c06d7177-kube-api-access-mp7rl" (OuterVolumeSpecName: "kube-api-access-mp7rl") pod "d37d20ec-69f2-49ce-a03f-5970c06d7177" (UID: "d37d20ec-69f2-49ce-a03f-5970c06d7177"). InnerVolumeSpecName "kube-api-access-mp7rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:35:18.875561 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.875452    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d37d20ec-69f2-49ce-a03f-5970c06d7177-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d37d20ec-69f2-49ce-a03f-5970c06d7177" (UID: "d37d20ec-69f2-49ce-a03f-5970c06d7177"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:35:18.875561 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.875470    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d37d20ec-69f2-49ce-a03f-5970c06d7177" (UID: "d37d20ec-69f2-49ce-a03f-5970c06d7177"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:35:18.875655 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.875556    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37d20ec-69f2-49ce-a03f-5970c06d7177-istio-token" (OuterVolumeSpecName: "istio-token") pod "d37d20ec-69f2-49ce-a03f-5970c06d7177" (UID: "d37d20ec-69f2-49ce-a03f-5970c06d7177"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:35:18.903175 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.903149    2578 generic.go:356] "Generic (PLEG): container finished" podID="d37d20ec-69f2-49ce-a03f-5970c06d7177" containerID="484dac02c7842e069e115bdc141f917c6efea6641a2fdb8c0a0ecdb69d881b29" exitCode=0
Sep 11 21:35:18.903278 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.903195    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"
Sep 11 21:35:18.903278 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.903224    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd" event={"ID":"d37d20ec-69f2-49ce-a03f-5970c06d7177","Type":"ContainerDied","Data":"484dac02c7842e069e115bdc141f917c6efea6641a2fdb8c0a0ecdb69d881b29"}
Sep 11 21:35:18.903278 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.903268    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd" event={"ID":"d37d20ec-69f2-49ce-a03f-5970c06d7177","Type":"ContainerDied","Data":"802cbe64baa1799982321d3560a4364625ffb85a797cc4907b2db445eb52b1ee"}
Sep 11 21:35:18.903432 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.903293    2578 scope.go:117] "RemoveContainer" containerID="484dac02c7842e069e115bdc141f917c6efea6641a2fdb8c0a0ecdb69d881b29"
Sep 11 21:35:18.914625 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.914608    2578 scope.go:117] "RemoveContainer" containerID="34d288c8fee081297b61f09ff4657425b2efdc703b492e16e3c91e26695ba7eb"
Sep 11 21:35:18.922176 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.922160    2578 scope.go:117] "RemoveContainer" containerID="484dac02c7842e069e115bdc141f917c6efea6641a2fdb8c0a0ecdb69d881b29"
Sep 11 21:35:18.922437 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:35:18.922420    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"484dac02c7842e069e115bdc141f917c6efea6641a2fdb8c0a0ecdb69d881b29\": container with ID starting with 484dac02c7842e069e115bdc141f917c6efea6641a2fdb8c0a0ecdb69d881b29 not found: ID does not exist" containerID="484dac02c7842e069e115bdc141f917c6efea6641a2fdb8c0a0ecdb69d881b29"
Sep 11 21:35:18.922485 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.922445    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"484dac02c7842e069e115bdc141f917c6efea6641a2fdb8c0a0ecdb69d881b29"} err="failed to get container status \"484dac02c7842e069e115bdc141f917c6efea6641a2fdb8c0a0ecdb69d881b29\": rpc error: code = NotFound desc = could not find container \"484dac02c7842e069e115bdc141f917c6efea6641a2fdb8c0a0ecdb69d881b29\": container with ID starting with 484dac02c7842e069e115bdc141f917c6efea6641a2fdb8c0a0ecdb69d881b29 not found: ID does not exist"
Sep 11 21:35:18.922485 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.922460    2578 scope.go:117] "RemoveContainer" containerID="34d288c8fee081297b61f09ff4657425b2efdc703b492e16e3c91e26695ba7eb"
Sep 11 21:35:18.922712 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:35:18.922695    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34d288c8fee081297b61f09ff4657425b2efdc703b492e16e3c91e26695ba7eb\": container with ID starting with 34d288c8fee081297b61f09ff4657425b2efdc703b492e16e3c91e26695ba7eb not found: ID does not exist" containerID="34d288c8fee081297b61f09ff4657425b2efdc703b492e16e3c91e26695ba7eb"
Sep 11 21:35:18.922768 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.922721    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d288c8fee081297b61f09ff4657425b2efdc703b492e16e3c91e26695ba7eb"} err="failed to get container status \"34d288c8fee081297b61f09ff4657425b2efdc703b492e16e3c91e26695ba7eb\": rpc error: code = NotFound desc = could not find container \"34d288c8fee081297b61f09ff4657425b2efdc703b492e16e3c91e26695ba7eb\": container with ID starting with 34d288c8fee081297b61f09ff4657425b2efdc703b492e16e3c91e26695ba7eb not found: ID does not exist"
Sep 11 21:35:18.926085 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.926066    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"]
Sep 11 21:35:18.929608 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.929591    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2bd772-openshift-default-57c966f864-sjzvd"]
Sep 11 21:35:18.974402 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.974378    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-istio-envoy\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:18.974402 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.974400    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp7rl\" (UniqueName: \"kubernetes.io/projected/d37d20ec-69f2-49ce-a03f-5970c06d7177-kube-api-access-mp7rl\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:18.974530 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.974415    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d37d20ec-69f2-49ce-a03f-5970c06d7177-istiod-ca-cert\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:18.974530 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.974427    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d37d20ec-69f2-49ce-a03f-5970c06d7177-istio-token\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:18.974530 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.974438    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d37d20ec-69f2-49ce-a03f-5970c06d7177-istio-podinfo\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:18.974530 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:18.974450    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d37d20ec-69f2-49ce-a03f-5970c06d7177-istio-data\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:19.927031 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:19.926992    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d37d20ec-69f2-49ce-a03f-5970c06d7177" path="/var/lib/kubelet/pods/d37d20ec-69f2-49ce-a03f-5970c06d7177/volumes"
Sep 11 21:35:24.621152 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:24.621119    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb"]
Sep 11 21:35:24.621783 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:24.621722    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb" podUID="b781df93-f077-495c-b3c0-9c817514e6a9" containerName="payload-pre-processing" containerID="cri-o://c2c82d935203b25761848f3262677c522d01f5b498612fa4b8cdd437e476dfdb" gracePeriod=30
Sep 11 21:35:24.801473 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:24.801437    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-713791-rvv4k"]
Sep 11 21:35:24.803339 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:24.803310    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-713791-rvv4k"]
Sep 11 21:35:24.861762 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:24.861741    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb"
Sep 11 21:35:24.930756 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:24.930724    2578 generic.go:356] "Generic (PLEG): container finished" podID="b781df93-f077-495c-b3c0-9c817514e6a9" containerID="c2c82d935203b25761848f3262677c522d01f5b498612fa4b8cdd437e476dfdb" exitCode=0
Sep 11 21:35:24.930892 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:24.930775    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb"
Sep 11 21:35:24.930892 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:24.930786    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb" event={"ID":"b781df93-f077-495c-b3c0-9c817514e6a9","Type":"ContainerDied","Data":"c2c82d935203b25761848f3262677c522d01f5b498612fa4b8cdd437e476dfdb"}
Sep 11 21:35:24.930892 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:24.930855    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb" event={"ID":"b781df93-f077-495c-b3c0-9c817514e6a9","Type":"ContainerDied","Data":"b1e8761090f366ecdee0214ecd67e952efae4e6717fdcc6a53555a6ee7435b58"}
Sep 11 21:35:24.930892 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:24.930890    2578 scope.go:117] "RemoveContainer" containerID="c2c82d935203b25761848f3262677c522d01f5b498612fa4b8cdd437e476dfdb"
Sep 11 21:35:24.938586 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:24.938569    2578 scope.go:117] "RemoveContainer" containerID="c2c82d935203b25761848f3262677c522d01f5b498612fa4b8cdd437e476dfdb"
Sep 11 21:35:24.938845 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:35:24.938810    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c82d935203b25761848f3262677c522d01f5b498612fa4b8cdd437e476dfdb\": container with ID starting with c2c82d935203b25761848f3262677c522d01f5b498612fa4b8cdd437e476dfdb not found: ID does not exist" containerID="c2c82d935203b25761848f3262677c522d01f5b498612fa4b8cdd437e476dfdb"
Sep 11 21:35:24.938929 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:24.938855    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c82d935203b25761848f3262677c522d01f5b498612fa4b8cdd437e476dfdb"} err="failed to get container status \"c2c82d935203b25761848f3262677c522d01f5b498612fa4b8cdd437e476dfdb\": rpc error: code = NotFound desc = could not find container \"c2c82d935203b25761848f3262677c522d01f5b498612fa4b8cdd437e476dfdb\": container with ID starting with c2c82d935203b25761848f3262677c522d01f5b498612fa4b8cdd437e476dfdb not found: ID does not exist"
Sep 11 21:35:25.015275 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:25.015252    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrqdc\" (UniqueName: \"kubernetes.io/projected/b781df93-f077-495c-b3c0-9c817514e6a9-kube-api-access-zrqdc\") pod \"b781df93-f077-495c-b3c0-9c817514e6a9\" (UID: \"b781df93-f077-495c-b3c0-9c817514e6a9\") "
Sep 11 21:35:25.015371 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:25.015306    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b781df93-f077-495c-b3c0-9c817514e6a9-plugins-config-volume\") pod \"b781df93-f077-495c-b3c0-9c817514e6a9\" (UID: \"b781df93-f077-495c-b3c0-9c817514e6a9\") "
Sep 11 21:35:25.015668 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:25.015640    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b781df93-f077-495c-b3c0-9c817514e6a9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b781df93-f077-495c-b3c0-9c817514e6a9" (UID: "b781df93-f077-495c-b3c0-9c817514e6a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:35:25.017200 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:25.017181    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b781df93-f077-495c-b3c0-9c817514e6a9-kube-api-access-zrqdc" (OuterVolumeSpecName: "kube-api-access-zrqdc") pod "b781df93-f077-495c-b3c0-9c817514e6a9" (UID: "b781df93-f077-495c-b3c0-9c817514e6a9"). InnerVolumeSpecName "kube-api-access-zrqdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:35:25.115877 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:25.115851    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrqdc\" (UniqueName: \"kubernetes.io/projected/b781df93-f077-495c-b3c0-9c817514e6a9-kube-api-access-zrqdc\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:25.115877 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:25.115873    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b781df93-f077-495c-b3c0-9c817514e6a9-plugins-config-volume\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:35:25.255054 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:25.255024    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb"]
Sep 11 21:35:25.261115 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:25.261092    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-713791-7b97b87c4-6mmrb"]
Sep 11 21:35:25.926800 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:25.926729    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9611c70c-bc9e-4106-a27e-4b377231944d" path="/var/lib/kubelet/pods/9611c70c-bc9e-4106-a27e-4b377231944d/volumes"
Sep 11 21:35:25.927143 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:25.927132    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b781df93-f077-495c-b3c0-9c817514e6a9" path="/var/lib/kubelet/pods/b781df93-f077-495c-b3c0-9c817514e6a9/volumes"
Sep 11 21:35:27.216311 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:27.216279    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:35:36.431687 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:35:36.431657    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:36:56.293485 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:36:56.293413    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:37:21.761539 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:37:21.761501    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:37:29.066753 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:37:29.066715    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:37:36.595667 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:37:36.595637    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:37:44.977119 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:37:44.977084    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:37:54.135369 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:37:54.135334    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:38:01.225297 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:38:01.225268    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:38:09.812279 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:38:09.812249    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:38:17.726399 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:38:17.726361    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:39:34.312055 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:34.312023    2578 scope.go:117] "RemoveContainer" containerID="0bf4b534d0553256b7c52218ab7ad0587ee0d24c1e7f92e590531c8b452ee505"
Sep 11 21:39:45.219726 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:45.219692    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:39:51.226103 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:51.226069    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:39:56.919954 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:56.919875    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"]
Sep 11 21:39:56.920285 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:56.920241    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d37d20ec-69f2-49ce-a03f-5970c06d7177" containerName="istio-proxy"
Sep 11 21:39:56.920285 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:56.920254    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d37d20ec-69f2-49ce-a03f-5970c06d7177" containerName="istio-proxy"
Sep 11 21:39:56.920285 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:56.920262    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b781df93-f077-495c-b3c0-9c817514e6a9" containerName="payload-pre-processing"
Sep 11 21:39:56.920285 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:56.920267    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b781df93-f077-495c-b3c0-9c817514e6a9" containerName="payload-pre-processing"
Sep 11 21:39:56.920285 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:56.920282    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9611c70c-bc9e-4106-a27e-4b377231944d" containerName="revoke-keys"
Sep 11 21:39:56.920285 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:56.920288    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9611c70c-bc9e-4106-a27e-4b377231944d" containerName="revoke-keys"
Sep 11 21:39:56.920625 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:56.920334    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d37d20ec-69f2-49ce-a03f-5970c06d7177" containerName="istio-proxy"
Sep 11 21:39:56.920625 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:56.920340    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9611c70c-bc9e-4106-a27e-4b377231944d" containerName="revoke-keys"
Sep 11 21:39:56.920625 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:56.920346    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d37d20ec-69f2-49ce-a03f-5970c06d7177" containerName="istio-proxy"
Sep 11 21:39:56.920625 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:56.920351    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b781df93-f077-495c-b3c0-9c817514e6a9" containerName="payload-pre-processing"
Sep 11 21:39:56.920625 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:56.920358    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8d3677f-0ae8-4756-9ee2-dcc7b5875e9f" containerName="istio-proxy"
Sep 11 21:39:56.922921 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:56.922899    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:39:56.925345 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:56.925328    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-a7233f2e-kserve-self-signed-certs\""
Sep 11 21:39:56.928684 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:56.928662    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"]
Sep 11 21:39:57.020964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:57.020933    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-home\") pod \"e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw\" (UID: \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\") " pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:39:57.021114 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:57.020974    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqlmv\" (UniqueName: \"kubernetes.io/projected/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-kube-api-access-wqlmv\") pod \"e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw\" (UID: \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\") " pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:39:57.021114 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:57.020992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-model-cache\") pod \"e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw\" (UID: \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\") " pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:39:57.021114 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:57.021059    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-tmp-dir\") pod \"e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw\" (UID: \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\") " pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:39:57.021114 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:57.021107    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-dshm\") pod \"e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw\" (UID: \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\") " pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:39:57.021250 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:57.021125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-tls-certs\") pod \"e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw\" (UID: \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\") " pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:39:57.122190 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:57.122142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqlmv\" (UniqueName: \"kubernetes.io/projected/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-kube-api-access-wqlmv\") pod \"e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw\" (UID: \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\") " pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:39:57.122190 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:57.122192    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-model-cache\") pod \"e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw\" (UID: \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\") " pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:39:57.122399 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:57.122225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-tmp-dir\") pod \"e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw\" (UID: \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\") " pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:39:57.122399 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:57.122258    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-dshm\") pod \"e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw\" (UID: \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\") " pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:39:57.122495 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:57.122471    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-tls-certs\") pod \"e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw\" (UID: \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\") " pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:39:57.122558 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:57.122540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-home\") pod \"e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw\" (UID: \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\") " pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:39:57.122700 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:57.122674    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-tmp-dir\") pod \"e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw\" (UID: \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\") " pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:39:57.122868 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:57.122727    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-model-cache\") pod \"e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw\" (UID: \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\") " pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:39:57.122959 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:57.122872    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-home\") pod \"e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw\" (UID: \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\") " pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:39:57.124446 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:57.124408    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-dshm\") pod \"e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw\" (UID: \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\") " pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:39:57.124772 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:57.124754    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-tls-certs\") pod \"e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw\" (UID: \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\") " pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:39:57.129711 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:57.129694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqlmv\" (UniqueName: \"kubernetes.io/projected/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-kube-api-access-wqlmv\") pod \"e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw\" (UID: \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\") " pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:39:57.233070 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:57.233026    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:39:57.502069 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:57.501989    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"]
Sep 11 21:39:57.504974 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:39:57.504942    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94f747a2_bc0b_4ad5_89e0_f47ad6d7ff9b.slice/crio-d48b0ac6ba47f15c681b442606a7914d98aa5ebad50d3710918b17e6fe1fa87b WatchSource:0}: Error finding container d48b0ac6ba47f15c681b442606a7914d98aa5ebad50d3710918b17e6fe1fa87b: Status 404 returned error can't find the container with id d48b0ac6ba47f15c681b442606a7914d98aa5ebad50d3710918b17e6fe1fa87b
Sep 11 21:39:58.033835 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:58.033795    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw" event={"ID":"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b","Type":"ContainerStarted","Data":"5e0a3a64e0e7e7c2f6b4e5e0843aca9675c5d2f9846f1f18cf34abe9c809bbf9"}
Sep 11 21:39:58.034287 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:58.033848    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw" event={"ID":"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b","Type":"ContainerStarted","Data":"d48b0ac6ba47f15c681b442606a7914d98aa5ebad50d3710918b17e6fe1fa87b"}
Sep 11 21:39:58.056712 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:58.056672    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw" podStartSLOduration=2.05665873 podStartE2EDuration="2.05665873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:39:58.055488021 +0000 UTC m=+2064.759457310" watchObservedRunningTime="2026-09-11 21:39:58.05665873 +0000 UTC m=+2064.760628015"
Sep 11 21:39:59.522502 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:39:59.522470    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:40:07.233839 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:07.233794    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:40:07.234309 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:07.233861    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:40:07.245870 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:07.245847    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:40:08.085965 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:08.085937    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:40:09.319895 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.319809    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"]
Sep 11 21:40:09.320271 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.320080    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d37d20ec-69f2-49ce-a03f-5970c06d7177" containerName="istio-proxy"
Sep 11 21:40:09.320271 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.320090    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d37d20ec-69f2-49ce-a03f-5970c06d7177" containerName="istio-proxy"
Sep 11 21:40:09.322909 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.322894    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:09.325289 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.325266    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-a7233f2e-kserve-self-signed-certs\""
Sep 11 21:40:09.337834 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.337795    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"]
Sep 11 21:40:09.409977 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.409946    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f83f9287-e642-4505-82a1-9fcae634b99b-tls-certs\") pod \"e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7\" (UID: \"f83f9287-e642-4505-82a1-9fcae634b99b\") " pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:09.410141 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.409987    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvf56\" (UniqueName: \"kubernetes.io/projected/f83f9287-e642-4505-82a1-9fcae634b99b-kube-api-access-wvf56\") pod \"e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7\" (UID: \"f83f9287-e642-4505-82a1-9fcae634b99b\") " pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:09.410141 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.410020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f83f9287-e642-4505-82a1-9fcae634b99b-dshm\") pod \"e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7\" (UID: \"f83f9287-e642-4505-82a1-9fcae634b99b\") " pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:09.410141 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.410092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f83f9287-e642-4505-82a1-9fcae634b99b-model-cache\") pod \"e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7\" (UID: \"f83f9287-e642-4505-82a1-9fcae634b99b\") " pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:09.410141 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.410121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f83f9287-e642-4505-82a1-9fcae634b99b-tmp-dir\") pod \"e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7\" (UID: \"f83f9287-e642-4505-82a1-9fcae634b99b\") " pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:09.410279 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.410174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f83f9287-e642-4505-82a1-9fcae634b99b-home\") pod \"e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7\" (UID: \"f83f9287-e642-4505-82a1-9fcae634b99b\") " pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:09.511406 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.511370    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f83f9287-e642-4505-82a1-9fcae634b99b-tls-certs\") pod \"e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7\" (UID: \"f83f9287-e642-4505-82a1-9fcae634b99b\") " pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:09.511576 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.511423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvf56\" (UniqueName: \"kubernetes.io/projected/f83f9287-e642-4505-82a1-9fcae634b99b-kube-api-access-wvf56\") pod \"e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7\" (UID: \"f83f9287-e642-4505-82a1-9fcae634b99b\") " pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:09.511576 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.511459    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f83f9287-e642-4505-82a1-9fcae634b99b-dshm\") pod \"e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7\" (UID: \"f83f9287-e642-4505-82a1-9fcae634b99b\") " pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:09.511576 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.511501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f83f9287-e642-4505-82a1-9fcae634b99b-model-cache\") pod \"e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7\" (UID: \"f83f9287-e642-4505-82a1-9fcae634b99b\") " pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:09.511576 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.511524    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f83f9287-e642-4505-82a1-9fcae634b99b-tmp-dir\") pod \"e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7\" (UID: \"f83f9287-e642-4505-82a1-9fcae634b99b\") " pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:09.511576 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.511563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f83f9287-e642-4505-82a1-9fcae634b99b-home\") pod \"e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7\" (UID: \"f83f9287-e642-4505-82a1-9fcae634b99b\") " pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:09.511952 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.511926    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f83f9287-e642-4505-82a1-9fcae634b99b-home\") pod \"e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7\" (UID: \"f83f9287-e642-4505-82a1-9fcae634b99b\") " pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:09.512044 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.511997    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f83f9287-e642-4505-82a1-9fcae634b99b-model-cache\") pod \"e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7\" (UID: \"f83f9287-e642-4505-82a1-9fcae634b99b\") " pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:09.512165 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.512144    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f83f9287-e642-4505-82a1-9fcae634b99b-tmp-dir\") pod \"e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7\" (UID: \"f83f9287-e642-4505-82a1-9fcae634b99b\") " pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:09.514196 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.514174    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f83f9287-e642-4505-82a1-9fcae634b99b-dshm\") pod \"e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7\" (UID: \"f83f9287-e642-4505-82a1-9fcae634b99b\") " pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:09.514391 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.514329    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f83f9287-e642-4505-82a1-9fcae634b99b-tls-certs\") pod \"e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7\" (UID: \"f83f9287-e642-4505-82a1-9fcae634b99b\") " pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:09.519737 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.519712    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvf56\" (UniqueName: \"kubernetes.io/projected/f83f9287-e642-4505-82a1-9fcae634b99b-kube-api-access-wvf56\") pod \"e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7\" (UID: \"f83f9287-e642-4505-82a1-9fcae634b99b\") " pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:09.636104 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.636013    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:09.905031 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:09.904954    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"]
Sep 11 21:40:09.907816 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:40:09.907784    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf83f9287_e642_4505_82a1_9fcae634b99b.slice/crio-39ff0de85c01c6768c8a74edce4ee767b74bd2a617e1116eb3a9080e78db4a03 WatchSource:0}: Error finding container 39ff0de85c01c6768c8a74edce4ee767b74bd2a617e1116eb3a9080e78db4a03: Status 404 returned error can't find the container with id 39ff0de85c01c6768c8a74edce4ee767b74bd2a617e1116eb3a9080e78db4a03
Sep 11 21:40:10.084270 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:10.084240    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7" event={"ID":"f83f9287-e642-4505-82a1-9fcae634b99b","Type":"ContainerStarted","Data":"d5485b8fb74297f686e3624a99486213890d06a67025809a59b97033fb3938a1"}
Sep 11 21:40:10.084459 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:10.084287    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7" event={"ID":"f83f9287-e642-4505-82a1-9fcae634b99b","Type":"ContainerStarted","Data":"39ff0de85c01c6768c8a74edce4ee767b74bd2a617e1116eb3a9080e78db4a03"}
Sep 11 21:40:10.100923 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:10.100868    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7" podStartSLOduration=1.100852098 podStartE2EDuration="1.100852098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:40:10.099948226 +0000 UTC m=+2076.803917514" watchObservedRunningTime="2026-09-11 21:40:10.100852098 +0000 UTC m=+2076.804821401"
Sep 11 21:40:12.513223 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:12.513192    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:40:19.636627 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:19.636598    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:19.636627 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:19.636634    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:19.648523 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:19.648502    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:20.137341 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:20.137315    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:34.013938 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:34.013901    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"]
Sep 11 21:40:35.722554 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:35.722518    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"]
Sep 11 21:40:36.222577 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:36.222535    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:40:45.051683 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:45.051652    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:40:49.034390 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:49.034331    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw" podUID="94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b" containerName="main" containerID="cri-o://5e0a3a64e0e7e7c2f6b4e5e0843aca9675c5d2f9846f1f18cf34abe9c809bbf9" gracePeriod=45
Sep 11 21:40:49.246864 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:49.246808    2578 generic.go:356] "Generic (PLEG): container finished" podID="94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b" containerID="5e0a3a64e0e7e7c2f6b4e5e0843aca9675c5d2f9846f1f18cf34abe9c809bbf9" exitCode=0
Sep 11 21:40:49.247015 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:49.246881    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw" event={"ID":"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b","Type":"ContainerDied","Data":"5e0a3a64e0e7e7c2f6b4e5e0843aca9675c5d2f9846f1f18cf34abe9c809bbf9"}
Sep 11 21:40:49.274182 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:49.274159    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:40:49.413281 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:49.413206    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-tmp-dir\") pod \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\" (UID: \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\") "
Sep 11 21:40:49.413281 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:49.413274    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-home\") pod \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\" (UID: \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\") "
Sep 11 21:40:49.413506 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:49.413293    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-tls-certs\") pod \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\" (UID: \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\") "
Sep 11 21:40:49.413506 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:49.413312    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqlmv\" (UniqueName: \"kubernetes.io/projected/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-kube-api-access-wqlmv\") pod \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\" (UID: \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\") "
Sep 11 21:40:49.413506 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:49.413325    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-dshm\") pod \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\" (UID: \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\") "
Sep 11 21:40:49.413506 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:49.413348    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-model-cache\") pod \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\" (UID: \"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b\") "
Sep 11 21:40:49.413506 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:49.413471    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b" (UID: "94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:40:49.413738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:49.413600    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-home" (OuterVolumeSpecName: "home") pod "94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b" (UID: "94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:40:49.413738 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:49.413678    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-model-cache" (OuterVolumeSpecName: "model-cache") pod "94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b" (UID: "94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:40:49.415451 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:49.415428    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-dshm" (OuterVolumeSpecName: "dshm") pod "94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b" (UID: "94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:40:49.415547 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:49.415502    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b" (UID: "94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:40:49.415604 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:49.415570    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-kube-api-access-wqlmv" (OuterVolumeSpecName: "kube-api-access-wqlmv") pod "94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b" (UID: "94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b"). InnerVolumeSpecName "kube-api-access-wqlmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:40:49.513997 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:49.513968    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-home\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:40:49.513997 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:49.513993    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-tls-certs\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:40:49.514153 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:49.514009    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqlmv\" (UniqueName: \"kubernetes.io/projected/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-kube-api-access-wqlmv\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:40:49.514153 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:49.514020    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-dshm\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:40:49.514153 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:49.514031    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-model-cache\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:40:49.514153 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:49.514042    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b-tmp-dir\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:40:50.252357 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:50.252265    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw" event={"ID":"94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b","Type":"ContainerDied","Data":"d48b0ac6ba47f15c681b442606a7914d98aa5ebad50d3710918b17e6fe1fa87b"}
Sep 11 21:40:50.252357 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:50.252287    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"
Sep 11 21:40:50.252357 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:50.252323    2578 scope.go:117] "RemoveContainer" containerID="5e0a3a64e0e7e7c2f6b4e5e0843aca9675c5d2f9846f1f18cf34abe9c809bbf9"
Sep 11 21:40:50.268031 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:50.268009    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"]
Sep 11 21:40:50.270087 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:50.270069    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-a7233f2e-kserve-547cd97df5-gsprw"]
Sep 11 21:40:50.740655 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:50.740616    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7" podUID="f83f9287-e642-4505-82a1-9fcae634b99b" containerName="main" containerID="cri-o://d5485b8fb74297f686e3624a99486213890d06a67025809a59b97033fb3938a1" gracePeriod=45
Sep 11 21:40:50.973930 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:50.973909    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:51.127981 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.127917    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f83f9287-e642-4505-82a1-9fcae634b99b-tmp-dir\") pod \"f83f9287-e642-4505-82a1-9fcae634b99b\" (UID: \"f83f9287-e642-4505-82a1-9fcae634b99b\") "
Sep 11 21:40:51.127981 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.127953    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f83f9287-e642-4505-82a1-9fcae634b99b-dshm\") pod \"f83f9287-e642-4505-82a1-9fcae634b99b\" (UID: \"f83f9287-e642-4505-82a1-9fcae634b99b\") "
Sep 11 21:40:51.127981 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.127982    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f83f9287-e642-4505-82a1-9fcae634b99b-tls-certs\") pod \"f83f9287-e642-4505-82a1-9fcae634b99b\" (UID: \"f83f9287-e642-4505-82a1-9fcae634b99b\") "
Sep 11 21:40:51.128230 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.128002    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f83f9287-e642-4505-82a1-9fcae634b99b-home\") pod \"f83f9287-e642-4505-82a1-9fcae634b99b\" (UID: \"f83f9287-e642-4505-82a1-9fcae634b99b\") "
Sep 11 21:40:51.128230 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.128020    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvf56\" (UniqueName: \"kubernetes.io/projected/f83f9287-e642-4505-82a1-9fcae634b99b-kube-api-access-wvf56\") pod \"f83f9287-e642-4505-82a1-9fcae634b99b\" (UID: \"f83f9287-e642-4505-82a1-9fcae634b99b\") "
Sep 11 21:40:51.128230 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.128060    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f83f9287-e642-4505-82a1-9fcae634b99b-model-cache\") pod \"f83f9287-e642-4505-82a1-9fcae634b99b\" (UID: \"f83f9287-e642-4505-82a1-9fcae634b99b\") "
Sep 11 21:40:51.128230 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.128183    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83f9287-e642-4505-82a1-9fcae634b99b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f83f9287-e642-4505-82a1-9fcae634b99b" (UID: "f83f9287-e642-4505-82a1-9fcae634b99b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:40:51.128440 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.128324    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f83f9287-e642-4505-82a1-9fcae634b99b-tmp-dir\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:40:51.128440 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.128367    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83f9287-e642-4505-82a1-9fcae634b99b-home" (OuterVolumeSpecName: "home") pod "f83f9287-e642-4505-82a1-9fcae634b99b" (UID: "f83f9287-e642-4505-82a1-9fcae634b99b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:40:51.128440 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.128405    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83f9287-e642-4505-82a1-9fcae634b99b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f83f9287-e642-4505-82a1-9fcae634b99b" (UID: "f83f9287-e642-4505-82a1-9fcae634b99b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:40:51.130260 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.130235    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f83f9287-e642-4505-82a1-9fcae634b99b-kube-api-access-wvf56" (OuterVolumeSpecName: "kube-api-access-wvf56") pod "f83f9287-e642-4505-82a1-9fcae634b99b" (UID: "f83f9287-e642-4505-82a1-9fcae634b99b"). InnerVolumeSpecName "kube-api-access-wvf56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:40:51.130363 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.130273    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f83f9287-e642-4505-82a1-9fcae634b99b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f83f9287-e642-4505-82a1-9fcae634b99b" (UID: "f83f9287-e642-4505-82a1-9fcae634b99b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:40:51.130363 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.130303    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83f9287-e642-4505-82a1-9fcae634b99b-dshm" (OuterVolumeSpecName: "dshm") pod "f83f9287-e642-4505-82a1-9fcae634b99b" (UID: "f83f9287-e642-4505-82a1-9fcae634b99b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:40:51.229087 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.229065    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f83f9287-e642-4505-82a1-9fcae634b99b-tls-certs\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:40:51.229087 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.229084    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f83f9287-e642-4505-82a1-9fcae634b99b-home\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:40:51.229220 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.229094    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvf56\" (UniqueName: \"kubernetes.io/projected/f83f9287-e642-4505-82a1-9fcae634b99b-kube-api-access-wvf56\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:40:51.229220 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.229102    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f83f9287-e642-4505-82a1-9fcae634b99b-model-cache\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:40:51.229220 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.229110    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f83f9287-e642-4505-82a1-9fcae634b99b-dshm\") on node \"ip-10-0-142-224.ec2.internal\" DevicePath \"\""
Sep 11 21:40:51.257473 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.257446    2578 generic.go:356] "Generic (PLEG): container finished" podID="f83f9287-e642-4505-82a1-9fcae634b99b" containerID="d5485b8fb74297f686e3624a99486213890d06a67025809a59b97033fb3938a1" exitCode=0
Sep 11 21:40:51.257815 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.257487    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"
Sep 11 21:40:51.257815 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.257538    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7" event={"ID":"f83f9287-e642-4505-82a1-9fcae634b99b","Type":"ContainerDied","Data":"d5485b8fb74297f686e3624a99486213890d06a67025809a59b97033fb3938a1"}
Sep 11 21:40:51.257815 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.257579    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7" event={"ID":"f83f9287-e642-4505-82a1-9fcae634b99b","Type":"ContainerDied","Data":"39ff0de85c01c6768c8a74edce4ee767b74bd2a617e1116eb3a9080e78db4a03"}
Sep 11 21:40:51.257815 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.257599    2578 scope.go:117] "RemoveContainer" containerID="d5485b8fb74297f686e3624a99486213890d06a67025809a59b97033fb3938a1"
Sep 11 21:40:51.265691 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.265676    2578 scope.go:117] "RemoveContainer" containerID="d5485b8fb74297f686e3624a99486213890d06a67025809a59b97033fb3938a1"
Sep 11 21:40:51.265944 ip-10-0-142-224 kubenswrapper[2578]: E0911 21:40:51.265929    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5485b8fb74297f686e3624a99486213890d06a67025809a59b97033fb3938a1\": container with ID starting with d5485b8fb74297f686e3624a99486213890d06a67025809a59b97033fb3938a1 not found: ID does not exist" containerID="d5485b8fb74297f686e3624a99486213890d06a67025809a59b97033fb3938a1"
Sep 11 21:40:51.265996 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.265951    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5485b8fb74297f686e3624a99486213890d06a67025809a59b97033fb3938a1"} err="failed to get container status \"d5485b8fb74297f686e3624a99486213890d06a67025809a59b97033fb3938a1\": rpc error: code = NotFound desc = could not find container \"d5485b8fb74297f686e3624a99486213890d06a67025809a59b97033fb3938a1\": container with ID starting with d5485b8fb74297f686e3624a99486213890d06a67025809a59b97033fb3938a1 not found: ID does not exist"
Sep 11 21:40:51.279285 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.279264    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"]
Sep 11 21:40:51.285703 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.285681    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-a7233f2e-kserve-66858dc875-x2jr7"]
Sep 11 21:40:51.926934 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.926902    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b" path="/var/lib/kubelet/pods/94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b/volumes"
Sep 11 21:40:51.927285 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:51.927272    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f83f9287-e642-4505-82a1-9fcae634b99b" path="/var/lib/kubelet/pods/f83f9287-e642-4505-82a1-9fcae634b99b/volumes"
Sep 11 21:40:54.511658 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:40:54.511631    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:41:02.322157 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:02.322124    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:41:03.208529 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.208482    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5dfc94d75-zbgjw"]
Sep 11 21:41:03.208795 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.208783    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b" containerName="main"
Sep 11 21:41:03.208855 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.208797    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b" containerName="main"
Sep 11 21:41:03.208855 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.208810    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f83f9287-e642-4505-82a1-9fcae634b99b" containerName="main"
Sep 11 21:41:03.208855 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.208815    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f83f9287-e642-4505-82a1-9fcae634b99b" containerName="main"
Sep 11 21:41:03.208955 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.208882    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f83f9287-e642-4505-82a1-9fcae634b99b" containerName="main"
Sep 11 21:41:03.208955 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.208889    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="94f747a2-bc0b-4ad5-89e0-f47ad6d7ff9b" containerName="main"
Sep 11 21:41:03.212991 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.212968    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5dfc94d75-zbgjw"
Sep 11 21:41:03.216791 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.216769    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 21:41:03.216946 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.216878    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 21:41:03.217025 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.216891    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4pvmq\""
Sep 11 21:41:03.217025 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.216893    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:41:03.217025 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.216893    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 21:41:03.221712 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.221692    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5dfc94d75-zbgjw"]
Sep 11 21:41:03.310669 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.310645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a0a009d-28d3-4caa-bfb7-7515aebb9b7c-openshift-service-ca\") pod \"maas-api-5dfc94d75-zbgjw\" (UID: \"4a0a009d-28d3-4caa-bfb7-7515aebb9b7c\") " pod="odh-ai-gateway-infra/maas-api-5dfc94d75-zbgjw"
Sep 11 21:41:03.310794 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.310686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a0a009d-28d3-4caa-bfb7-7515aebb9b7c-maas-api-tls\") pod \"maas-api-5dfc94d75-zbgjw\" (UID: \"4a0a009d-28d3-4caa-bfb7-7515aebb9b7c\") " pod="odh-ai-gateway-infra/maas-api-5dfc94d75-zbgjw"
Sep 11 21:41:03.310794 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.310713    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pcfn\" (UniqueName: \"kubernetes.io/projected/4a0a009d-28d3-4caa-bfb7-7515aebb9b7c-kube-api-access-6pcfn\") pod \"maas-api-5dfc94d75-zbgjw\" (UID: \"4a0a009d-28d3-4caa-bfb7-7515aebb9b7c\") " pod="odh-ai-gateway-infra/maas-api-5dfc94d75-zbgjw"
Sep 11 21:41:03.310794 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.310778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a0a009d-28d3-4caa-bfb7-7515aebb9b7c-metrics-certs\") pod \"maas-api-5dfc94d75-zbgjw\" (UID: \"4a0a009d-28d3-4caa-bfb7-7515aebb9b7c\") " pod="odh-ai-gateway-infra/maas-api-5dfc94d75-zbgjw"
Sep 11 21:41:03.411583 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.411560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pcfn\" (UniqueName: \"kubernetes.io/projected/4a0a009d-28d3-4caa-bfb7-7515aebb9b7c-kube-api-access-6pcfn\") pod \"maas-api-5dfc94d75-zbgjw\" (UID: \"4a0a009d-28d3-4caa-bfb7-7515aebb9b7c\") " pod="odh-ai-gateway-infra/maas-api-5dfc94d75-zbgjw"
Sep 11 21:41:03.411944 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.411592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a0a009d-28d3-4caa-bfb7-7515aebb9b7c-metrics-certs\") pod \"maas-api-5dfc94d75-zbgjw\" (UID: \"4a0a009d-28d3-4caa-bfb7-7515aebb9b7c\") " pod="odh-ai-gateway-infra/maas-api-5dfc94d75-zbgjw"
Sep 11 21:41:03.411944 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.411628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a0a009d-28d3-4caa-bfb7-7515aebb9b7c-openshift-service-ca\") pod \"maas-api-5dfc94d75-zbgjw\" (UID: \"4a0a009d-28d3-4caa-bfb7-7515aebb9b7c\") " pod="odh-ai-gateway-infra/maas-api-5dfc94d75-zbgjw"
Sep 11 21:41:03.411944 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.411667    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a0a009d-28d3-4caa-bfb7-7515aebb9b7c-maas-api-tls\") pod \"maas-api-5dfc94d75-zbgjw\" (UID: \"4a0a009d-28d3-4caa-bfb7-7515aebb9b7c\") " pod="odh-ai-gateway-infra/maas-api-5dfc94d75-zbgjw"
Sep 11 21:41:03.412347 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.412326    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a0a009d-28d3-4caa-bfb7-7515aebb9b7c-openshift-service-ca\") pod \"maas-api-5dfc94d75-zbgjw\" (UID: \"4a0a009d-28d3-4caa-bfb7-7515aebb9b7c\") " pod="odh-ai-gateway-infra/maas-api-5dfc94d75-zbgjw"
Sep 11 21:41:03.413959 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.413944    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a0a009d-28d3-4caa-bfb7-7515aebb9b7c-metrics-certs\") pod \"maas-api-5dfc94d75-zbgjw\" (UID: \"4a0a009d-28d3-4caa-bfb7-7515aebb9b7c\") " pod="odh-ai-gateway-infra/maas-api-5dfc94d75-zbgjw"
Sep 11 21:41:03.414019 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.413999    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a0a009d-28d3-4caa-bfb7-7515aebb9b7c-maas-api-tls\") pod \"maas-api-5dfc94d75-zbgjw\" (UID: \"4a0a009d-28d3-4caa-bfb7-7515aebb9b7c\") " pod="odh-ai-gateway-infra/maas-api-5dfc94d75-zbgjw"
Sep 11 21:41:03.420262 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.420239    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pcfn\" (UniqueName: \"kubernetes.io/projected/4a0a009d-28d3-4caa-bfb7-7515aebb9b7c-kube-api-access-6pcfn\") pod \"maas-api-5dfc94d75-zbgjw\" (UID: \"4a0a009d-28d3-4caa-bfb7-7515aebb9b7c\") " pod="odh-ai-gateway-infra/maas-api-5dfc94d75-zbgjw"
Sep 11 21:41:03.523243 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.523171    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5dfc94d75-zbgjw"
Sep 11 21:41:03.791810 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.791787    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5dfc94d75-zbgjw"]
Sep 11 21:41:03.794423 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:41:03.794389    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0a009d_28d3_4caa_bfb7_7515aebb9b7c.slice/crio-d53bb5e58e91b56b6866c5f0972a58858a4b48bda13cb18870b5ff1a07a20fc9 WatchSource:0}: Error finding container d53bb5e58e91b56b6866c5f0972a58858a4b48bda13cb18870b5ff1a07a20fc9: Status 404 returned error can't find the container with id d53bb5e58e91b56b6866c5f0972a58858a4b48bda13cb18870b5ff1a07a20fc9
Sep 11 21:41:03.795735 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:03.795718    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:41:04.314812 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:04.314771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5dfc94d75-zbgjw" event={"ID":"4a0a009d-28d3-4caa-bfb7-7515aebb9b7c","Type":"ContainerStarted","Data":"d53bb5e58e91b56b6866c5f0972a58858a4b48bda13cb18870b5ff1a07a20fc9"}
Sep 11 21:41:06.325030 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:06.324950    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5dfc94d75-zbgjw" event={"ID":"4a0a009d-28d3-4caa-bfb7-7515aebb9b7c","Type":"ContainerStarted","Data":"c38a4338ed3d96ce72e55e128c84fa958e24d82a76af54b90fc87681f4f7bfae"}
Sep 11 21:41:06.325377 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:06.325067    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5dfc94d75-zbgjw"
Sep 11 21:41:06.343309 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:06.343262    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5dfc94d75-zbgjw" podStartSLOduration=1.184939049 podStartE2EDuration="3.343249152s" totalImagesPullingTime="2.158310103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:41:06.342107201 +0000 UTC m=+2133.046076489" watchObservedRunningTime="2026-09-11 21:41:06.343249152 +0000 UTC m=+2133.047218491"
Sep 11 21:41:09.216643 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:09.216609    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:41:12.334771 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:12.334743    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5dfc94d75-zbgjw"
Sep 11 21:41:15.713892 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:15.713858    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:41:21.715672 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:21.715642    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wq4ps"]
Sep 11 21:41:24.719529 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:24.719494    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bbdbcc98c-pn522_e1542ea2-457c-41d8-9a59-aec6f3e2240d/authorino/0.log"
Sep 11 21:41:30.853722 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:30.853640    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8457b95c77-b4xt2_d6d22dfa-1693-49d0-b4d6-087a181b3b57/copy-manifests/0.log"
Sep 11 21:41:30.865695 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:30.865670    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8457b95c77-b4xt2_d6d22dfa-1693-49d0-b4d6-087a181b3b57/manager/0.log"
Sep 11 21:41:31.477858 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:31.477809    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5dfc94d75-zbgjw_4a0a009d-28d3-4caa-bfb7-7515aebb9b7c/maas-api/0.log"
Sep 11 21:41:31.711293 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:31.711265    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819370-bvjnh_9e85ed78-f7a5-436b-ae76-941ad037f7b4/cleanup/0.log"
Sep 11 21:41:32.577483 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:32.577449    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bbdbcc98c-pn522_e1542ea2-457c-41d8-9a59-aec6f3e2240d/authorino/0.log"
Sep 11 21:41:32.923927 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:32.923901    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-tl8b4_5cf77d2a-487f-45f5-bf80-ff3768504fa8/kuadrant-console-plugin/0.log"
Sep 11 21:41:33.176980 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:33.176893    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-wq4ps_b2f8ff99-9d97-4ed7-9e33-3b186aa765ff/limitador/0.log"
Sep 11 21:41:34.016285 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:34.016253    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-677bd9b6b8-p99rw_6c8aa1b3-5d8a-4bce-ad61-b2b79c5c1a88/kube-auth-proxy/0.log"
Sep 11 21:41:34.332966 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:34.332892    2578 scope.go:117] "RemoveContainer" containerID="01bad8100da0fe22567bac1f117b1d5e5ddd2bfa4417c269fe0719cf61959127"
Sep 11 21:41:34.341653 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:34.341632    2578 scope.go:117] "RemoveContainer" containerID="c56688d33dde36008ad9da06165bd6594681b9ee05727d7e3a79ec7a7e7694ae"
Sep 11 21:41:35.309698 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:35.309673    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhdm8r_e9e59a89-cdba-4e67-971d-351cabc2607c/main/0.log"
Sep 11 21:41:35.535713 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:35.535648    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-tzvzf_5b863e7e-d89a-4f7f-a062-0e7be1c656b3/main/0.log"
Sep 11 21:41:44.452147 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:44.452118    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mbqxp_a06b3567-7898-44a3-b981-5d5180c6130e/global-pull-secret-syncer/0.log"
Sep 11 21:41:44.621726 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:44.621696    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fx2dp_1c3adb7a-bb53-45a7-a118-aa8478dcf7bc/konnectivity-agent/0.log"
Sep 11 21:41:44.693765 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:44.693737    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-224.ec2.internal_f18dae482bb48fb8ae2e58c513d48e19/haproxy/0.log"
Sep 11 21:41:49.400377 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:49.400350    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bbdbcc98c-pn522_e1542ea2-457c-41d8-9a59-aec6f3e2240d/authorino/0.log"
Sep 11 21:41:49.503976 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:49.503950    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-tl8b4_5cf77d2a-487f-45f5-bf80-ff3768504fa8/kuadrant-console-plugin/0.log"
Sep 11 21:41:49.571016 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:49.570987    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-wq4ps_b2f8ff99-9d97-4ed7-9e33-3b186aa765ff/limitador/0.log"
Sep 11 21:41:51.883833 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:51.883791    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hm52m_1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7/node-exporter/0.log"
Sep 11 21:41:51.913390 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:51.913368    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hm52m_1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7/kube-rbac-proxy/0.log"
Sep 11 21:41:51.952323 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:51.952302    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hm52m_1134f7c2-f8c5-4ab9-b028-3a76c9e0b4a7/init-textfile/0.log"
Sep 11 21:41:55.193801 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:55.193769    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27"]
Sep 11 21:41:55.197116 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:55.197100    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27"
Sep 11 21:41:55.199619 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:55.199598    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nrqhg\"/\"openshift-service-ca.crt\""
Sep 11 21:41:55.199744 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:55.199722    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nrqhg\"/\"default-dockercfg-bjkff\""
Sep 11 21:41:55.200365 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:55.200343    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nrqhg\"/\"kube-root-ca.crt\""
Sep 11 21:41:55.217399 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:55.217366    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27"]
Sep 11 21:41:55.295783 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:55.295754    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl84k\" (UniqueName: \"kubernetes.io/projected/22829ef6-1c2d-4f77-87b4-769493f93966-kube-api-access-gl84k\") pod \"perf-node-gather-daemonset-v6r27\" (UID: \"22829ef6-1c2d-4f77-87b4-769493f93966\") " pod="openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27"
Sep 11 21:41:55.295964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:55.295845    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22829ef6-1c2d-4f77-87b4-769493f93966-sys\") pod \"perf-node-gather-daemonset-v6r27\" (UID: \"22829ef6-1c2d-4f77-87b4-769493f93966\") " pod="openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27"
Sep 11 21:41:55.295964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:55.295873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22829ef6-1c2d-4f77-87b4-769493f93966-lib-modules\") pod \"perf-node-gather-daemonset-v6r27\" (UID: \"22829ef6-1c2d-4f77-87b4-769493f93966\") " pod="openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27"
Sep 11 21:41:55.295964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:55.295890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/22829ef6-1c2d-4f77-87b4-769493f93966-proc\") pod \"perf-node-gather-daemonset-v6r27\" (UID: \"22829ef6-1c2d-4f77-87b4-769493f93966\") " pod="openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27"
Sep 11 21:41:55.295964 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:55.295909    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/22829ef6-1c2d-4f77-87b4-769493f93966-podres\") pod \"perf-node-gather-daemonset-v6r27\" (UID: \"22829ef6-1c2d-4f77-87b4-769493f93966\") " pod="openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27"
Sep 11 21:41:55.396792 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:55.396762    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22829ef6-1c2d-4f77-87b4-769493f93966-sys\") pod \"perf-node-gather-daemonset-v6r27\" (UID: \"22829ef6-1c2d-4f77-87b4-769493f93966\") " pod="openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27"
Sep 11 21:41:55.396792 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:55.396796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22829ef6-1c2d-4f77-87b4-769493f93966-lib-modules\") pod \"perf-node-gather-daemonset-v6r27\" (UID: \"22829ef6-1c2d-4f77-87b4-769493f93966\") " pod="openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27"
Sep 11 21:41:55.397037 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:55.396812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/22829ef6-1c2d-4f77-87b4-769493f93966-proc\") pod \"perf-node-gather-daemonset-v6r27\" (UID: \"22829ef6-1c2d-4f77-87b4-769493f93966\") " pod="openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27"
Sep 11 21:41:55.397037 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:55.396847    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/22829ef6-1c2d-4f77-87b4-769493f93966-podres\") pod \"perf-node-gather-daemonset-v6r27\" (UID: \"22829ef6-1c2d-4f77-87b4-769493f93966\") " pod="openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27"
Sep 11 21:41:55.397037 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:55.396877    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl84k\" (UniqueName: \"kubernetes.io/projected/22829ef6-1c2d-4f77-87b4-769493f93966-kube-api-access-gl84k\") pod \"perf-node-gather-daemonset-v6r27\" (UID: \"22829ef6-1c2d-4f77-87b4-769493f93966\") " pod="openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27"
Sep 11 21:41:55.397037 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:55.396883    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22829ef6-1c2d-4f77-87b4-769493f93966-sys\") pod \"perf-node-gather-daemonset-v6r27\" (UID: \"22829ef6-1c2d-4f77-87b4-769493f93966\") " pod="openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27"
Sep 11 21:41:55.397037 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:55.396903    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/22829ef6-1c2d-4f77-87b4-769493f93966-proc\") pod \"perf-node-gather-daemonset-v6r27\" (UID: \"22829ef6-1c2d-4f77-87b4-769493f93966\") " pod="openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27"
Sep 11 21:41:55.397037 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:55.396966    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22829ef6-1c2d-4f77-87b4-769493f93966-lib-modules\") pod \"perf-node-gather-daemonset-v6r27\" (UID: \"22829ef6-1c2d-4f77-87b4-769493f93966\") " pod="openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27"
Sep 11 21:41:55.397037 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:55.396996    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/22829ef6-1c2d-4f77-87b4-769493f93966-podres\") pod \"perf-node-gather-daemonset-v6r27\" (UID: \"22829ef6-1c2d-4f77-87b4-769493f93966\") " pod="openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27"
Sep 11 21:41:55.405002 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:55.404986    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl84k\" (UniqueName: \"kubernetes.io/projected/22829ef6-1c2d-4f77-87b4-769493f93966-kube-api-access-gl84k\") pod \"perf-node-gather-daemonset-v6r27\" (UID: \"22829ef6-1c2d-4f77-87b4-769493f93966\") " pod="openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27"
Sep 11 21:41:55.506578 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:55.506511    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27"
Sep 11 21:41:55.776452 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:55.776361    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27"]
Sep 11 21:41:55.779568 ip-10-0-142-224 kubenswrapper[2578]: W0911 21:41:55.779542    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod22829ef6_1c2d_4f77_87b4_769493f93966.slice/crio-f389c8d54291d16f530e284f33709eea056ffc4faa9141f60fda2cae54ea7ca8 WatchSource:0}: Error finding container f389c8d54291d16f530e284f33709eea056ffc4faa9141f60fda2cae54ea7ca8: Status 404 returned error can't find the container with id f389c8d54291d16f530e284f33709eea056ffc4faa9141f60fda2cae54ea7ca8
Sep 11 21:41:56.532390 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:56.532358    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27" event={"ID":"22829ef6-1c2d-4f77-87b4-769493f93966","Type":"ContainerStarted","Data":"e358e6868425f0149492c43ca1ca31a3cb70691fa4ac8cfee008b7351af55bed"}
Sep 11 21:41:56.534777 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:56.532403    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27" event={"ID":"22829ef6-1c2d-4f77-87b4-769493f93966","Type":"ContainerStarted","Data":"f389c8d54291d16f530e284f33709eea056ffc4faa9141f60fda2cae54ea7ca8"}
Sep 11 21:41:56.534777 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:56.532474    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27"
Sep 11 21:41:57.218220 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:57.218194    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hgf4n_a4135cd6-e9bc-45f9-978d-eead761ac9df/dns/0.log"
Sep 11 21:41:57.236658 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:57.236637    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hgf4n_a4135cd6-e9bc-45f9-978d-eead761ac9df/kube-rbac-proxy/0.log"
Sep 11 21:41:57.350655 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:57.350634    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-chmvt_7c9e9fa2-dc0c-4832-9841-fe2468faea5a/dns-node-resolver/0.log"
Sep 11 21:41:58.034506 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:58.034478    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2dd4w_5c8ceec8-5ea9-41ef-864c-c6e65ea6f483/node-ca/0.log"
Sep 11 21:41:59.376108 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:41:59.376076    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-677bd9b6b8-p99rw_6c8aa1b3-5d8a-4bce-ad61-b2b79c5c1a88/kube-auth-proxy/0.log"
Sep 11 21:42:00.426194 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:00.426161    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lhdrr_d2af5c92-e222-4927-8013-0764ee9a4e8d/serve-healthcheck-canary/0.log"
Sep 11 21:42:00.967102 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:00.967075    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c28km_e5cb0d0f-a841-4121-b7bf-d6d9628faded/kube-rbac-proxy/0.log"
Sep 11 21:42:00.990275 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:00.990253    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c28km_e5cb0d0f-a841-4121-b7bf-d6d9628faded/exporter/0.log"
Sep 11 21:42:01.011020 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:01.010997    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c28km_e5cb0d0f-a841-4121-b7bf-d6d9628faded/extractor/0.log"
Sep 11 21:42:02.545672 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:02.545646    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27"
Sep 11 21:42:02.562732 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:02.562687    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nrqhg/perf-node-gather-daemonset-v6r27" podStartSLOduration=7.562674124 podStartE2EDuration="7.562674124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:41:56.557947315 +0000 UTC m=+2183.261916603" watchObservedRunningTime="2026-09-11 21:42:02.562674124 +0000 UTC m=+2189.266643404"
Sep 11 21:42:03.727717 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:03.727685    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8457b95c77-b4xt2_d6d22dfa-1693-49d0-b4d6-087a181b3b57/manager/0.log"
Sep 11 21:42:03.744394 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:03.744372    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8457b95c77-b4xt2_d6d22dfa-1693-49d0-b4d6-087a181b3b57/copy-manifests/0.log"
Sep 11 21:42:05.025625 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:05.025600    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-65cc4f94dc-m467m_507d53e5-22b0-4a5a-99f4-ad9e39456e29/manager/0.log"
Sep 11 21:42:05.053681 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:05.053660    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-hnblw_d01dd788-065f-4a2d-bf22-bdefe71c6e3d/openshift-lws-operator/0.log"
Sep 11 21:42:05.182638 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:05.182605    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-677bd9b6b8-p99rw_6c8aa1b3-5d8a-4bce-ad61-b2b79c5c1a88/kube-auth-proxy/0.log"
Sep 11 21:42:07.247557 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:07.247530    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-677bd9b6b8-p99rw_6c8aa1b3-5d8a-4bce-ad61-b2b79c5c1a88/kube-auth-proxy/0.log"
Sep 11 21:42:10.415662 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:10.415636    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-mggdp_6f098d43-3053-4e83-ae6a-7c4bd9a09a97/migrator/0.log"
Sep 11 21:42:10.435370 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:10.435349    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-mggdp_6f098d43-3053-4e83-ae6a-7c4bd9a09a97/graceful-termination/0.log"
Sep 11 21:42:12.019141 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:12.019118    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sptwp_c15a91fd-755a-457c-aa8a-ac6b811025e4/kube-multus-additional-cni-plugins/0.log"
Sep 11 21:42:12.043433 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:12.043411    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sptwp_c15a91fd-755a-457c-aa8a-ac6b811025e4/egress-router-binary-copy/0.log"
Sep 11 21:42:12.068605 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:12.068580    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sptwp_c15a91fd-755a-457c-aa8a-ac6b811025e4/cni-plugins/0.log"
Sep 11 21:42:12.090948 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:12.090933    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sptwp_c15a91fd-755a-457c-aa8a-ac6b811025e4/bond-cni-plugin/0.log"
Sep 11 21:42:12.111893 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:12.111868    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sptwp_c15a91fd-755a-457c-aa8a-ac6b811025e4/routeoverride-cni/0.log"
Sep 11 21:42:12.132042 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:12.132024    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sptwp_c15a91fd-755a-457c-aa8a-ac6b811025e4/whereabouts-cni-bincopy/0.log"
Sep 11 21:42:12.150933 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:12.150914    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sptwp_c15a91fd-755a-457c-aa8a-ac6b811025e4/whereabouts-cni/0.log"
Sep 11 21:42:12.385482 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:12.385409    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ngfwx_674ba88c-8fb0-440e-b4e9-9ee972de3473/kube-multus/0.log"
Sep 11 21:42:12.458177 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:12.458144    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4gct2_03071acd-860e-4da8-b813-079c6af540b6/network-metrics-daemon/0.log"
Sep 11 21:42:12.474511 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:12.474489    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4gct2_03071acd-860e-4da8-b813-079c6af540b6/kube-rbac-proxy/0.log"
Sep 11 21:42:13.416568 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:13.416540    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hk624_8f5b49a6-5bb6-40bb-ba24-91886289da00/ovn-controller/0.log"
Sep 11 21:42:13.432133 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:13.432114    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hk624_8f5b49a6-5bb6-40bb-ba24-91886289da00/ovn-acl-logging/0.log"
Sep 11 21:42:13.454693 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:13.454676    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hk624_8f5b49a6-5bb6-40bb-ba24-91886289da00/kube-rbac-proxy-node/0.log"
Sep 11 21:42:13.496637 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:13.496617    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hk624_8f5b49a6-5bb6-40bb-ba24-91886289da00/kube-rbac-proxy-ovn-metrics/0.log"
Sep 11 21:42:13.516633 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:13.516608    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hk624_8f5b49a6-5bb6-40bb-ba24-91886289da00/northd/0.log"
Sep 11 21:42:13.536261 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:13.536242    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hk624_8f5b49a6-5bb6-40bb-ba24-91886289da00/nbdb/0.log"
Sep 11 21:42:13.556861 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:13.556843    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hk624_8f5b49a6-5bb6-40bb-ba24-91886289da00/sbdb/0.log"
Sep 11 21:42:13.611476 ip-10-0-142-224 kubenswrapper[2578]: I0911 21:42:13.611457    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hk624_8f5b49a6-5bb6-40bb-ba24-91886289da00/ovnkube-controller/0.log"