Sep 10 20:56:54.661246 ip-10-0-138-232 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 20:56:55.269058 ip-10-0-138-232 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 20:56:55.269058 ip-10-0-138-232 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 20:56:55.269058 ip-10-0-138-232 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 20:56:55.269058 ip-10-0-138-232 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 20:56:55.269058 ip-10-0-138-232 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 20:56:55.271826 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.271762    2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 20:56:55.276241 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276223    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:56:55.276241 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276238    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:56:55.276241 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276242    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:56:55.276241 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276245    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:56:55.276372 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276248    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:56:55.276372 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276251    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:56:55.276372 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276254    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:56:55.276372 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276256    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:56:55.276372 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276259    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:56:55.276372 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276262    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:56:55.276372 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276264    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:56:55.276372 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276267    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:56:55.276372 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276269    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:56:55.276372 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276272    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:56:55.276372 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276274    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:56:55.276372 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276277    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:56:55.276372 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276280    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:56:55.276372 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276283    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:56:55.276372 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276286    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:56:55.276372 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276288    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:56:55.276372 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276291    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:56:55.276372 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276294    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:56:55.276372 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276296    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:56:55.276372 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276299    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:56:55.276844 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276301    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:56:55.276844 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276304    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:56:55.276844 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276306    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:56:55.276844 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276309    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:56:55.276844 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276311    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:56:55.276844 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276314    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:56:55.276844 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276316    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:56:55.276844 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276319    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:56:55.276844 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276328    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:56:55.276844 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276330    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:56:55.276844 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276333    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:56:55.276844 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276335    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:56:55.276844 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276339    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:56:55.276844 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276343    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:56:55.276844 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276345    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:56:55.276844 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276348    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:56:55.276844 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276351    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:56:55.276844 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276353    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:56:55.276844 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276356    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:56:55.277328 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276358    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:56:55.277328 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276361    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:56:55.277328 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276364    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:56:55.277328 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276366    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:56:55.277328 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276369    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:56:55.277328 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276371    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:56:55.277328 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276374    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:56:55.277328 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276376    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:56:55.277328 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276379    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:56:55.277328 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276382    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:56:55.277328 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276385    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:56:55.277328 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276387    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:56:55.277328 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276390    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:56:55.277328 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276394    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:56:55.277328 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276398    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:56:55.277328 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276401    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:56:55.277328 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276403    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:56:55.277328 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276406    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:56:55.277328 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276409    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:56:55.277772 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276412    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:56:55.277772 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276415    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:56:55.277772 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276418    2572 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:56:55.277772 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276421    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:56:55.277772 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276423    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:56:55.277772 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276426    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:56:55.277772 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276429    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:56:55.277772 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276432    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:56:55.277772 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276435    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:56:55.277772 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276437    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:56:55.277772 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276440    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:56:55.277772 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276442    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:56:55.277772 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276445    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:56:55.277772 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276448    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:56:55.277772 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276452    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:56:55.277772 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276456    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:56:55.277772 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276459    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:56:55.277772 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276461    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:56:55.277772 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276465    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:56:55.277772 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276467    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:56:55.278270 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276470    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:56:55.278270 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276473    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:56:55.278270 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276475    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:56:55.278270 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276478    2572 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:56:55.278270 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276481    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:56:55.278270 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276483    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:56:55.278270 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276486    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:56:55.278270 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276488    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:56:55.278270 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276491    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:56:55.278270 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276495    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:56:55.278270 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276498    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:56:55.278270 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276501    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:56:55.278270 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276504    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:56:55.278270 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276507    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:56:55.278270 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276510    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:56:55.278270 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276513    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:56:55.278270 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.276515    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:56:55.278270 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277202    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:56:55.278270 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277210    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:56:55.278720 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277214    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:56:55.278720 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277217    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:56:55.278720 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277221    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:56:55.278720 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277224    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:56:55.278720 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277226    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:56:55.278720 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277230    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:56:55.278720 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277234    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:56:55.278720 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277237    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:56:55.278720 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277240    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:56:55.278720 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277243    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:56:55.278720 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277246    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:56:55.278720 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277249    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:56:55.278720 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277251    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:56:55.278720 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277254    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:56:55.278720 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277256    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:56:55.278720 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277259    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:56:55.278720 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277262    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:56:55.278720 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277265    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:56:55.278720 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277267    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:56:55.279214 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277270    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:56:55.279214 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277272    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:56:55.279214 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277275    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:56:55.279214 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277277    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:56:55.279214 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277280    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:56:55.279214 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277282    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:56:55.279214 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277285    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:56:55.279214 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277287    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:56:55.279214 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277290    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:56:55.279214 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277292    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:56:55.279214 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277294    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:56:55.279214 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277297    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:56:55.279214 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277300    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:56:55.279214 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277303    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:56:55.279214 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277305    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:56:55.279214 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277308    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:56:55.279214 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277310    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:56:55.279214 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277313    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:56:55.279214 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277315    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:56:55.279214 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277318    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:56:55.279715 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277320    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:56:55.279715 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277323    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:56:55.279715 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277325    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:56:55.279715 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277328    2572 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:56:55.279715 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277331    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:56:55.279715 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277335    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:56:55.279715 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277339    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:56:55.279715 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277342    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:56:55.279715 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277344    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:56:55.279715 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277348    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:56:55.279715 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277351    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:56:55.279715 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277354    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:56:55.279715 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277356    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:56:55.279715 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277359    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:56:55.279715 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277361    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:56:55.279715 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277363    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:56:55.279715 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277366    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:56:55.279715 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277369    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:56:55.279715 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277372    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:56:55.279715 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277374    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:56:55.280217 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277377    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:56:55.280217 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277379    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:56:55.280217 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277382    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:56:55.280217 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277384    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:56:55.280217 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277387    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:56:55.280217 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277390    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:56:55.280217 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277393    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:56:55.280217 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277396    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:56:55.280217 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277398    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:56:55.280217 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277402    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:56:55.280217 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277405    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:56:55.280217 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277407    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:56:55.280217 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277410    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:56:55.280217 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277412    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:56:55.280217 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277414    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:56:55.280217 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277417    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:56:55.280217 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277419    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:56:55.280217 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277422    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:56:55.280217 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277424    2572 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:56:55.280217 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277426    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:56:55.280705 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277429    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:56:55.280705 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277431    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:56:55.280705 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277434    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:56:55.280705 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277436    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:56:55.280705 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277439    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:56:55.280705 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277441    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:56:55.280705 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277444    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:56:55.280705 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277446    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:56:55.280705 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277450    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:56:55.280705 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277453    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:56:55.280705 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277456    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:56:55.280705 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277459    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:56:55.280705 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277461    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:56:55.280705 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277464    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:56:55.280705 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277467    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:56:55.280705 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277469    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:56:55.280705 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277474    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:56:55.280705 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277477    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:56:55.280705 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277863    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:56:55.280705 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277868    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:56:55.281238 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277871    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:56:55.281238 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277874    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:56:55.281238 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277877    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:56:55.281238 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277880    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:56:55.281238 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277883    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:56:55.281238 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277886    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:56:55.281238 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277888    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:56:55.281238 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277891    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:56:55.281238 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277893    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:56:55.281238 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277896    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:56:55.281238 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277898    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:56:55.281238 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277902    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:56:55.281238 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277906    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:56:55.281238 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277909    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:56:55.281238 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277912    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:56:55.281238 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277915    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:56:55.281238 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277917    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:56:55.281238 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277920    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:56:55.281238 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277923    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:56:55.281741 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277925    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:56:55.281741 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277928    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:56:55.281741 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277930    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:56:55.281741 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277933    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:56:55.281741 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277935    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:56:55.281741 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277937    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:56:55.281741 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277940    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:56:55.281741 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277943    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:56:55.281741 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277945    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:56:55.281741 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277949    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:56:55.281741 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277952    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:56:55.281741 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277954    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:56:55.281741 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277956    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:56:55.281741 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277960    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:56:55.281741 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277962    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:56:55.281741 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277965    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:56:55.281741 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277967    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:56:55.281741 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277970    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:56:55.281741 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277972    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:56:55.281741 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277974    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:56:55.282248 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277993    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:56:55.282248 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277996    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:56:55.282248 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.277999    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:56:55.282248 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278002    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:56:55.282248 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278006    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:56:55.282248 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278009    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:56:55.282248 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278012    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:56:55.282248 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278015    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:56:55.282248 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278018    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:56:55.282248 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278021    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:56:55.282248 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278024    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:56:55.282248 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278026    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:56:55.282248 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278029    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:56:55.282248 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278031    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:56:55.282248 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278033    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:56:55.282248 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278036    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:56:55.282248 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278038    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:56:55.282248 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278041    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:56:55.282248 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278043    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:56:55.282248 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278046    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:56:55.282763 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278049    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:56:55.282763 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278051    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:56:55.282763 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278054    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:56:55.282763 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278056    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:56:55.282763 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278059    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:56:55.282763 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278061    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:56:55.282763 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278064    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:56:55.282763 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278067    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:56:55.282763 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278069    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:56:55.282763 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278072    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:56:55.282763 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278074    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:56:55.282763 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278076    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:56:55.282763 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278079    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:56:55.282763 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278081    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:56:55.282763 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278084    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:56:55.282763 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278086    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:56:55.282763 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278089    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:56:55.282763 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278091    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:56:55.282763 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278095    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:56:55.283261 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278098    2572 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:56:55.283261 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278100    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:56:55.283261 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278111    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:56:55.283261 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278114    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:56:55.283261 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278116    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:56:55.283261 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278119    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:56:55.283261 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278121    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:56:55.283261 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278123    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:56:55.283261 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278126    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:56:55.283261 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278128    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:56:55.283261 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278131    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:56:55.283261 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278135    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:56:55.283261 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278138    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:56:55.283261 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278140    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:56:55.283261 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278143    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:56:55.283261 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278146    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:56:55.283261 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278149    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:56:55.283261 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278151    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:56:55.283261 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278154    2572 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:56:55.283261 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278220    2572 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 20:56:55.283744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278255    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 20:56:55.283744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278263    2572 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 20:56:55.283744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278267    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 20:56:55.283744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278271    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 20:56:55.283744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278275    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 20:56:55.283744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278279    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 20:56:55.283744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278283    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 20:56:55.283744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278287    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 20:56:55.283744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278289    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 20:56:55.283744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278293    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 20:56:55.283744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278296    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 20:56:55.283744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278299    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 20:56:55.283744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278301    2572 flags.go:64] FLAG: --cgroup-root=""
Sep 10 20:56:55.283744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278304    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 20:56:55.283744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278307    2572 flags.go:64] FLAG: --client-ca-file=""
Sep 10 20:56:55.283744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278310    2572 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 20:56:55.283744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278313    2572 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 20:56:55.283744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278317    2572 flags.go:64] FLAG: --cluster-domain=""
Sep 10 20:56:55.283744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278319    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 20:56:55.283744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278322    2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 20:56:55.283744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278325    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 20:56:55.283744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278328    2572 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 20:56:55.283744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278332    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 20:56:55.283744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278335    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278338    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278342    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278344    2572 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278347    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278350    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278353    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278355    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278359    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278362    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278365    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278367    2572 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278370    2572 flags.go:64] FLAG: --enable-server="true"
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278373    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278380    2572 flags.go:64] FLAG: --event-burst="100"
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278382    2572 flags.go:64] FLAG: --event-qps="50"
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278385    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278388    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278391    2572 flags.go:64] FLAG: --eviction-hard=""
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278394    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278397    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278399    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278402    2572 flags.go:64] FLAG: --eviction-soft=""
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278405    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278407    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 20:56:55.284343 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278416    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 20:56:55.284960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278420    2572 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 20:56:55.284960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278423    2572 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 20:56:55.284960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278426    2572 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 20:56:55.284960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278429    2572 flags.go:64] FLAG: --feature-gates=""
Sep 10 20:56:55.284960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278432    2572 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 20:56:55.284960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278435    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 20:56:55.284960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278438    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 20:56:55.284960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278441    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 20:56:55.284960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278444    2572 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 20:56:55.284960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278447    2572 flags.go:64] FLAG: --help="false"
Sep 10 20:56:55.284960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278449    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-138-232.ec2.internal"
Sep 10 20:56:55.284960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278453    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 20:56:55.284960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278455    2572 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 20:56:55.284960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278458    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 20:56:55.284960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278461    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 20:56:55.284960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278464    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 20:56:55.284960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278467    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 20:56:55.284960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278469    2572 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 20:56:55.284960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278473    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 20:56:55.284960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278475    2572 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 20:56:55.284960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278478    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 20:56:55.284960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278481    2572 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 20:56:55.284960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278483    2572 flags.go:64] FLAG: --kube-reserved=""
Sep 10 20:56:55.284960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278486    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278489    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278492    2572 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278494    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278497    2572 flags.go:64] FLAG: --lock-file=""
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278500    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278503    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278505    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278510    2572 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278515    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278518    2572 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278521    2572 flags.go:64] FLAG: --logging-format="text"
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278524    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278527    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278530    2572 flags.go:64] FLAG: --manifest-url=""
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278533    2572 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278537    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278540    2572 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278544    2572 flags.go:64] FLAG: --max-pods="110"
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278547    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278550    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278552    2572 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278555    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278558    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278561    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 20:56:55.285564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278564    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278570    2572 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278573    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278576    2572 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278579    2572 flags.go:64] FLAG: --pod-cidr=""
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278581    2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278587    2572 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278589    2572 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278592    2572 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278595    2572 flags.go:64] FLAG: --port="10250"
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278598    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278600    2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07e2054699ffec868"
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278603    2572 flags.go:64] FLAG: --qos-reserved=""
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278606    2572 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278609    2572 flags.go:64] FLAG: --register-node="true"
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278612    2572 flags.go:64] FLAG: --register-with-taints=""
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278616    2572 flags.go:64] FLAG: --registry-burst="10"
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278619    2572 flags.go:64] FLAG: --registry-qps="5"
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278622    2572 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278625    2572 flags.go:64] FLAG: --reserved-memory=""
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278628    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278631    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278634    2572 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278637    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278640    2572 flags.go:64] FLAG: --runonce="false"
Sep 10 20:56:55.286252 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278642    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278645    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278648    2572 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278651    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278654    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278656    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278659    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278662    2572 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278665    2572 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278667    2572 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278670    2572 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278673    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278675    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278678    2572 flags.go:64] FLAG: --system-cgroups=""
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278681    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278686    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278689    2572 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278691    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278697    2572 flags.go:64] FLAG: --tls-min-version=""
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278700    2572 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278703    2572 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278705    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278708    2572 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278713    2572 flags.go:64] FLAG: --v="2"
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278717    2572 flags.go:64] FLAG: --version="false"
Sep 10 20:56:55.286859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278722    2572 flags.go:64] FLAG: --vmodule=""
Sep 10 20:56:55.287481 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278725    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 20:56:55.287481 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.278728    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 20:56:55.287481 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278842    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:56:55.287481 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278845    2572 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:56:55.287481 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278851    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:56:55.287481 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278854    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:56:55.287481 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278856    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:56:55.287481 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278859    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:56:55.287481 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278861    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:56:55.287481 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278864    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:56:55.287481 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278866    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:56:55.287481 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278869    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:56:55.287481 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278871    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:56:55.287481 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278874    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:56:55.287481 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278876    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:56:55.287481 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278879    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:56:55.287481 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278881    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:56:55.287481 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278884    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:56:55.287481 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278887    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:56:55.287481 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278889    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:56:55.287996 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278891    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:56:55.287996 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278894    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:56:55.287996 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278897    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:56:55.287996 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278900    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:56:55.287996 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278902    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:56:55.287996 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278905    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:56:55.287996 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278908    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:56:55.287996 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278910    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:56:55.287996 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278913    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:56:55.287996 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278917    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:56:55.287996 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278919    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:56:55.287996 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278921    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:56:55.287996 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278925    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:56:55.287996 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278928    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:56:55.287996 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278930    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:56:55.287996 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278933    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:56:55.287996 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278936    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:56:55.287996 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278939    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:56:55.287996 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278941    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:56:55.287996 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278944    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:56:55.288479 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278946    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:56:55.288479 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278949    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:56:55.288479 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278952    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:56:55.288479 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278954    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:56:55.288479 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278957    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:56:55.288479 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278959    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:56:55.288479 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278963    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:56:55.288479 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278967    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:56:55.288479 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278970    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:56:55.288479 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278973    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:56:55.288479 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278976    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:56:55.288479 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278993    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:56:55.288479 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278996    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:56:55.288479 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.278998    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:56:55.288479 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279001    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:56:55.288479 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279004    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:56:55.288479 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279006    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:56:55.288479 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279009    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:56:55.288479 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279011    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:56:55.288959 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279013    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:56:55.288959 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279016    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:56:55.288959 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279020    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:56:55.288959 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279023    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:56:55.288959 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279025    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:56:55.288959 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279028    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:56:55.288959 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279031    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:56:55.288959 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279033    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:56:55.288959 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279036    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:56:55.288959 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279040    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:56:55.288959 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279042    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:56:55.288959 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279045    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:56:55.288959 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279047    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:56:55.288959 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279050    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:56:55.288959 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279053    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:56:55.288959 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279055    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:56:55.288959 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279059    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:56:55.288959 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279063    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:56:55.288959 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279065    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:56:55.289474 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279068    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:56:55.289474 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279071    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:56:55.289474 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279074    2572 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:56:55.289474 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279076    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:56:55.289474 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279079    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:56:55.289474 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279082    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:56:55.289474 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279085    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:56:55.289474 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279088    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:56:55.289474 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279091    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:56:55.289474 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279093    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:56:55.289474 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279096    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:56:55.289474 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279098    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:56:55.289474 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279101    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:56:55.289474 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279103    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:56:55.289474 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279106    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:56:55.289474 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279109    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:56:55.289474 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279112    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:56:55.289474 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279114    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:56:55.289474 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279117    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:56:55.289474 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279119    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:56:55.289949 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279122    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:56:55.289949 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279125    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:56:55.289949 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.279129    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:56:55.289949 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.280061    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 20:56:55.289949 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.287505    2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 20:56:55.289949 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.287520    2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 20:56:55.289949 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.288485    2572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 20:56:55.289949 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.288496    2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 20:56:55.289949 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288573    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:56:55.289949 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288579    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:56:55.289949 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288582    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:56:55.289949 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288585    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:56:55.289949 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288589    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:56:55.289949 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288591    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:56:55.289949 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288595    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:56:55.290331 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288598    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:56:55.290331 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288601    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:56:55.290331 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288603    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:56:55.290331 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288606    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:56:55.290331 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288609    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:56:55.290331 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288611    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:56:55.290331 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288614    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:56:55.290331 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288617    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:56:55.290331 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288619    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:56:55.290331 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288622    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:56:55.290331 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288624    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:56:55.290331 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288627    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:56:55.290331 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288630    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:56:55.290331 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288632    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:56:55.290331 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288635    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:56:55.290331 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288637    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:56:55.290331 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288640    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:56:55.290331 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288649    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:56:55.290331 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288653    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:56:55.290788 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288656    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:56:55.290788 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288659    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:56:55.290788 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288662    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:56:55.290788 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288665    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:56:55.290788 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288668    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:56:55.290788 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288670    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:56:55.290788 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288673    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:56:55.290788 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288675    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:56:55.290788 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288678    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:56:55.290788 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288681    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:56:55.290788 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288684    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:56:55.290788 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288688    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:56:55.290788 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288691    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:56:55.290788 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288693    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:56:55.290788 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288696    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:56:55.290788 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288699    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:56:55.290788 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288701    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:56:55.290788 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288704    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:56:55.290788 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288707    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:56:55.291299 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288709    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:56:55.291299 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288712    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:56:55.291299 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288715    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:56:55.291299 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288717    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:56:55.291299 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288720    2572 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:56:55.291299 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288722    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:56:55.291299 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288725    2572 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:56:55.291299 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288727    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:56:55.291299 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288730    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:56:55.291299 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288732    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:56:55.291299 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288735    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:56:55.291299 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288737    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:56:55.291299 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288745    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:56:55.291299 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288748    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:56:55.291299 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288750    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:56:55.291299 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288753    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:56:55.291299 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288756    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:56:55.291299 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288758    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:56:55.291299 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288761    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:56:55.291299 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288764    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:56:55.291777 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288767    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:56:55.291777 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288769    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:56:55.291777 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288772    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:56:55.291777 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288775    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:56:55.291777 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288777    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:56:55.291777 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288780    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:56:55.291777 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288782    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:56:55.291777 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288785    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:56:55.291777 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288787    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:56:55.291777 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288790    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:56:55.291777 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288792    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:56:55.291777 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288796    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:56:55.291777 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288799    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:56:55.291777 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288802    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:56:55.291777 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288805    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:56:55.291777 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288807    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:56:55.291777 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288810    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:56:55.291777 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288812    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:56:55.291777 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288815    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:56:55.291777 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288818    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:56:55.292283 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288821    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:56:55.292283 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288823    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:56:55.292283 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288826    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:56:55.292283 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288829    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:56:55.292283 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288831    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:56:55.292283 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288842    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:56:55.292283 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288845    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:56:55.292283 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288848    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:56:55.292283 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288851    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:56:55.292283 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288854    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:56:55.292283 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288857    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:56:55.292283 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288859    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:56:55.292283 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288861    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:56:55.292283 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.288864    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:56:55.292283 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.288869    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 20:56:55.292643 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289024    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:56:55.292643 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289031    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:56:55.292643 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289035    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:56:55.292643 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289038    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:56:55.292643 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289040    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:56:55.292643 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289043    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:56:55.292643 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289046    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:56:55.292643 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289048    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:56:55.292643 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289051    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:56:55.292643 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289053    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:56:55.292643 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289056    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:56:55.292643 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289058    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:56:55.292643 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289061    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:56:55.292643 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289063    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:56:55.292643 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289065    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:56:55.292643 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289068    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:56:55.292643 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289070    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:56:55.292643 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289073    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:56:55.292643 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289081    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:56:55.293178 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289084    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:56:55.293178 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289087    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:56:55.293178 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289089    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:56:55.293178 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289092    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:56:55.293178 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289099    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:56:55.293178 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289102    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:56:55.293178 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289104    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:56:55.293178 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289107    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:56:55.293178 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289110    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:56:55.293178 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289114    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:56:55.293178 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289117    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:56:55.293178 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289120    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:56:55.293178 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289123    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:56:55.293178 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289125    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:56:55.293178 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289128    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:56:55.293178 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289130    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:56:55.293178 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289133    2572 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:56:55.293178 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289135    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:56:55.293178 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289138    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:56:55.293178 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289140    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:56:55.293653 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289143    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:56:55.293653 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289145    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:56:55.293653 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289147    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:56:55.293653 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289150    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:56:55.293653 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289152    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:56:55.293653 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289154    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:56:55.293653 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289157    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:56:55.293653 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289159    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:56:55.293653 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289161    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:56:55.293653 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289164    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:56:55.293653 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289166    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:56:55.293653 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289169    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:56:55.293653 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289171    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:56:55.293653 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289174    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:56:55.293653 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289176    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:56:55.293653 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289179    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:56:55.293653 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289181    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:56:55.293653 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289189    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:56:55.293653 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289192    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:56:55.293653 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289194    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:56:55.294279 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289197    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:56:55.294279 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289199    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:56:55.294279 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289202    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:56:55.294279 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289204    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:56:55.294279 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289206    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:56:55.294279 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289209    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:56:55.294279 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289211    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:56:55.294279 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289213    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:56:55.294279 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289216    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:56:55.294279 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289218    2572 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:56:55.294279 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289221    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:56:55.294279 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289223    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:56:55.294279 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289225    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:56:55.294279 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289228    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:56:55.294279 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289230    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:56:55.294279 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289233    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:56:55.294279 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289235    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:56:55.294279 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289238    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:56:55.294279 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289240    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:56:55.294279 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289242    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:56:55.294757 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289245    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:56:55.294757 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289247    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:56:55.294757 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289249    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:56:55.294757 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289254    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:56:55.294757 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289257    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:56:55.294757 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289260    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:56:55.294757 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289262    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:56:55.294757 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289265    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:56:55.294757 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289268    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:56:55.294757 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289270    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:56:55.294757 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289278    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:56:55.294757 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289281    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:56:55.294757 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289284    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:56:55.294757 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289286    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:56:55.294757 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289288    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:56:55.294757 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289291    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:56:55.294757 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289293    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:56:55.294757 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289296    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:56:55.294757 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289298    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:56:55.294757 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:55.289300    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:56:55.295415 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.289305    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 20:56:55.295415 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.289391    2572 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 20:56:55.295415 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.292674    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 20:56:55.295415 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.293705    2572 server.go:1019] "Starting client certificate rotation"
Sep 10 20:56:55.295415 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.293859    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 20:56:55.295415 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.293905    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 20:56:55.296690 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.296677    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 20:56:55.300311 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.300284    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 20:56:55.315682 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.315662    2572 log.go:25] "Validated CRI v1 runtime API"
Sep 10 20:56:55.318909 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.318894    2572 log.go:25] "Validated CRI v1 image API"
Sep 10 20:56:55.319748 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.319730    2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 20:56:55.322088 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.322068    2572 fs.go:135] Filesystem UUIDs: map[6e2fa3e5-196b-4818-afb6-0c1832d967b1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8177f572-3a35-46a4-a7ec-993a7d712e70:/dev/nvme0n1p3]
Sep 10 20:56:55.322173 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.322088    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 20:56:55.327116 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.327005    2572 manager.go:216] Machine: {Timestamp:2026-09-10 20:56:55.325527373 +0000 UTC m=+0.537992284 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102457 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28eae12bc91a812b5bae3ebed288ea SystemUUID:ec28eae1-2bc9-1a81-2b5b-ae3ebed288ea BootID:d1388a55-6a2e-43ff-9f0c-a51fc05b056f Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e1:bd:ea:f7:6b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e1:bd:ea:f7:6b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:d2:84:bc:8b:89 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 20:56:55.327116 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.327105    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 20:56:55.327277 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.327202    2572 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 20:56:55.330720 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.330694    2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 20:56:55.330872 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.330722    2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-232.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 20:56:55.331541 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.331528    2572 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 20:56:55.331603 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.331546    2572 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 20:56:55.331603 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.331556    2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 20:56:55.332533 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.332522    2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 20:56:55.333689 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.333677    2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 20:56:55.337906 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.337893    2572 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 20:56:55.338042 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.338031    2572 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 20:56:55.338109 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.338074    2572 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 20:56:55.338109 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.338087    2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 20:56:55.338109 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.338102    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 20:56:55.338223 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.338114    2572 kubelet.go:398] "Adding apiserver pod source"
Sep 10 20:56:55.338223 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.338125    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 20:56:55.338456 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.338442    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 20:56:55.338512 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.338466    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 20:56:55.340825 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.340807    2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 20:56:55.343655 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.343641    2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 20:56:55.343720 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.343671    2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 20:56:55.345483 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.345472    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 20:56:55.345529 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.345488    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 20:56:55.345529 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.345509    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 20:56:55.345529 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.345515    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 20:56:55.345529 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.345523    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 20:56:55.345529 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.345528    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 20:56:55.345657 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.345541    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 20:56:55.345657 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.345546    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 20:56:55.345657 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.345559    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 20:56:55.345657 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.345564    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 20:56:55.345657 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.345578    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 20:56:55.345657 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.345583    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 20:56:55.346437 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.346427    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 20:56:55.346437 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.346436    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 20:56:55.350155 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.350137    2572 server.go:1268] "Started kubelet"
Sep 10 20:56:55.350682 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.350268    2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 20:56:55.350682 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.350257    2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 20:56:55.350682 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.350351    2572 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 20:56:55.350682 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.350489    2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-232.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 20:56:55.350788 ip-10-0-138-232 systemd[1]: Started Kubernetes Kubelet.
Sep 10 20:56:55.351779 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.351305    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-232.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 20:56:55.351779 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.351304    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 20:56:55.352089 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.352071    2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 20:56:55.352967 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.352866    2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 20:56:55.353056 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.352902    2572 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 20:56:55.354784 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.354759    2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 20:56:55.358900 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.358878    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 20:56:55.359113 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.358000    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-232.ec2.internal.18d41127f8e55c00  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-232.ec2.internal,UID:ip-10-0-138-232.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-232.ec2.internal,},FirstTimestamp:2026-09-10 20:56:55.350107136 +0000 UTC m=+0.562572046,LastTimestamp:2026-09-10 20:56:55.350107136 +0000 UTC m=+0.562572046,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-232.ec2.internal,}"
Sep 10 20:56:55.359576 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.359561    2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 20:56:55.359665 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.359647    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-232.ec2.internal\" not found"
Sep 10 20:56:55.360462 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.360390    2572 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 20:56:55.360462 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.360408    2572 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 20:56:55.360575 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.360417    2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 20:56:55.360575 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.360493    2572 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 20:56:55.360575 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.360532    2572 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 20:56:55.360745 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.360602    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 20:56:55.360745 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.360614    2572 factory.go:55] Registering systemd factory
Sep 10 20:56:55.360745 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.360621    2572 factory.go:223] Registration of the systemd container factory successfully
Sep 10 20:56:55.361119 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.361103    2572 factory.go:153] Registering CRI-O factory
Sep 10 20:56:55.361194 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.361122    2572 factory.go:223] Registration of the crio container factory successfully
Sep 10 20:56:55.361194 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.361139    2572 factory.go:103] Registering Raw factory
Sep 10 20:56:55.361194 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.361163    2572 manager.go:1196] Started watching for new ooms in manager
Sep 10 20:56:55.361561 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.361548    2572 manager.go:349] Starting recovery of all containers
Sep 10 20:56:55.362791 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.362768    2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-232.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 20:56:55.364012 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.363968    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 20:56:55.367494 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.367471    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 20:56:55.371187 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.371148    2572 manager.go:354] Recovery completed
Sep 10 20:56:55.375398 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.375381    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:56:55.378003 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.377969    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-232.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:56:55.378096 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.378015    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-232.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:56:55.378096 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.378025    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-232.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:56:55.378536 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.378522    2572 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 20:56:55.378536 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.378533    2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 20:56:55.378642 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.378538    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:56:55.378642 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.378547    2572 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 20:56:55.379282 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.379262    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-232.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:56:55.379336 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.379287    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-232.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:56:55.379336 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.379296    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-232.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:56:55.380340 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.380264    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-232.ec2.internal.18d41127fa8f000d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-232.ec2.internal,UID:ip-10-0-138-232.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-232.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-232.ec2.internal,},FirstTimestamp:2026-09-10 20:56:55.378001933 +0000 UTC m=+0.590466843,LastTimestamp:2026-09-10 20:56:55.378001933 +0000 UTC m=+0.590466843,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-232.ec2.internal,}"
Sep 10 20:56:55.380708 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.380694    2572 policy_none.go:49] "None policy: Start"
Sep 10 20:56:55.380753 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.380712    2572 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 20:56:55.380753 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.380722    2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 20:56:55.382664 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.382653    2572 policy_none.go:47] "Start"
Sep 10 20:56:55.399955 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.399894    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-232.ec2.internal.18d41127fa8f461d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-232.ec2.internal,UID:ip-10-0-138-232.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-232.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-232.ec2.internal,},FirstTimestamp:2026-09-10 20:56:55.378019869 +0000 UTC m=+0.590484780,LastTimestamp:2026-09-10 20:56:55.378019869 +0000 UTC m=+0.590484780,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-232.ec2.internal,}"
Sep 10 20:56:55.406414 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.406329    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-232.ec2.internal.18d41127fa8f68dc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-232.ec2.internal,UID:ip-10-0-138-232.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-232.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-232.ec2.internal,},FirstTimestamp:2026-09-10 20:56:55.378028764 +0000 UTC m=+0.590493674,LastTimestamp:2026-09-10 20:56:55.378028764 +0000 UTC m=+0.590493674,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-232.ec2.internal,}"
Sep 10 20:56:55.410621 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.410437    2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-232.ec2.internal.18d41127fa8f000d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-232.ec2.internal.18d41127fa8f000d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-232.ec2.internal,UID:ip-10-0-138-232.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-232.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-232.ec2.internal,},FirstTimestamp:2026-09-10 20:56:55.378001933 +0000 UTC m=+0.590466843,LastTimestamp:2026-09-10 20:56:55.379275052 +0000 UTC m=+0.591739962,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-232.ec2.internal,}"
Sep 10 20:56:55.421514 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.421447    2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-232.ec2.internal.18d41127fa8f461d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-232.ec2.internal.18d41127fa8f461d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-232.ec2.internal,UID:ip-10-0-138-232.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-232.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-232.ec2.internal,},FirstTimestamp:2026-09-10 20:56:55.378019869 +0000 UTC m=+0.590484780,LastTimestamp:2026-09-10 20:56:55.379291114 +0000 UTC m=+0.591756025,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-232.ec2.internal,}"
Sep 10 20:56:55.444376 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.421563    2572 manager.go:337] "Starting Device Plugin manager"
Sep 10 20:56:55.444376 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.421600    2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 20:56:55.444376 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.421609    2572 server.go:87] "Starting device plugin registration server"
Sep 10 20:56:55.444376 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.421955    2572 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 20:56:55.444376 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.421968    2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 20:56:55.444376 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.422074    2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 20:56:55.444376 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.422144    2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 20:56:55.444376 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.422152    2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 20:56:55.444376 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.422627    2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 20:56:55.444376 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.422681    2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-232.ec2.internal\" not found"
Sep 10 20:56:55.444376 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.431434    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 20:56:55.444376 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.431480    2572 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 20:56:55.444376 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.431496    2572 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 20:56:55.444376 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.431530    2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 20:56:55.444376 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.432308    2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-232.ec2.internal.18d41127fa8f68dc\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-232.ec2.internal.18d41127fa8f68dc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-232.ec2.internal,UID:ip-10-0-138-232.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-232.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-232.ec2.internal,},FirstTimestamp:2026-09-10 20:56:55.378028764 +0000 UTC m=+0.590493674,LastTimestamp:2026-09-10 20:56:55.379299647 +0000 UTC m=+0.591764559,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-232.ec2.internal,}"
Sep 10 20:56:55.445526 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.445468    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-232.ec2.internal.18d41127fd49388d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-232.ec2.internal,UID:ip-10-0-138-232.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-138-232.ec2.internal,},FirstTimestamp:2026-09-10 20:56:55.423760525 +0000 UTC m=+0.636225422,LastTimestamp:2026-09-10 20:56:55.423760525 +0000 UTC m=+0.636225422,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-232.ec2.internal,}"
Sep 10 20:56:55.445632 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.445591    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 10 20:56:55.479944 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.479924    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qj8jn"
Sep 10 20:56:55.495833 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.495814    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qj8jn"
Sep 10 20:56:55.522199 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.522182    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:56:55.522973 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.522957    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-232.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:56:55.523077 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.523009    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-232.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:56:55.523077 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.523026    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-232.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:56:55.523077 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.523053    2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-232.ec2.internal"
Sep 10 20:56:55.532366 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.532341    2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-232.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-232.ec2.internal"]
Sep 10 20:56:55.532460 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.532408    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:56:55.533174 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.533159    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-232.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:56:55.533257 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.533190    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-232.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:56:55.533257 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.533205    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-232.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:56:55.535382 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.535367    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:56:55.535505 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.535490    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-232.ec2.internal"
Sep 10 20:56:55.535579 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.535526    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:56:55.536081 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.536056    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-232.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:56:55.536160 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.536086    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-232.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:56:55.536160 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.536095    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-232.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:56:55.536160 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.536061    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-232.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:56:55.536250 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.536165    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-232.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:56:55.536250 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.536177    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-232.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:56:55.538255 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.538240    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-232.ec2.internal"
Sep 10 20:56:55.538309 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.538264    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:56:55.538869 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.538853    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-232.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:56:55.538931 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.538876    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-232.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:56:55.538931 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.538885    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-232.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:56:55.544251 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.544236    2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-232.ec2.internal"
Sep 10 20:56:55.544339 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.544258    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-232.ec2.internal\": node \"ip-10-0-138-232.ec2.internal\" not found"
Sep 10 20:56:55.560526 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.560506    2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-232.ec2.internal\" not found" node="ip-10-0-138-232.ec2.internal"
Sep 10 20:56:55.563908 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.563893    2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-232.ec2.internal\" not found" node="ip-10-0-138-232.ec2.internal"
Sep 10 20:56:55.579171 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.579154    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-232.ec2.internal\" not found"
Sep 10 20:56:55.662132 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.662083    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cfb73c088fb2994699656a729ac976af-config\") pod \"kube-apiserver-proxy-ip-10-0-138-232.ec2.internal\" (UID: \"cfb73c088fb2994699656a729ac976af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-232.ec2.internal"
Sep 10 20:56:55.662132 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.662106    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5251df6d6d13e895d3869fc5a6e732f6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-232.ec2.internal\" (UID: \"5251df6d6d13e895d3869fc5a6e732f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-232.ec2.internal"
Sep 10 20:56:55.662132 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.662126    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5251df6d6d13e895d3869fc5a6e732f6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-232.ec2.internal\" (UID: \"5251df6d6d13e895d3869fc5a6e732f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-232.ec2.internal"
Sep 10 20:56:55.680565 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.680544    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-232.ec2.internal\" not found"
Sep 10 20:56:55.763354 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.763323    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5251df6d6d13e895d3869fc5a6e732f6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-232.ec2.internal\" (UID: \"5251df6d6d13e895d3869fc5a6e732f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-232.ec2.internal"
Sep 10 20:56:55.763468 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.763353    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5251df6d6d13e895d3869fc5a6e732f6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-232.ec2.internal\" (UID: \"5251df6d6d13e895d3869fc5a6e732f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-232.ec2.internal"
Sep 10 20:56:55.763468 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.763384    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cfb73c088fb2994699656a729ac976af-config\") pod \"kube-apiserver-proxy-ip-10-0-138-232.ec2.internal\" (UID: \"cfb73c088fb2994699656a729ac976af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-232.ec2.internal"
Sep 10 20:56:55.763468 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.763449    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cfb73c088fb2994699656a729ac976af-config\") pod \"kube-apiserver-proxy-ip-10-0-138-232.ec2.internal\" (UID: \"cfb73c088fb2994699656a729ac976af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-232.ec2.internal"
Sep 10 20:56:55.763468 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.763457    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5251df6d6d13e895d3869fc5a6e732f6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-232.ec2.internal\" (UID: \"5251df6d6d13e895d3869fc5a6e732f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-232.ec2.internal"
Sep 10 20:56:55.763468 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.763449    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5251df6d6d13e895d3869fc5a6e732f6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-232.ec2.internal\" (UID: \"5251df6d6d13e895d3869fc5a6e732f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-232.ec2.internal"
Sep 10 20:56:55.781821 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.781795    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-232.ec2.internal\" not found"
Sep 10 20:56:55.864106 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.864080    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-232.ec2.internal"
Sep 10 20:56:55.867976 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:55.867952    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-232.ec2.internal"
Sep 10 20:56:55.881898 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.881874    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-232.ec2.internal\" not found"
Sep 10 20:56:55.983305 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:55.983245    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-232.ec2.internal\" not found"
Sep 10 20:56:56.083897 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:56.083873    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-232.ec2.internal\" not found"
Sep 10 20:56:56.184270 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:56.184244    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-232.ec2.internal\" not found"
Sep 10 20:56:56.227871 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.227848    2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:56:56.261147 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.261085    2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-232.ec2.internal"
Sep 10 20:56:56.276438 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.276417    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 20:56:56.278227 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.278209    2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-232.ec2.internal"
Sep 10 20:56:56.294053 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.294030    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 20:56:56.294176 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.294157    2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 20:56:56.294258 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:56.294223    2572 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://aac3db14753964129840f95d4045a205-233310ca75ee96f2.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kube-system/pods\": read tcp 10.0.138.232:46628->3.226.62.232:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-232.ec2.internal"
Sep 10 20:56:56.340923 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.340899    2572 apiserver.go:52] "Watching apiserver"
Sep 10 20:56:56.358637 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.358609    2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 20:56:56.359078 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.359047    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-p2jdt","openshift-multus/network-metrics-daemon-p9wtc","openshift-network-diagnostics/network-check-target-j6qrf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd","openshift-dns/node-resolver-v86hl","openshift-image-registry/node-ca-7jnhf","openshift-multus/multus-additional-cni-plugins-hwvmr","openshift-network-operator/iptables-alerter-5xlmr","openshift-ovn-kubernetes/ovnkube-node-r6fkc","kube-system/global-pull-secret-syncer-sm62m","kube-system/konnectivity-agent-jfw2v","kube-system/kube-apiserver-proxy-ip-10-0-138-232.ec2.internal","openshift-cluster-node-tuning-operator/tuned-5cngp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-232.ec2.internal"]
Sep 10 20:56:56.359184 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.359059    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 20:56:56.361710 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.361677    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hwvmr"
Sep 10 20:56:56.365147 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.365121    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.365288 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.365205    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 20:56:56.365881 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.365852    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 20:56:56.366669 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.366638    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 20:56:56.366806 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.366651    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97856763-2459-4493-9a2e-f34819281bc6-cni-binary-copy\") pod \"multus-additional-cni-plugins-hwvmr\" (UID: \"97856763-2459-4493-9a2e-f34819281bc6\") " pod="openshift-multus/multus-additional-cni-plugins-hwvmr"
Sep 10 20:56:56.366806 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.366700    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/97856763-2459-4493-9a2e-f34819281bc6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hwvmr\" (UID: \"97856763-2459-4493-9a2e-f34819281bc6\") " pod="openshift-multus/multus-additional-cni-plugins-hwvmr"
Sep 10 20:56:56.366806 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.366721    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/97856763-2459-4493-9a2e-f34819281bc6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hwvmr\" (UID: \"97856763-2459-4493-9a2e-f34819281bc6\") " pod="openshift-multus/multus-additional-cni-plugins-hwvmr"
Sep 10 20:56:56.366806 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.366742    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97856763-2459-4493-9a2e-f34819281bc6-system-cni-dir\") pod \"multus-additional-cni-plugins-hwvmr\" (UID: \"97856763-2459-4493-9a2e-f34819281bc6\") " pod="openshift-multus/multus-additional-cni-plugins-hwvmr"
Sep 10 20:56:56.366806 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.366761    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97856763-2459-4493-9a2e-f34819281bc6-cnibin\") pod \"multus-additional-cni-plugins-hwvmr\" (UID: \"97856763-2459-4493-9a2e-f34819281bc6\") " pod="openshift-multus/multus-additional-cni-plugins-hwvmr"
Sep 10 20:56:56.366806 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.366681    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 20:56:56.366806 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.366783    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97856763-2459-4493-9a2e-f34819281bc6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hwvmr\" (UID: \"97856763-2459-4493-9a2e-f34819281bc6\") " pod="openshift-multus/multus-additional-cni-plugins-hwvmr"
Sep 10 20:56:56.367122 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.366853    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dddgs\""
Sep 10 20:56:56.367122 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.366852    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c72w5\" (UniqueName: \"kubernetes.io/projected/97856763-2459-4493-9a2e-f34819281bc6-kube-api-access-c72w5\") pod \"multus-additional-cni-plugins-hwvmr\" (UID: \"97856763-2459-4493-9a2e-f34819281bc6\") " pod="openshift-multus/multus-additional-cni-plugins-hwvmr"
Sep 10 20:56:56.367480 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.367453    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:56:56.367669 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:56.367578    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6qrf" podUID="c3c750a9-6302-4266-9b0d-5a681ccdf76d"
Sep 10 20:56:56.367669 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.367653    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:56:56.367801 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.367681    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 20:56:56.368177 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.368146    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 20:56:56.368339 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.368315    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jzsrp\""
Sep 10 20:56:56.370025 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.369995    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd"
Sep 10 20:56:56.372235 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.372210    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 20:56:56.372537 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.372520    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v86hl"
Sep 10 20:56:56.373217 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.373192    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 20:56:56.373217 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.373212    2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 20:56:56.373556 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.373530    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 20:56:56.373677 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.373595    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cbr4f\""
Sep 10 20:56:56.375192 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.374781    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7jnhf"
Sep 10 20:56:56.376986 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.376940    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5kfnq\""
Sep 10 20:56:56.377139 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.377047    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 20:56:56.377450 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.377429    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.378313 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.378292    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 20:56:56.379288 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.379269    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 20:56:56.379372 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.379307    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 20:56:56.379372 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.379315    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 20:56:56.379625 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.379609    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2bmjc\""
Sep 10 20:56:56.379987 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.379969    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.380811 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.380788    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tcwhr\""
Sep 10 20:56:56.381121 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.381103    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 20:56:56.381258 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.381241    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 20:56:56.382594 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.382574    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:56:56.382675 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:56.382631    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9wtc" podUID="27f82b26-5156-4cbe-ac0c-23a386fa331b"
Sep 10 20:56:56.382903 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.382874    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 20:56:56.382903 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.382874    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 20:56:56.383097 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.382920    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 20:56:56.383097 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.382933    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 20:56:56.384765 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.384743    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-c4mkz\""
Sep 10 20:56:56.385021 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.384997    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 20:56:56.385267 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.385247    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:56:56.385360 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:56.385336    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm62m" podUID="f720dd2d-8c6a-4365-986a-68bbf39853cb"
Sep 10 20:56:56.387912 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.387889    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jfw2v"
Sep 10 20:56:56.390844 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.390818    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 20:56:56.390946 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.390821    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dh52p\""
Sep 10 20:56:56.390946 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.390915    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5xlmr"
Sep 10 20:56:56.391597 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.391579    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 20:56:56.395304 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.395265    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 20:56:56.395418 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.395334    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9fq9x\""
Sep 10 20:56:56.395718 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.395698    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:56:56.396029 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.395975    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 20:56:56.461388 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.461370    2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 20:56:56.463284 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.463266    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-k9bdz"
Sep 10 20:56:56.467474 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.467458    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-multus-cni-dir\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.467542 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.467484    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-host-run-k8s-cni-cncf-io\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.467542 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.467501    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-host-var-lib-cni-multus\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.467542 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.467517    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8873b843-8d37-4612-93c0-57eedc38ea18-multus-daemon-config\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.467542 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.467538    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97856763-2459-4493-9a2e-f34819281bc6-cni-binary-copy\") pod \"multus-additional-cni-plugins-hwvmr\" (UID: \"97856763-2459-4493-9a2e-f34819281bc6\") " pod="openshift-multus/multus-additional-cni-plugins-hwvmr"
Sep 10 20:56:56.467698 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.467553    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-cnibin\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.467698 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.467593    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b6a04231-4e0b-4a89-b8a0-276cc93d6a94-device-dir\") pod \"aws-ebs-csi-driver-node-822jd\" (UID: \"b6a04231-4e0b-4a89-b8a0-276cc93d6a94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd"
Sep 10 20:56:56.467698 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.467618    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/be238cba-4fb7-4176-9a0a-cd247ac53feb-serviceca\") pod \"node-ca-7jnhf\" (UID: \"be238cba-4fb7-4176-9a0a-cd247ac53feb\") " pod="openshift-image-registry/node-ca-7jnhf"
Sep 10 20:56:56.467698 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.467645    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f720dd2d-8c6a-4365-986a-68bbf39853cb-kubelet-config\") pod \"global-pull-secret-syncer-sm62m\" (UID: \"f720dd2d-8c6a-4365-986a-68bbf39853cb\") " pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:56:56.467698 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.467663    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmjcm\" (UniqueName: \"kubernetes.io/projected/33e1772c-e184-4b7d-89a8-05da5870b62c-kube-api-access-bmjcm\") pod \"node-resolver-v86hl\" (UID: \"33e1772c-e184-4b7d-89a8-05da5870b62c\") " pod="openshift-dns/node-resolver-v86hl"
Sep 10 20:56:56.467698 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.467678    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-etc-kubernetes\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.467993 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.467692    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-lib-modules\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.467993 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.467731    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4mlb\" (UniqueName: \"kubernetes.io/projected/9c335d4d-369f-4327-b77f-64a323fcefaa-kube-api-access-c4mlb\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.467993 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.467747    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-host-kubelet\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.467993 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.467786    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-run-openvswitch\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.467993 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.467819    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-run-ovn\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.467993 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.467845    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-host-cni-netd\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.467993 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.467872    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq4wp\" (UniqueName: \"kubernetes.io/projected/b9050703-03d5-49d4-8c74-dd1588f75210-kube-api-access-dq4wp\") pod \"iptables-alerter-5xlmr\" (UID: \"b9050703-03d5-49d4-8c74-dd1588f75210\") " pod="openshift-network-operator/iptables-alerter-5xlmr"
Sep 10 20:56:56.467993 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.467899    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-etc-sysconfig\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.467993 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.467931    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97856763-2459-4493-9a2e-f34819281bc6-cnibin\") pod \"multus-additional-cni-plugins-hwvmr\" (UID: \"97856763-2459-4493-9a2e-f34819281bc6\") " pod="openshift-multus/multus-additional-cni-plugins-hwvmr"
Sep 10 20:56:56.467993 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.467957    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c72w5\" (UniqueName: \"kubernetes.io/projected/97856763-2459-4493-9a2e-f34819281bc6-kube-api-access-c72w5\") pod \"multus-additional-cni-plugins-hwvmr\" (UID: \"97856763-2459-4493-9a2e-f34819281bc6\") " pod="openshift-multus/multus-additional-cni-plugins-hwvmr"
Sep 10 20:56:56.467993 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.467982    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-systemd-units\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.468403 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468015    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97856763-2459-4493-9a2e-f34819281bc6-cnibin\") pod \"multus-additional-cni-plugins-hwvmr\" (UID: \"97856763-2459-4493-9a2e-f34819281bc6\") " pod="openshift-multus/multus-additional-cni-plugins-hwvmr"
Sep 10 20:56:56.468403 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468020    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97856763-2459-4493-9a2e-f34819281bc6-cni-binary-copy\") pod \"multus-additional-cni-plugins-hwvmr\" (UID: \"97856763-2459-4493-9a2e-f34819281bc6\") " pod="openshift-multus/multus-additional-cni-plugins-hwvmr"
Sep 10 20:56:56.468403 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468031    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-host-run-netns\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.468403 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468107    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-node-log\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.468403 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468124    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phtfd\" (UniqueName: \"kubernetes.io/projected/86fbadd7-7507-494c-ba3f-0d90f598cf26-kube-api-access-phtfd\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.468403 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468147    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-etc-sysctl-d\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.468403 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468162    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-etc-systemd\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.468403 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468180    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/97856763-2459-4493-9a2e-f34819281bc6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hwvmr\" (UID: \"97856763-2459-4493-9a2e-f34819281bc6\") " pod="openshift-multus/multus-additional-cni-plugins-hwvmr"
Sep 10 20:56:56.468403 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468198    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-run-systemd\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.468403 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468219    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b9050703-03d5-49d4-8c74-dd1588f75210-iptables-alerter-script\") pod \"iptables-alerter-5xlmr\" (UID: \"b9050703-03d5-49d4-8c74-dd1588f75210\") " pod="openshift-network-operator/iptables-alerter-5xlmr"
Sep 10 20:56:56.468403 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468234    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33e1772c-e184-4b7d-89a8-05da5870b62c-hosts-file\") pod \"node-resolver-v86hl\" (UID: \"33e1772c-e184-4b7d-89a8-05da5870b62c\") " pod="openshift-dns/node-resolver-v86hl"
Sep 10 20:56:56.468403 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468249    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-multus-conf-dir\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.468403 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468268    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8rxp\" (UniqueName: \"kubernetes.io/projected/b6a04231-4e0b-4a89-b8a0-276cc93d6a94-kube-api-access-r8rxp\") pod \"aws-ebs-csi-driver-node-822jd\" (UID: \"b6a04231-4e0b-4a89-b8a0-276cc93d6a94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd"
Sep 10 20:56:56.468403 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468294    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97856763-2459-4493-9a2e-f34819281bc6-system-cni-dir\") pod \"multus-additional-cni-plugins-hwvmr\" (UID: \"97856763-2459-4493-9a2e-f34819281bc6\") " pod="openshift-multus/multus-additional-cni-plugins-hwvmr"
Sep 10 20:56:56.468403 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468320    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97856763-2459-4493-9a2e-f34819281bc6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hwvmr\" (UID: \"97856763-2459-4493-9a2e-f34819281bc6\") " pod="openshift-multus/multus-additional-cni-plugins-hwvmr"
Sep 10 20:56:56.468403 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468351    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97856763-2459-4493-9a2e-f34819281bc6-system-cni-dir\") pod \"multus-additional-cni-plugins-hwvmr\" (UID: \"97856763-2459-4493-9a2e-f34819281bc6\") " pod="openshift-multus/multus-additional-cni-plugins-hwvmr"
Sep 10 20:56:56.468972 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468394    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-var-lib-openvswitch\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.468972 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468433    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-log-socket\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.468972 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468466    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/86fbadd7-7507-494c-ba3f-0d90f598cf26-ovn-node-metrics-cert\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.468972 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468493    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/86fbadd7-7507-494c-ba3f-0d90f598cf26-ovnkube-script-lib\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.468972 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468503    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97856763-2459-4493-9a2e-f34819281bc6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hwvmr\" (UID: \"97856763-2459-4493-9a2e-f34819281bc6\") " pod="openshift-multus/multus-additional-cni-plugins-hwvmr"
Sep 10 20:56:56.468972 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468517    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-etc-modprobe-d\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.468972 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468546    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/97856763-2459-4493-9a2e-f34819281bc6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hwvmr\" (UID: \"97856763-2459-4493-9a2e-f34819281bc6\") " pod="openshift-multus/multus-additional-cni-plugins-hwvmr"
Sep 10 20:56:56.468972 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468566    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-etc-sysctl-conf\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.468972 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468580    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c335d4d-369f-4327-b77f-64a323fcefaa-tmp\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.468972 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468595    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e1772c-e184-4b7d-89a8-05da5870b62c-tmp-dir\") pod \"node-resolver-v86hl\" (UID: \"33e1772c-e184-4b7d-89a8-05da5870b62c\") " pod="openshift-dns/node-resolver-v86hl"
Sep 10 20:56:56.468972 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468619    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-multus-socket-dir-parent\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.468972 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468637    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/97856763-2459-4493-9a2e-f34819281bc6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hwvmr\" (UID: \"97856763-2459-4493-9a2e-f34819281bc6\") " pod="openshift-multus/multus-additional-cni-plugins-hwvmr"
Sep 10 20:56:56.468972 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468657    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4v6g\" (UniqueName: \"kubernetes.io/projected/c3c750a9-6302-4266-9b0d-5a681ccdf76d-kube-api-access-t4v6g\") pod \"network-check-target-j6qrf\" (UID: \"c3c750a9-6302-4266-9b0d-5a681ccdf76d\") " pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:56:56.468972 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468700    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-etc-openvswitch\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.468972 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468726    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-host\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.468972 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468751    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b6a04231-4e0b-4a89-b8a0-276cc93d6a94-kubelet-dir\") pod \"aws-ebs-csi-driver-node-822jd\" (UID: \"b6a04231-4e0b-4a89-b8a0-276cc93d6a94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd"
Sep 10 20:56:56.469490 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468781    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-host-slash\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.469490 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468821    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9050703-03d5-49d4-8c74-dd1588f75210-host-slash\") pod \"iptables-alerter-5xlmr\" (UID: \"b9050703-03d5-49d4-8c74-dd1588f75210\") " pod="openshift-network-operator/iptables-alerter-5xlmr"
Sep 10 20:56:56.469490 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468855    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-host-var-lib-cni-bin\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.469490 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468885    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-host-run-multus-certs\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.469490 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468910    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/218e6e53-1d6e-4c3a-adf3-4e9c2d9f9afa-konnectivity-ca\") pod \"konnectivity-agent-jfw2v\" (UID: \"218e6e53-1d6e-4c3a-adf3-4e9c2d9f9afa\") " pod="kube-system/konnectivity-agent-jfw2v"
Sep 10 20:56:56.469490 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468926    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b6a04231-4e0b-4a89-b8a0-276cc93d6a94-socket-dir\") pod \"aws-ebs-csi-driver-node-822jd\" (UID: \"b6a04231-4e0b-4a89-b8a0-276cc93d6a94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd"
Sep 10 20:56:56.469490 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468953    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b6a04231-4e0b-4a89-b8a0-276cc93d6a94-registration-dir\") pod \"aws-ebs-csi-driver-node-822jd\" (UID: \"b6a04231-4e0b-4a89-b8a0-276cc93d6a94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd"
Sep 10 20:56:56.469490 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.468958    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/97856763-2459-4493-9a2e-f34819281bc6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hwvmr\" (UID: \"97856763-2459-4493-9a2e-f34819281bc6\") " pod="openshift-multus/multus-additional-cni-plugins-hwvmr"
Sep 10 20:56:56.469490 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469010    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-host-cni-bin\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.469490 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469031    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-run\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.469490 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469046    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-system-cni-dir\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.469490 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469060    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27f82b26-5156-4cbe-ac0c-23a386fa331b-metrics-certs\") pod \"network-metrics-daemon-p9wtc\" (UID: \"27f82b26-5156-4cbe-ac0c-23a386fa331b\") " pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:56:56.469490 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469082    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-sys\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.469490 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469101    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f720dd2d-8c6a-4365-986a-68bbf39853cb-original-pull-secret\") pod \"global-pull-secret-syncer-sm62m\" (UID: \"f720dd2d-8c6a-4365-986a-68bbf39853cb\") " pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:56:56.469490 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469116    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-etc-kubernetes\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.469490 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469131    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/218e6e53-1d6e-4c3a-adf3-4e9c2d9f9afa-agent-certs\") pod \"konnectivity-agent-jfw2v\" (UID: \"218e6e53-1d6e-4c3a-adf3-4e9c2d9f9afa\") " pod="kube-system/konnectivity-agent-jfw2v"
Sep 10 20:56:56.469988 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469155    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-host-run-ovn-kubernetes\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.469988 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469175    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v6zc\" (UniqueName: \"kubernetes.io/projected/27f82b26-5156-4cbe-ac0c-23a386fa331b-kube-api-access-8v6zc\") pod \"network-metrics-daemon-p9wtc\" (UID: \"27f82b26-5156-4cbe-ac0c-23a386fa331b\") " pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:56:56.469988 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469189    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b6a04231-4e0b-4a89-b8a0-276cc93d6a94-etc-selinux\") pod \"aws-ebs-csi-driver-node-822jd\" (UID: \"b6a04231-4e0b-4a89-b8a0-276cc93d6a94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd"
Sep 10 20:56:56.469988 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469203    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/86fbadd7-7507-494c-ba3f-0d90f598cf26-ovnkube-config\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.469988 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469219    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/86fbadd7-7507-494c-ba3f-0d90f598cf26-env-overrides\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.469988 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469241    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be238cba-4fb7-4176-9a0a-cd247ac53feb-host\") pod \"node-ca-7jnhf\" (UID: \"be238cba-4fb7-4176-9a0a-cd247ac53feb\") " pod="openshift-image-registry/node-ca-7jnhf"
Sep 10 20:56:56.469988 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469264    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c335d4d-369f-4327-b77f-64a323fcefaa-etc-tuned\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.469988 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469282    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-host-run-netns\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.469988 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469297    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-host-var-lib-kubelet\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.469988 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469313    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-hostroot\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.469988 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469343    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b6a04231-4e0b-4a89-b8a0-276cc93d6a94-sys-fs\") pod \"aws-ebs-csi-driver-node-822jd\" (UID: \"b6a04231-4e0b-4a89-b8a0-276cc93d6a94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd"
Sep 10 20:56:56.469988 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469360    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.469988 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469376    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8873b843-8d37-4612-93c0-57eedc38ea18-cni-binary-copy\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.469988 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469392    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8f7d\" (UniqueName: \"kubernetes.io/projected/8873b843-8d37-4612-93c0-57eedc38ea18-kube-api-access-k8f7d\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.469988 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469430    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkhth\" (UniqueName: \"kubernetes.io/projected/be238cba-4fb7-4176-9a0a-cd247ac53feb-kube-api-access-bkhth\") pod \"node-ca-7jnhf\" (UID: \"be238cba-4fb7-4176-9a0a-cd247ac53feb\") " pod="openshift-image-registry/node-ca-7jnhf"
Sep 10 20:56:56.469988 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469455    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f720dd2d-8c6a-4365-986a-68bbf39853cb-dbus\") pod \"global-pull-secret-syncer-sm62m\" (UID: \"f720dd2d-8c6a-4365-986a-68bbf39853cb\") " pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:56:56.470490 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.469469    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-var-lib-kubelet\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.482458 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.482438    2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 20:56:56.485815 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.485794    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c72w5\" (UniqueName: \"kubernetes.io/projected/97856763-2459-4493-9a2e-f34819281bc6-kube-api-access-c72w5\") pod \"multus-additional-cni-plugins-hwvmr\" (UID: \"97856763-2459-4493-9a2e-f34819281bc6\") " pod="openshift-multus/multus-additional-cni-plugins-hwvmr"
Sep 10 20:56:56.488477 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.488459    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-k9bdz"
Sep 10 20:56:56.501586 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.501542    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 20:51:56 +0000 UTC" deadline="2028-06-05 09:56:53.805444246 +0000 UTC"
Sep 10 20:56:56.501586 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.501579    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15204h59m57.303867516s"
Sep 10 20:56:56.529349 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.529289    2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:56:56.570448 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570423    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b6a04231-4e0b-4a89-b8a0-276cc93d6a94-kubelet-dir\") pod \"aws-ebs-csi-driver-node-822jd\" (UID: \"b6a04231-4e0b-4a89-b8a0-276cc93d6a94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd"
Sep 10 20:56:56.570564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570464    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-host-slash\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.570564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570489    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9050703-03d5-49d4-8c74-dd1588f75210-host-slash\") pod \"iptables-alerter-5xlmr\" (UID: \"b9050703-03d5-49d4-8c74-dd1588f75210\") " pod="openshift-network-operator/iptables-alerter-5xlmr"
Sep 10 20:56:56.570564 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570515    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-host-var-lib-cni-bin\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.570714 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570563    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b6a04231-4e0b-4a89-b8a0-276cc93d6a94-kubelet-dir\") pod \"aws-ebs-csi-driver-node-822jd\" (UID: \"b6a04231-4e0b-4a89-b8a0-276cc93d6a94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd"
Sep 10 20:56:56.570714 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570576    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-host-var-lib-cni-bin\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.570714 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570593    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9050703-03d5-49d4-8c74-dd1588f75210-host-slash\") pod \"iptables-alerter-5xlmr\" (UID: \"b9050703-03d5-49d4-8c74-dd1588f75210\") " pod="openshift-network-operator/iptables-alerter-5xlmr"
Sep 10 20:56:56.570714 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570620    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-host-slash\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.570714 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570648    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-host-run-multus-certs\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.570714 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570674    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/218e6e53-1d6e-4c3a-adf3-4e9c2d9f9afa-konnectivity-ca\") pod \"konnectivity-agent-jfw2v\" (UID: \"218e6e53-1d6e-4c3a-adf3-4e9c2d9f9afa\") " pod="kube-system/konnectivity-agent-jfw2v"
Sep 10 20:56:56.570714 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570700    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b6a04231-4e0b-4a89-b8a0-276cc93d6a94-socket-dir\") pod \"aws-ebs-csi-driver-node-822jd\" (UID: \"b6a04231-4e0b-4a89-b8a0-276cc93d6a94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd"
Sep 10 20:56:56.570714 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570709    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-host-run-multus-certs\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.571128 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570725    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b6a04231-4e0b-4a89-b8a0-276cc93d6a94-registration-dir\") pod \"aws-ebs-csi-driver-node-822jd\" (UID: \"b6a04231-4e0b-4a89-b8a0-276cc93d6a94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd"
Sep 10 20:56:56.571128 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570754    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-host-cni-bin\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.571128 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570779    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-run\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.571128 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570803    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-system-cni-dir\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.571128 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570828    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27f82b26-5156-4cbe-ac0c-23a386fa331b-metrics-certs\") pod \"network-metrics-daemon-p9wtc\" (UID: \"27f82b26-5156-4cbe-ac0c-23a386fa331b\") " pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:56:56.571128 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570828    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b6a04231-4e0b-4a89-b8a0-276cc93d6a94-registration-dir\") pod \"aws-ebs-csi-driver-node-822jd\" (UID: \"b6a04231-4e0b-4a89-b8a0-276cc93d6a94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd"
Sep 10 20:56:56.571128 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570863    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b6a04231-4e0b-4a89-b8a0-276cc93d6a94-socket-dir\") pod \"aws-ebs-csi-driver-node-822jd\" (UID: \"b6a04231-4e0b-4a89-b8a0-276cc93d6a94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd"
Sep 10 20:56:56.571128 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570873    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-run\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.571128 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570915    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-sys\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.571128 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570923    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-host-cni-bin\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.571128 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570938    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-system-cni-dir\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.571128 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570945    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f720dd2d-8c6a-4365-986a-68bbf39853cb-original-pull-secret\") pod \"global-pull-secret-syncer-sm62m\" (UID: \"f720dd2d-8c6a-4365-986a-68bbf39853cb\") " pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:56:56.571128 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.570980    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-sys\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.571128 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:56.571059    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:56:56.571128 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571082    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-etc-kubernetes\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.571128 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571109    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/218e6e53-1d6e-4c3a-adf3-4e9c2d9f9afa-agent-certs\") pod \"konnectivity-agent-jfw2v\" (UID: \"218e6e53-1d6e-4c3a-adf3-4e9c2d9f9afa\") " pod="kube-system/konnectivity-agent-jfw2v"
Sep 10 20:56:56.571128 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:56.571129    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f720dd2d-8c6a-4365-986a-68bbf39853cb-original-pull-secret podName:f720dd2d-8c6a-4365-986a-68bbf39853cb nodeName:}" failed. No retries permitted until 2026-09-10 20:56:57.071108149 +0000 UTC m=+2.283573073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f720dd2d-8c6a-4365-986a-68bbf39853cb-original-pull-secret") pod "global-pull-secret-syncer-sm62m" (UID: "f720dd2d-8c6a-4365-986a-68bbf39853cb") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:56:56.571963 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571128    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-etc-kubernetes\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.571963 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:56.571063    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:56:56.571963 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571161    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-host-run-ovn-kubernetes\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.571963 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:56.571196    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27f82b26-5156-4cbe-ac0c-23a386fa331b-metrics-certs podName:27f82b26-5156-4cbe-ac0c-23a386fa331b nodeName:}" failed. No retries permitted until 2026-09-10 20:56:57.071178642 +0000 UTC m=+2.283643555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27f82b26-5156-4cbe-ac0c-23a386fa331b-metrics-certs") pod "network-metrics-daemon-p9wtc" (UID: "27f82b26-5156-4cbe-ac0c-23a386fa331b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:56:56.571963 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571214    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-host-run-ovn-kubernetes\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.571963 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571302    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/218e6e53-1d6e-4c3a-adf3-4e9c2d9f9afa-konnectivity-ca\") pod \"konnectivity-agent-jfw2v\" (UID: \"218e6e53-1d6e-4c3a-adf3-4e9c2d9f9afa\") " pod="kube-system/konnectivity-agent-jfw2v"
Sep 10 20:56:56.571963 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571217    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v6zc\" (UniqueName: \"kubernetes.io/projected/27f82b26-5156-4cbe-ac0c-23a386fa331b-kube-api-access-8v6zc\") pod \"network-metrics-daemon-p9wtc\" (UID: \"27f82b26-5156-4cbe-ac0c-23a386fa331b\") " pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:56:56.571963 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571364    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b6a04231-4e0b-4a89-b8a0-276cc93d6a94-etc-selinux\") pod \"aws-ebs-csi-driver-node-822jd\" (UID: \"b6a04231-4e0b-4a89-b8a0-276cc93d6a94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd"
Sep 10 20:56:56.571963 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571393    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/86fbadd7-7507-494c-ba3f-0d90f598cf26-ovnkube-config\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.571963 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571419    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/86fbadd7-7507-494c-ba3f-0d90f598cf26-env-overrides\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.571963 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571446    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be238cba-4fb7-4176-9a0a-cd247ac53feb-host\") pod \"node-ca-7jnhf\" (UID: \"be238cba-4fb7-4176-9a0a-cd247ac53feb\") " pod="openshift-image-registry/node-ca-7jnhf"
Sep 10 20:56:56.571963 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571473    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c335d4d-369f-4327-b77f-64a323fcefaa-etc-tuned\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.571963 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571500    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-host-run-netns\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.571963 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571507    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/be238cba-4fb7-4176-9a0a-cd247ac53feb-host\") pod \"node-ca-7jnhf\" (UID: \"be238cba-4fb7-4176-9a0a-cd247ac53feb\") " pod="openshift-image-registry/node-ca-7jnhf"
Sep 10 20:56:56.571963 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571471    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b6a04231-4e0b-4a89-b8a0-276cc93d6a94-etc-selinux\") pod \"aws-ebs-csi-driver-node-822jd\" (UID: \"b6a04231-4e0b-4a89-b8a0-276cc93d6a94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd"
Sep 10 20:56:56.571963 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571525    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-host-var-lib-kubelet\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.571963 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571554    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-hostroot\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.572856 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571580    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b6a04231-4e0b-4a89-b8a0-276cc93d6a94-sys-fs\") pod \"aws-ebs-csi-driver-node-822jd\" (UID: \"b6a04231-4e0b-4a89-b8a0-276cc93d6a94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd"
Sep 10 20:56:56.572856 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571650    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.572856 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571680    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8873b843-8d37-4612-93c0-57eedc38ea18-cni-binary-copy\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.572856 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571710    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8f7d\" (UniqueName: \"kubernetes.io/projected/8873b843-8d37-4612-93c0-57eedc38ea18-kube-api-access-k8f7d\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.572856 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571737    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkhth\" (UniqueName: \"kubernetes.io/projected/be238cba-4fb7-4176-9a0a-cd247ac53feb-kube-api-access-bkhth\") pod \"node-ca-7jnhf\" (UID: \"be238cba-4fb7-4176-9a0a-cd247ac53feb\") " pod="openshift-image-registry/node-ca-7jnhf"
Sep 10 20:56:56.572856 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571764    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f720dd2d-8c6a-4365-986a-68bbf39853cb-dbus\") pod \"global-pull-secret-syncer-sm62m\" (UID: \"f720dd2d-8c6a-4365-986a-68bbf39853cb\") " pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:56:56.572856 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571790    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-var-lib-kubelet\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.572856 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571818    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-multus-cni-dir\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.572856 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571845    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-host-run-k8s-cni-cncf-io\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.572856 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571870    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-host-var-lib-cni-multus\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.572856 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571896    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8873b843-8d37-4612-93c0-57eedc38ea18-multus-daemon-config\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.572856 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571920    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-cnibin\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.572856 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571946    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b6a04231-4e0b-4a89-b8a0-276cc93d6a94-device-dir\") pod \"aws-ebs-csi-driver-node-822jd\" (UID: \"b6a04231-4e0b-4a89-b8a0-276cc93d6a94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd"
Sep 10 20:56:56.572856 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571969    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/be238cba-4fb7-4176-9a0a-cd247ac53feb-serviceca\") pod \"node-ca-7jnhf\" (UID: \"be238cba-4fb7-4176-9a0a-cd247ac53feb\") " pod="openshift-image-registry/node-ca-7jnhf"
Sep 10 20:56:56.572856 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.571994    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f720dd2d-8c6a-4365-986a-68bbf39853cb-kubelet-config\") pod \"global-pull-secret-syncer-sm62m\" (UID: \"f720dd2d-8c6a-4365-986a-68bbf39853cb\") " pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:56:56.572856 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572022    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmjcm\" (UniqueName: \"kubernetes.io/projected/33e1772c-e184-4b7d-89a8-05da5870b62c-kube-api-access-bmjcm\") pod \"node-resolver-v86hl\" (UID: \"33e1772c-e184-4b7d-89a8-05da5870b62c\") " pod="openshift-dns/node-resolver-v86hl"
Sep 10 20:56:56.572856 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572052    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-etc-kubernetes\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.573904 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572073    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-host-run-netns\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.573904 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572098    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-lib-modules\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.573904 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572117    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-host-var-lib-kubelet\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.573904 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572123    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4mlb\" (UniqueName: \"kubernetes.io/projected/9c335d4d-369f-4327-b77f-64a323fcefaa-kube-api-access-c4mlb\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.573904 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572160    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-host-kubelet\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.573904 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572187    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-run-openvswitch\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.573904 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572202    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/86fbadd7-7507-494c-ba3f-0d90f598cf26-env-overrides\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.573904 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572215    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-run-ovn\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.573904 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572239    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/86fbadd7-7507-494c-ba3f-0d90f598cf26-ovnkube-config\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.573904 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572249    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-host-cni-netd\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.573904 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572257    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-run-ovn\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.573904 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572287    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq4wp\" (UniqueName: \"kubernetes.io/projected/b9050703-03d5-49d4-8c74-dd1588f75210-kube-api-access-dq4wp\") pod \"iptables-alerter-5xlmr\" (UID: \"b9050703-03d5-49d4-8c74-dd1588f75210\") " pod="openshift-network-operator/iptables-alerter-5xlmr"
Sep 10 20:56:56.573904 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572297    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-hostroot\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.573904 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572314    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-etc-sysconfig\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.573904 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572344    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-systemd-units\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.573904 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572349    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b6a04231-4e0b-4a89-b8a0-276cc93d6a94-sys-fs\") pod \"aws-ebs-csi-driver-node-822jd\" (UID: \"b6a04231-4e0b-4a89-b8a0-276cc93d6a94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd"
Sep 10 20:56:56.573904 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572371    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-host-run-netns\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.573904 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572400    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-node-log\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.574921 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572414    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f720dd2d-8c6a-4365-986a-68bbf39853cb-kubelet-config\") pod \"global-pull-secret-syncer-sm62m\" (UID: \"f720dd2d-8c6a-4365-986a-68bbf39853cb\") " pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:56:56.574921 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572426    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phtfd\" (UniqueName: \"kubernetes.io/projected/86fbadd7-7507-494c-ba3f-0d90f598cf26-kube-api-access-phtfd\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.574921 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572456    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-etc-sysctl-d\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.574921 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572481    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-etc-systemd\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.574921 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572509    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b6a04231-4e0b-4a89-b8a0-276cc93d6a94-device-dir\") pod \"aws-ebs-csi-driver-node-822jd\" (UID: \"b6a04231-4e0b-4a89-b8a0-276cc93d6a94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd"
Sep 10 20:56:56.574921 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572529    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-run-systemd\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.574921 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572557    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b9050703-03d5-49d4-8c74-dd1588f75210-iptables-alerter-script\") pod \"iptables-alerter-5xlmr\" (UID: \"b9050703-03d5-49d4-8c74-dd1588f75210\") " pod="openshift-network-operator/iptables-alerter-5xlmr"
Sep 10 20:56:56.574921 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572583    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f720dd2d-8c6a-4365-986a-68bbf39853cb-dbus\") pod \"global-pull-secret-syncer-sm62m\" (UID: \"f720dd2d-8c6a-4365-986a-68bbf39853cb\") " pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:56:56.574921 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572585    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33e1772c-e184-4b7d-89a8-05da5870b62c-hosts-file\") pod \"node-resolver-v86hl\" (UID: \"33e1772c-e184-4b7d-89a8-05da5870b62c\") " pod="openshift-dns/node-resolver-v86hl"
Sep 10 20:56:56.574921 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572624    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-multus-conf-dir\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.574921 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572638    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33e1772c-e184-4b7d-89a8-05da5870b62c-hosts-file\") pod \"node-resolver-v86hl\" (UID: \"33e1772c-e184-4b7d-89a8-05da5870b62c\") " pod="openshift-dns/node-resolver-v86hl"
Sep 10 20:56:56.574921 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572662    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8rxp\" (UniqueName: \"kubernetes.io/projected/b6a04231-4e0b-4a89-b8a0-276cc93d6a94-kube-api-access-r8rxp\") pod \"aws-ebs-csi-driver-node-822jd\" (UID: \"b6a04231-4e0b-4a89-b8a0-276cc93d6a94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd"
Sep 10 20:56:56.574921 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572697    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-var-lib-openvswitch\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.574921 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572705    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-host-cni-netd\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.574921 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572712    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/be238cba-4fb7-4176-9a0a-cd247ac53feb-serviceca\") pod \"node-ca-7jnhf\" (UID: \"be238cba-4fb7-4176-9a0a-cd247ac53feb\") " pod="openshift-image-registry/node-ca-7jnhf"
Sep 10 20:56:56.574921 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572725    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-log-socket\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.574921 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572762    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/86fbadd7-7507-494c-ba3f-0d90f598cf26-ovn-node-metrics-cert\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.575576 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572785    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-host-run-netns\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.575576 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572790    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/86fbadd7-7507-494c-ba3f-0d90f598cf26-ovnkube-script-lib\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.575576 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572823    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-etc-modprobe-d\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.575576 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572829    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-node-log\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.575576 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572849    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-lib-modules\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.575576 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572853    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-etc-sysctl-conf\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.575576 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572898    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c335d4d-369f-4327-b77f-64a323fcefaa-tmp\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.575576 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572906    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-var-lib-kubelet\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.575576 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572926    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e1772c-e184-4b7d-89a8-05da5870b62c-tmp-dir\") pod \"node-resolver-v86hl\" (UID: \"33e1772c-e184-4b7d-89a8-05da5870b62c\") " pod="openshift-dns/node-resolver-v86hl"
Sep 10 20:56:56.575576 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572971    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-run-openvswitch\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.575576 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572981    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-multus-socket-dir-parent\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.575576 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.573016    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4v6g\" (UniqueName: \"kubernetes.io/projected/c3c750a9-6302-4266-9b0d-5a681ccdf76d-kube-api-access-t4v6g\") pod \"network-check-target-j6qrf\" (UID: \"c3c750a9-6302-4266-9b0d-5a681ccdf76d\") " pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:56:56.575576 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.573006    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-run-systemd\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.575576 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.573043    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-etc-openvswitch\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.575576 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.573092    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-host\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.575576 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.573142    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-etc-sysctl-d\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.575576 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.573178    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-host\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.575576 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.573193    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-etc-systemd\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.576223 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.573215    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e1772c-e184-4b7d-89a8-05da5870b62c-tmp-dir\") pod \"node-resolver-v86hl\" (UID: \"33e1772c-e184-4b7d-89a8-05da5870b62c\") " pod="openshift-dns/node-resolver-v86hl"
Sep 10 20:56:56.576223 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.573234    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-host-run-k8s-cni-cncf-io\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.576223 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.573252    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-multus-cni-dir\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.576223 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.573285    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-host-var-lib-cni-multus\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.576223 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572788    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-systemd-units\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.576223 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.573373    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-etc-modprobe-d\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.576223 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.573480    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-etc-sysctl-conf\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.576223 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.572951    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c335d4d-369f-4327-b77f-64a323fcefaa-etc-sysconfig\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.576223 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.573500    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-host-kubelet\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.576223 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.573551    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-multus-socket-dir-parent\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.576223 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.573623    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b9050703-03d5-49d4-8c74-dd1588f75210-iptables-alerter-script\") pod \"iptables-alerter-5xlmr\" (UID: \"b9050703-03d5-49d4-8c74-dd1588f75210\") " pod="openshift-network-operator/iptables-alerter-5xlmr"
Sep 10 20:56:56.576223 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.573683    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-etc-kubernetes\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.576223 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.573802    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-etc-openvswitch\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.576223 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.573860    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-var-lib-openvswitch\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.576223 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.573927    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-log-socket\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.576223 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.574028    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8873b843-8d37-4612-93c0-57eedc38ea18-cni-binary-copy\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.576223 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.574143    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-cnibin\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.576223 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.574189    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86fbadd7-7507-494c-ba3f-0d90f598cf26-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.576839 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.574222    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8873b843-8d37-4612-93c0-57eedc38ea18-multus-daemon-config\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.576839 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.574233    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8873b843-8d37-4612-93c0-57eedc38ea18-multus-conf-dir\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.576839 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.574568    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/86fbadd7-7507-494c-ba3f-0d90f598cf26-ovnkube-script-lib\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.576839 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.574821    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/218e6e53-1d6e-4c3a-adf3-4e9c2d9f9afa-agent-certs\") pod \"konnectivity-agent-jfw2v\" (UID: \"218e6e53-1d6e-4c3a-adf3-4e9c2d9f9afa\") " pod="kube-system/konnectivity-agent-jfw2v"
Sep 10 20:56:56.576839 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.575579    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c335d4d-369f-4327-b77f-64a323fcefaa-etc-tuned\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.576839 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.576315    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c335d4d-369f-4327-b77f-64a323fcefaa-tmp\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.576839 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.576461    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/86fbadd7-7507-494c-ba3f-0d90f598cf26-ovn-node-metrics-cert\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.584149 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:56.584124    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5251df6d6d13e895d3869fc5a6e732f6.slice/crio-802b82f1479d18ea58d3b9e88d487277de4bfe4b327ba51d25477aeb97c0c7d5 WatchSource:0}: Error finding container 802b82f1479d18ea58d3b9e88d487277de4bfe4b327ba51d25477aeb97c0c7d5: Status 404 returned error can't find the container with id 802b82f1479d18ea58d3b9e88d487277de4bfe4b327ba51d25477aeb97c0c7d5
Sep 10 20:56:56.588716 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.588699    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 20:56:56.594656 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:56.594458    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb73c088fb2994699656a729ac976af.slice/crio-425cab2ab9a0fd85e3b6dc966e4a2542bd8ee6c499d20434898eec1abc5c6de8 WatchSource:0}: Error finding container 425cab2ab9a0fd85e3b6dc966e4a2542bd8ee6c499d20434898eec1abc5c6de8: Status 404 returned error can't find the container with id 425cab2ab9a0fd85e3b6dc966e4a2542bd8ee6c499d20434898eec1abc5c6de8
Sep 10 20:56:56.597006 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.596989    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmjcm\" (UniqueName: \"kubernetes.io/projected/33e1772c-e184-4b7d-89a8-05da5870b62c-kube-api-access-bmjcm\") pod \"node-resolver-v86hl\" (UID: \"33e1772c-e184-4b7d-89a8-05da5870b62c\") " pod="openshift-dns/node-resolver-v86hl"
Sep 10 20:56:56.599043 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.598999    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq4wp\" (UniqueName: \"kubernetes.io/projected/b9050703-03d5-49d4-8c74-dd1588f75210-kube-api-access-dq4wp\") pod \"iptables-alerter-5xlmr\" (UID: \"b9050703-03d5-49d4-8c74-dd1588f75210\") " pod="openshift-network-operator/iptables-alerter-5xlmr"
Sep 10 20:56:56.600941 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.600922    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v6zc\" (UniqueName: \"kubernetes.io/projected/27f82b26-5156-4cbe-ac0c-23a386fa331b-kube-api-access-8v6zc\") pod \"network-metrics-daemon-p9wtc\" (UID: \"27f82b26-5156-4cbe-ac0c-23a386fa331b\") " pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:56:56.602520 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.602481    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4mlb\" (UniqueName: \"kubernetes.io/projected/9c335d4d-369f-4327-b77f-64a323fcefaa-kube-api-access-c4mlb\") pod \"tuned-5cngp\" (UID: \"9c335d4d-369f-4327-b77f-64a323fcefaa\") " pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.612712 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.612693    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8rxp\" (UniqueName: \"kubernetes.io/projected/b6a04231-4e0b-4a89-b8a0-276cc93d6a94-kube-api-access-r8rxp\") pod \"aws-ebs-csi-driver-node-822jd\" (UID: \"b6a04231-4e0b-4a89-b8a0-276cc93d6a94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd"
Sep 10 20:56:56.621190 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.621170    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8f7d\" (UniqueName: \"kubernetes.io/projected/8873b843-8d37-4612-93c0-57eedc38ea18-kube-api-access-k8f7d\") pod \"multus-p2jdt\" (UID: \"8873b843-8d37-4612-93c0-57eedc38ea18\") " pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.622805 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:56.622788    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:56:56.622901 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:56.622811    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:56:56.622901 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:56.622824    2572 projected.go:196] Error preparing data for projected volume kube-api-access-t4v6g for pod openshift-network-diagnostics/network-check-target-j6qrf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:56:56.622901 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:56.622881    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3c750a9-6302-4266-9b0d-5a681ccdf76d-kube-api-access-t4v6g podName:c3c750a9-6302-4266-9b0d-5a681ccdf76d nodeName:}" failed. No retries permitted until 2026-09-10 20:56:57.122868595 +0000 UTC m=+2.335333493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t4v6g" (UniqueName: "kubernetes.io/projected/c3c750a9-6302-4266-9b0d-5a681ccdf76d-kube-api-access-t4v6g") pod "network-check-target-j6qrf" (UID: "c3c750a9-6302-4266-9b0d-5a681ccdf76d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:56:56.624880 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.624859    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phtfd\" (UniqueName: \"kubernetes.io/projected/86fbadd7-7507-494c-ba3f-0d90f598cf26-kube-api-access-phtfd\") pod \"ovnkube-node-r6fkc\" (UID: \"86fbadd7-7507-494c-ba3f-0d90f598cf26\") " pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.631793 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.631770    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkhth\" (UniqueName: \"kubernetes.io/projected/be238cba-4fb7-4176-9a0a-cd247ac53feb-kube-api-access-bkhth\") pod \"node-ca-7jnhf\" (UID: \"be238cba-4fb7-4176-9a0a-cd247ac53feb\") " pod="openshift-image-registry/node-ca-7jnhf"
Sep 10 20:56:56.670888 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.670862    2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:56:56.681141 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.681118    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hwvmr"
Sep 10 20:56:56.691412 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:56.691392    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97856763_2459_4493_9a2e_f34819281bc6.slice/crio-9f04adc1149695261534158f0231195aee4e57cd7a88cf0fd27c3a5501f6e810 WatchSource:0}: Error finding container 9f04adc1149695261534158f0231195aee4e57cd7a88cf0fd27c3a5501f6e810: Status 404 returned error can't find the container with id 9f04adc1149695261534158f0231195aee4e57cd7a88cf0fd27c3a5501f6e810
Sep 10 20:56:56.698103 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.698084    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5cngp"
Sep 10 20:56:56.706585 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:56.706564    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c335d4d_369f_4327_b77f_64a323fcefaa.slice/crio-f55102e9c47c4ce9bf91d07d7d510f628c58835857a9c2c4292934cc651b9f28 WatchSource:0}: Error finding container f55102e9c47c4ce9bf91d07d7d510f628c58835857a9c2c4292934cc651b9f28: Status 404 returned error can't find the container with id f55102e9c47c4ce9bf91d07d7d510f628c58835857a9c2c4292934cc651b9f28
Sep 10 20:56:56.722705 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.722682    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd"
Sep 10 20:56:56.725430 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.725411    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v86hl"
Sep 10 20:56:56.731990 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:56.731970    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a04231_4e0b_4a89_b8a0_276cc93d6a94.slice/crio-51a111a9732318f9bd25b51c0bfc250dc64beea7b9ae56d5e4c1eb5e7f6aeffa WatchSource:0}: Error finding container 51a111a9732318f9bd25b51c0bfc250dc64beea7b9ae56d5e4c1eb5e7f6aeffa: Status 404 returned error can't find the container with id 51a111a9732318f9bd25b51c0bfc250dc64beea7b9ae56d5e4c1eb5e7f6aeffa
Sep 10 20:56:56.735123 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:56.735102    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e1772c_e184_4b7d_89a8_05da5870b62c.slice/crio-72bea8e068f723fb8680892e3d5937f16161d7411f6cca84310be4cbfc970912 WatchSource:0}: Error finding container 72bea8e068f723fb8680892e3d5937f16161d7411f6cca84310be4cbfc970912: Status 404 returned error can't find the container with id 72bea8e068f723fb8680892e3d5937f16161d7411f6cca84310be4cbfc970912
Sep 10 20:56:56.750055 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.750038    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7jnhf"
Sep 10 20:56:56.759193 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.759175    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:56:56.759609 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:56.759592    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe238cba_4fb7_4176_9a0a_cd247ac53feb.slice/crio-99a54f3848884a70779cb9ba78fa5f97d13def1d378357653fc42dfa89675406 WatchSource:0}: Error finding container 99a54f3848884a70779cb9ba78fa5f97d13def1d378357653fc42dfa89675406: Status 404 returned error can't find the container with id 99a54f3848884a70779cb9ba78fa5f97d13def1d378357653fc42dfa89675406
Sep 10 20:56:56.766911 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.766891    2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:56:56.767506 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:56.767487    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86fbadd7_7507_494c_ba3f_0d90f598cf26.slice/crio-3ea24c2d47d98ffc4fd7d1ec45c7961831ec9ab06fc6f5b021ed59e2b267cdab WatchSource:0}: Error finding container 3ea24c2d47d98ffc4fd7d1ec45c7961831ec9ab06fc6f5b021ed59e2b267cdab: Status 404 returned error can't find the container with id 3ea24c2d47d98ffc4fd7d1ec45c7961831ec9ab06fc6f5b021ed59e2b267cdab
Sep 10 20:56:56.776613 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.776592    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p2jdt"
Sep 10 20:56:56.784824 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:56.784807    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8873b843_8d37_4612_93c0_57eedc38ea18.slice/crio-29209bccef0b28fc4ff4a9ab92fc3a894d90a3c8551f99a563672495c29492c1 WatchSource:0}: Error finding container 29209bccef0b28fc4ff4a9ab92fc3a894d90a3c8551f99a563672495c29492c1: Status 404 returned error can't find the container with id 29209bccef0b28fc4ff4a9ab92fc3a894d90a3c8551f99a563672495c29492c1
Sep 10 20:56:56.803222 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.803203    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jfw2v"
Sep 10 20:56:56.810670 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:56.810655    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5xlmr"
Sep 10 20:56:56.810939 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:56.810915    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218e6e53_1d6e_4c3a_adf3_4e9c2d9f9afa.slice/crio-1f458eef99569e5fe40d3220daa5bbb3bafc217d558b22bfd7fdac1a8a88bdf2 WatchSource:0}: Error finding container 1f458eef99569e5fe40d3220daa5bbb3bafc217d558b22bfd7fdac1a8a88bdf2: Status 404 returned error can't find the container with id 1f458eef99569e5fe40d3220daa5bbb3bafc217d558b22bfd7fdac1a8a88bdf2
Sep 10 20:56:56.820307 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:56:56.820289    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9050703_03d5_49d4_8c74_dd1588f75210.slice/crio-c8c8f34505aaaf4338fec3bf308f5e49bdc0fbe390ff81205993af535f30b903 WatchSource:0}: Error finding container c8c8f34505aaaf4338fec3bf308f5e49bdc0fbe390ff81205993af535f30b903: Status 404 returned error can't find the container with id c8c8f34505aaaf4338fec3bf308f5e49bdc0fbe390ff81205993af535f30b903
Sep 10 20:56:57.077448 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:57.077379    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27f82b26-5156-4cbe-ac0c-23a386fa331b-metrics-certs\") pod \"network-metrics-daemon-p9wtc\" (UID: \"27f82b26-5156-4cbe-ac0c-23a386fa331b\") " pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:56:57.077448 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:57.077429    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f720dd2d-8c6a-4365-986a-68bbf39853cb-original-pull-secret\") pod \"global-pull-secret-syncer-sm62m\" (UID: \"f720dd2d-8c6a-4365-986a-68bbf39853cb\") " pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:56:57.077607 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:57.077486    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:56:57.077607 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:57.077533    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:56:57.077607 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:57.077554    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27f82b26-5156-4cbe-ac0c-23a386fa331b-metrics-certs podName:27f82b26-5156-4cbe-ac0c-23a386fa331b nodeName:}" failed. No retries permitted until 2026-09-10 20:56:58.077534614 +0000 UTC m=+3.289999524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27f82b26-5156-4cbe-ac0c-23a386fa331b-metrics-certs") pod "network-metrics-daemon-p9wtc" (UID: "27f82b26-5156-4cbe-ac0c-23a386fa331b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:56:57.077607 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:57.077580    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f720dd2d-8c6a-4365-986a-68bbf39853cb-original-pull-secret podName:f720dd2d-8c6a-4365-986a-68bbf39853cb nodeName:}" failed. No retries permitted until 2026-09-10 20:56:58.07756584 +0000 UTC m=+3.290030755 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f720dd2d-8c6a-4365-986a-68bbf39853cb-original-pull-secret") pod "global-pull-secret-syncer-sm62m" (UID: "f720dd2d-8c6a-4365-986a-68bbf39853cb") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:56:57.179076 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:57.178508    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4v6g\" (UniqueName: \"kubernetes.io/projected/c3c750a9-6302-4266-9b0d-5a681ccdf76d-kube-api-access-t4v6g\") pod \"network-check-target-j6qrf\" (UID: \"c3c750a9-6302-4266-9b0d-5a681ccdf76d\") " pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:56:57.179076 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:57.178692    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:56:57.179076 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:57.178716    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:56:57.179076 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:57.178729    2572 projected.go:196] Error preparing data for projected volume kube-api-access-t4v6g for pod openshift-network-diagnostics/network-check-target-j6qrf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:56:57.179076 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:57.178782    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3c750a9-6302-4266-9b0d-5a681ccdf76d-kube-api-access-t4v6g podName:c3c750a9-6302-4266-9b0d-5a681ccdf76d nodeName:}" failed. No retries permitted until 2026-09-10 20:56:58.178762292 +0000 UTC m=+3.391227193 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t4v6g" (UniqueName: "kubernetes.io/projected/c3c750a9-6302-4266-9b0d-5a681ccdf76d-kube-api-access-t4v6g") pod "network-check-target-j6qrf" (UID: "c3c750a9-6302-4266-9b0d-5a681ccdf76d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:56:57.462025 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:57.461809    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5xlmr" event={"ID":"b9050703-03d5-49d4-8c74-dd1588f75210","Type":"ContainerStarted","Data":"c8c8f34505aaaf4338fec3bf308f5e49bdc0fbe390ff81205993af535f30b903"}
Sep 10 20:56:57.477253 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:57.477218    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd" event={"ID":"b6a04231-4e0b-4a89-b8a0-276cc93d6a94","Type":"ContainerStarted","Data":"51a111a9732318f9bd25b51c0bfc250dc64beea7b9ae56d5e4c1eb5e7f6aeffa"}
Sep 10 20:56:57.492595 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:57.492520    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 20:51:57 +0000 UTC" deadline="2028-06-14 15:06:12.875060342 +0000 UTC"
Sep 10 20:56:57.492595 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:57.492551    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15426h9m15.382514909s"
Sep 10 20:56:57.495151 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:57.495077    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-232.ec2.internal" event={"ID":"cfb73c088fb2994699656a729ac976af","Type":"ContainerStarted","Data":"425cab2ab9a0fd85e3b6dc966e4a2542bd8ee6c499d20434898eec1abc5c6de8"}
Sep 10 20:56:57.512139 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:57.512106    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-232.ec2.internal" event={"ID":"5251df6d6d13e895d3869fc5a6e732f6","Type":"ContainerStarted","Data":"802b82f1479d18ea58d3b9e88d487277de4bfe4b327ba51d25477aeb97c0c7d5"}
Sep 10 20:56:57.527227 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:57.527197    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jfw2v" event={"ID":"218e6e53-1d6e-4c3a-adf3-4e9c2d9f9afa","Type":"ContainerStarted","Data":"1f458eef99569e5fe40d3220daa5bbb3bafc217d558b22bfd7fdac1a8a88bdf2"}
Sep 10 20:56:57.529459 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:57.529433    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p2jdt" event={"ID":"8873b843-8d37-4612-93c0-57eedc38ea18","Type":"ContainerStarted","Data":"29209bccef0b28fc4ff4a9ab92fc3a894d90a3c8551f99a563672495c29492c1"}
Sep 10 20:56:57.541852 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:57.541810    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc" event={"ID":"86fbadd7-7507-494c-ba3f-0d90f598cf26","Type":"ContainerStarted","Data":"3ea24c2d47d98ffc4fd7d1ec45c7961831ec9ab06fc6f5b021ed59e2b267cdab"}
Sep 10 20:56:57.549701 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:57.548679    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7jnhf" event={"ID":"be238cba-4fb7-4176-9a0a-cd247ac53feb","Type":"ContainerStarted","Data":"99a54f3848884a70779cb9ba78fa5f97d13def1d378357653fc42dfa89675406"}
Sep 10 20:56:57.561304 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:57.561250    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v86hl" event={"ID":"33e1772c-e184-4b7d-89a8-05da5870b62c","Type":"ContainerStarted","Data":"72bea8e068f723fb8680892e3d5937f16161d7411f6cca84310be4cbfc970912"}
Sep 10 20:56:57.579080 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:57.578987    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5cngp" event={"ID":"9c335d4d-369f-4327-b77f-64a323fcefaa","Type":"ContainerStarted","Data":"f55102e9c47c4ce9bf91d07d7d510f628c58835857a9c2c4292934cc651b9f28"}
Sep 10 20:56:57.591889 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:57.591862    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwvmr" event={"ID":"97856763-2459-4493-9a2e-f34819281bc6","Type":"ContainerStarted","Data":"9f04adc1149695261534158f0231195aee4e57cd7a88cf0fd27c3a5501f6e810"}
Sep 10 20:56:58.086042 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:58.085989    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27f82b26-5156-4cbe-ac0c-23a386fa331b-metrics-certs\") pod \"network-metrics-daemon-p9wtc\" (UID: \"27f82b26-5156-4cbe-ac0c-23a386fa331b\") " pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:56:58.086042 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:58.086062    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f720dd2d-8c6a-4365-986a-68bbf39853cb-original-pull-secret\") pod \"global-pull-secret-syncer-sm62m\" (UID: \"f720dd2d-8c6a-4365-986a-68bbf39853cb\") " pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:56:58.086310 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:58.086224    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:56:58.086310 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:58.086288    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f720dd2d-8c6a-4365-986a-68bbf39853cb-original-pull-secret podName:f720dd2d-8c6a-4365-986a-68bbf39853cb nodeName:}" failed. No retries permitted until 2026-09-10 20:57:00.086268613 +0000 UTC m=+5.298733521 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f720dd2d-8c6a-4365-986a-68bbf39853cb-original-pull-secret") pod "global-pull-secret-syncer-sm62m" (UID: "f720dd2d-8c6a-4365-986a-68bbf39853cb") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:56:58.086760 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:58.086738    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:56:58.086840 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:58.086805    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27f82b26-5156-4cbe-ac0c-23a386fa331b-metrics-certs podName:27f82b26-5156-4cbe-ac0c-23a386fa331b nodeName:}" failed. No retries permitted until 2026-09-10 20:57:00.086787225 +0000 UTC m=+5.299252130 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27f82b26-5156-4cbe-ac0c-23a386fa331b-metrics-certs") pod "network-metrics-daemon-p9wtc" (UID: "27f82b26-5156-4cbe-ac0c-23a386fa331b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:56:58.186804 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:58.186761    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4v6g\" (UniqueName: \"kubernetes.io/projected/c3c750a9-6302-4266-9b0d-5a681ccdf76d-kube-api-access-t4v6g\") pod \"network-check-target-j6qrf\" (UID: \"c3c750a9-6302-4266-9b0d-5a681ccdf76d\") " pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:56:58.186981 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:58.186964    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:56:58.187095 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:58.186984    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:56:58.187095 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:58.186999    2572 projected.go:196] Error preparing data for projected volume kube-api-access-t4v6g for pod openshift-network-diagnostics/network-check-target-j6qrf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:56:58.187095 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:58.187076    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3c750a9-6302-4266-9b0d-5a681ccdf76d-kube-api-access-t4v6g podName:c3c750a9-6302-4266-9b0d-5a681ccdf76d nodeName:}" failed. No retries permitted until 2026-09-10 20:57:00.187057898 +0000 UTC m=+5.399522802 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t4v6g" (UniqueName: "kubernetes.io/projected/c3c750a9-6302-4266-9b0d-5a681ccdf76d-kube-api-access-t4v6g") pod "network-check-target-j6qrf" (UID: "c3c750a9-6302-4266-9b0d-5a681ccdf76d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:56:58.431975 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:58.431683    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:56:58.431975 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:58.431812    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6qrf" podUID="c3c750a9-6302-4266-9b0d-5a681ccdf76d"
Sep 10 20:56:58.432228 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:58.432126    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:56:58.432291 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:58.432126    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:56:58.432291 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:58.432243    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm62m" podUID="f720dd2d-8c6a-4365-986a-68bbf39853cb"
Sep 10 20:56:58.432394 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:56:58.432350    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9wtc" podUID="27f82b26-5156-4cbe-ac0c-23a386fa331b"
Sep 10 20:56:58.493208 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:58.493141    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 20:51:57 +0000 UTC" deadline="2028-04-25 23:41:37.616495922 +0000 UTC"
Sep 10 20:56:58.493208 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:58.493179    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14234h44m39.123321807s"
Sep 10 20:56:58.686412 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:56:58.686329    2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:57:00.102634 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:00.102403    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27f82b26-5156-4cbe-ac0c-23a386fa331b-metrics-certs\") pod \"network-metrics-daemon-p9wtc\" (UID: \"27f82b26-5156-4cbe-ac0c-23a386fa331b\") " pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:57:00.102634 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:00.102456    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f720dd2d-8c6a-4365-986a-68bbf39853cb-original-pull-secret\") pod \"global-pull-secret-syncer-sm62m\" (UID: \"f720dd2d-8c6a-4365-986a-68bbf39853cb\") " pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:57:00.103201 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:00.102687    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:00.103201 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:00.102758    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27f82b26-5156-4cbe-ac0c-23a386fa331b-metrics-certs podName:27f82b26-5156-4cbe-ac0c-23a386fa331b nodeName:}" failed. No retries permitted until 2026-09-10 20:57:04.102736585 +0000 UTC m=+9.315201496 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27f82b26-5156-4cbe-ac0c-23a386fa331b-metrics-certs") pod "network-metrics-daemon-p9wtc" (UID: "27f82b26-5156-4cbe-ac0c-23a386fa331b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:00.103201 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:00.103182    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:00.103374 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:00.103245    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f720dd2d-8c6a-4365-986a-68bbf39853cb-original-pull-secret podName:f720dd2d-8c6a-4365-986a-68bbf39853cb nodeName:}" failed. No retries permitted until 2026-09-10 20:57:04.103220508 +0000 UTC m=+9.315685410 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f720dd2d-8c6a-4365-986a-68bbf39853cb-original-pull-secret") pod "global-pull-secret-syncer-sm62m" (UID: "f720dd2d-8c6a-4365-986a-68bbf39853cb") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:00.202839 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:00.202803    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4v6g\" (UniqueName: \"kubernetes.io/projected/c3c750a9-6302-4266-9b0d-5a681ccdf76d-kube-api-access-t4v6g\") pod \"network-check-target-j6qrf\" (UID: \"c3c750a9-6302-4266-9b0d-5a681ccdf76d\") " pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:57:00.203077 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:00.203045    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:57:00.203077 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:00.203078    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:57:00.203209 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:00.203092    2572 projected.go:196] Error preparing data for projected volume kube-api-access-t4v6g for pod openshift-network-diagnostics/network-check-target-j6qrf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:00.203209 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:00.203146    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3c750a9-6302-4266-9b0d-5a681ccdf76d-kube-api-access-t4v6g podName:c3c750a9-6302-4266-9b0d-5a681ccdf76d nodeName:}" failed. No retries permitted until 2026-09-10 20:57:04.203127807 +0000 UTC m=+9.415592710 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t4v6g" (UniqueName: "kubernetes.io/projected/c3c750a9-6302-4266-9b0d-5a681ccdf76d-kube-api-access-t4v6g") pod "network-check-target-j6qrf" (UID: "c3c750a9-6302-4266-9b0d-5a681ccdf76d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:00.432128 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:00.432041    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:57:00.432298 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:00.432193    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9wtc" podUID="27f82b26-5156-4cbe-ac0c-23a386fa331b"
Sep 10 20:57:00.432603 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:00.432578    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:57:00.433476 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:00.432692    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6qrf" podUID="c3c750a9-6302-4266-9b0d-5a681ccdf76d"
Sep 10 20:57:00.433476 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:00.432758    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:57:00.433476 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:00.432829    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm62m" podUID="f720dd2d-8c6a-4365-986a-68bbf39853cb"
Sep 10 20:57:02.432676 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:02.432166    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:57:02.432676 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:02.432172    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:57:02.432676 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:02.432322    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9wtc" podUID="27f82b26-5156-4cbe-ac0c-23a386fa331b"
Sep 10 20:57:02.432676 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:02.432407    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm62m" podUID="f720dd2d-8c6a-4365-986a-68bbf39853cb"
Sep 10 20:57:02.432676 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:02.432423    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:57:02.432676 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:02.432528    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6qrf" podUID="c3c750a9-6302-4266-9b0d-5a681ccdf76d"
Sep 10 20:57:04.135676 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:04.135019    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27f82b26-5156-4cbe-ac0c-23a386fa331b-metrics-certs\") pod \"network-metrics-daemon-p9wtc\" (UID: \"27f82b26-5156-4cbe-ac0c-23a386fa331b\") " pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:57:04.135676 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:04.135068    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f720dd2d-8c6a-4365-986a-68bbf39853cb-original-pull-secret\") pod \"global-pull-secret-syncer-sm62m\" (UID: \"f720dd2d-8c6a-4365-986a-68bbf39853cb\") " pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:57:04.135676 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:04.135172    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:04.135676 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:04.135211    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:04.135676 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:04.135250    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27f82b26-5156-4cbe-ac0c-23a386fa331b-metrics-certs podName:27f82b26-5156-4cbe-ac0c-23a386fa331b nodeName:}" failed. No retries permitted until 2026-09-10 20:57:12.135227039 +0000 UTC m=+17.347691949 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27f82b26-5156-4cbe-ac0c-23a386fa331b-metrics-certs") pod "network-metrics-daemon-p9wtc" (UID: "27f82b26-5156-4cbe-ac0c-23a386fa331b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:04.135676 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:04.135273    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f720dd2d-8c6a-4365-986a-68bbf39853cb-original-pull-secret podName:f720dd2d-8c6a-4365-986a-68bbf39853cb nodeName:}" failed. No retries permitted until 2026-09-10 20:57:12.135262985 +0000 UTC m=+17.347727887 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f720dd2d-8c6a-4365-986a-68bbf39853cb-original-pull-secret") pod "global-pull-secret-syncer-sm62m" (UID: "f720dd2d-8c6a-4365-986a-68bbf39853cb") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:04.236472 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:04.236442    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4v6g\" (UniqueName: \"kubernetes.io/projected/c3c750a9-6302-4266-9b0d-5a681ccdf76d-kube-api-access-t4v6g\") pod \"network-check-target-j6qrf\" (UID: \"c3c750a9-6302-4266-9b0d-5a681ccdf76d\") " pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:57:04.236641 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:04.236590    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:57:04.236641 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:04.236613    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:57:04.236641 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:04.236627    2572 projected.go:196] Error preparing data for projected volume kube-api-access-t4v6g for pod openshift-network-diagnostics/network-check-target-j6qrf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:04.236817 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:04.236695    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3c750a9-6302-4266-9b0d-5a681ccdf76d-kube-api-access-t4v6g podName:c3c750a9-6302-4266-9b0d-5a681ccdf76d nodeName:}" failed. No retries permitted until 2026-09-10 20:57:12.236676158 +0000 UTC m=+17.449141072 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t4v6g" (UniqueName: "kubernetes.io/projected/c3c750a9-6302-4266-9b0d-5a681ccdf76d-kube-api-access-t4v6g") pod "network-check-target-j6qrf" (UID: "c3c750a9-6302-4266-9b0d-5a681ccdf76d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:04.432658 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:04.432577    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:57:04.432658 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:04.432631    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:57:04.432884 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:04.432732    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9wtc" podUID="27f82b26-5156-4cbe-ac0c-23a386fa331b"
Sep 10 20:57:04.432884 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:04.432783    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6qrf" podUID="c3c750a9-6302-4266-9b0d-5a681ccdf76d"
Sep 10 20:57:04.432884 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:04.432833    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:57:04.433082 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:04.432903    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm62m" podUID="f720dd2d-8c6a-4365-986a-68bbf39853cb"
Sep 10 20:57:06.432995 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:06.432465    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:57:06.432995 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:06.432601    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6qrf" podUID="c3c750a9-6302-4266-9b0d-5a681ccdf76d"
Sep 10 20:57:06.432995 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:06.432702    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:57:06.432995 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:06.432787    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm62m" podUID="f720dd2d-8c6a-4365-986a-68bbf39853cb"
Sep 10 20:57:06.432995 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:06.432850    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:57:06.432995 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:06.432952    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9wtc" podUID="27f82b26-5156-4cbe-ac0c-23a386fa331b"
Sep 10 20:57:08.431716 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:08.431684    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:57:08.432172 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:08.431777    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:57:08.432172 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:08.431780    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6qrf" podUID="c3c750a9-6302-4266-9b0d-5a681ccdf76d"
Sep 10 20:57:08.432172 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:08.431894    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm62m" podUID="f720dd2d-8c6a-4365-986a-68bbf39853cb"
Sep 10 20:57:08.432172 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:08.431945    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:57:08.432172 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:08.432059    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9wtc" podUID="27f82b26-5156-4cbe-ac0c-23a386fa331b"
Sep 10 20:57:10.432744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:10.432708    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:57:10.432744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:10.432731    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:57:10.433239 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:10.432820    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:57:10.433239 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:10.432830    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9wtc" podUID="27f82b26-5156-4cbe-ac0c-23a386fa331b"
Sep 10 20:57:10.433239 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:10.432877    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6qrf" podUID="c3c750a9-6302-4266-9b0d-5a681ccdf76d"
Sep 10 20:57:10.433239 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:10.432947    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm62m" podUID="f720dd2d-8c6a-4365-986a-68bbf39853cb"
Sep 10 20:57:12.194681 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:12.194648    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27f82b26-5156-4cbe-ac0c-23a386fa331b-metrics-certs\") pod \"network-metrics-daemon-p9wtc\" (UID: \"27f82b26-5156-4cbe-ac0c-23a386fa331b\") " pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:57:12.195189 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:12.194688    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f720dd2d-8c6a-4365-986a-68bbf39853cb-original-pull-secret\") pod \"global-pull-secret-syncer-sm62m\" (UID: \"f720dd2d-8c6a-4365-986a-68bbf39853cb\") " pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:57:12.195189 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:12.194797    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:12.195189 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:12.194798    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:12.195189 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:12.194852    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f720dd2d-8c6a-4365-986a-68bbf39853cb-original-pull-secret podName:f720dd2d-8c6a-4365-986a-68bbf39853cb nodeName:}" failed. No retries permitted until 2026-09-10 20:57:28.194837567 +0000 UTC m=+33.407302485 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f720dd2d-8c6a-4365-986a-68bbf39853cb-original-pull-secret") pod "global-pull-secret-syncer-sm62m" (UID: "f720dd2d-8c6a-4365-986a-68bbf39853cb") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:12.195189 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:12.194868    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27f82b26-5156-4cbe-ac0c-23a386fa331b-metrics-certs podName:27f82b26-5156-4cbe-ac0c-23a386fa331b nodeName:}" failed. No retries permitted until 2026-09-10 20:57:28.194860559 +0000 UTC m=+33.407325455 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27f82b26-5156-4cbe-ac0c-23a386fa331b-metrics-certs") pod "network-metrics-daemon-p9wtc" (UID: "27f82b26-5156-4cbe-ac0c-23a386fa331b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:12.295154 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:12.295116    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4v6g\" (UniqueName: \"kubernetes.io/projected/c3c750a9-6302-4266-9b0d-5a681ccdf76d-kube-api-access-t4v6g\") pod \"network-check-target-j6qrf\" (UID: \"c3c750a9-6302-4266-9b0d-5a681ccdf76d\") " pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:57:12.295336 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:12.295260    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:57:12.295336 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:12.295275    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:57:12.295336 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:12.295283    2572 projected.go:196] Error preparing data for projected volume kube-api-access-t4v6g for pod openshift-network-diagnostics/network-check-target-j6qrf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:12.295336 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:12.295330    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3c750a9-6302-4266-9b0d-5a681ccdf76d-kube-api-access-t4v6g podName:c3c750a9-6302-4266-9b0d-5a681ccdf76d nodeName:}" failed. No retries permitted until 2026-09-10 20:57:28.29531331 +0000 UTC m=+33.507778207 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t4v6g" (UniqueName: "kubernetes.io/projected/c3c750a9-6302-4266-9b0d-5a681ccdf76d-kube-api-access-t4v6g") pod "network-check-target-j6qrf" (UID: "c3c750a9-6302-4266-9b0d-5a681ccdf76d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:12.431798 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:12.431763    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:57:12.431955 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:12.431892    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm62m" podUID="f720dd2d-8c6a-4365-986a-68bbf39853cb"
Sep 10 20:57:12.432308 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:12.432277    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:57:12.432427 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:12.432385    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9wtc" podUID="27f82b26-5156-4cbe-ac0c-23a386fa331b"
Sep 10 20:57:12.432493 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:12.432437    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:57:12.432546 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:12.432507    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6qrf" podUID="c3c750a9-6302-4266-9b0d-5a681ccdf76d"
Sep 10 20:57:14.432630 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:14.432355    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:57:14.432630 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:14.432395    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:57:14.432630 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:14.432471    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6qrf" podUID="c3c750a9-6302-4266-9b0d-5a681ccdf76d"
Sep 10 20:57:14.432630 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:14.432546    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm62m" podUID="f720dd2d-8c6a-4365-986a-68bbf39853cb"
Sep 10 20:57:14.432630 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:14.432582    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:57:14.433237 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:14.432685    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9wtc" podUID="27f82b26-5156-4cbe-ac0c-23a386fa331b"
Sep 10 20:57:16.432623 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:16.432587    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:57:16.433077 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:16.432701    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:57:16.433077 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:16.432706    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:57:16.433077 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:16.432806    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6qrf" podUID="c3c750a9-6302-4266-9b0d-5a681ccdf76d"
Sep 10 20:57:16.433077 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:16.432881    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9wtc" podUID="27f82b26-5156-4cbe-ac0c-23a386fa331b"
Sep 10 20:57:16.433077 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:16.432704    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm62m" podUID="f720dd2d-8c6a-4365-986a-68bbf39853cb"
Sep 10 20:57:17.653489 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:17.653290    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc" event={"ID":"86fbadd7-7507-494c-ba3f-0d90f598cf26","Type":"ContainerStarted","Data":"72384a88d3878261bffb15cd036b8a63e0e1ae2e2fc4a884e0e8e532d7e8f24a"}
Sep 10 20:57:17.654055 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:17.653594    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc" event={"ID":"86fbadd7-7507-494c-ba3f-0d90f598cf26","Type":"ContainerStarted","Data":"f0c1bd7a956add664b6ac35df6740b4bee1d3667b3f331c6ed0218f0b08c69d2"}
Sep 10 20:57:17.659932 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:17.659698    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5cngp" event={"ID":"9c335d4d-369f-4327-b77f-64a323fcefaa","Type":"ContainerStarted","Data":"7f4507d9562a1223bf02a4560f7ffc5c06be1a3d9bcf53cdf3a34e44df3971f9"}
Sep 10 20:57:17.667084 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:17.667057    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-232.ec2.internal" event={"ID":"cfb73c088fb2994699656a729ac976af","Type":"ContainerStarted","Data":"87575eb3b37c535582390ff329f044c41a5d025b6ad61a40ae543142f838454e"}
Sep 10 20:57:17.677520 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:17.677114    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p2jdt" event={"ID":"8873b843-8d37-4612-93c0-57eedc38ea18","Type":"ContainerStarted","Data":"c118c4b31063de71751f4f3a32c7355817d570ba1db800d05128a09fa6293133"}
Sep 10 20:57:17.683777 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:17.683717    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5cngp" podStartSLOduration=1.0986434919999999 podStartE2EDuration="21.683701045s" totalImagesPullingTime="20.585057553s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:17.683447286 +0000 UTC m=+22.895912216" watchObservedRunningTime="2026-09-10 20:57:17.683701045 +0000 UTC m=+22.896165965"
Sep 10 20:57:17.743295 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:17.743065    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-p2jdt" podStartSLOduration=1.093039221 podStartE2EDuration="21.742961235s" totalImagesPullingTime="20.649922014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:17.740069605 +0000 UTC m=+22.952534525" watchObservedRunningTime="2026-09-10 20:57:17.742961235 +0000 UTC m=+22.955426154"
Sep 10 20:57:17.744238 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:17.744159    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-232.ec2.internal" podStartSLOduration=20.744145332 podStartE2EDuration="20.744145332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:17.706339085 +0000 UTC m=+22.918804005" watchObservedRunningTime="2026-09-10 20:57:17.744145332 +0000 UTC m=+22.956610254"
Sep 10 20:57:18.432289 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:18.432216    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:57:18.432289 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:18.432253    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:57:18.432289 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:18.432254    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:57:18.432511 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:18.432303    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm62m" podUID="f720dd2d-8c6a-4365-986a-68bbf39853cb"
Sep 10 20:57:18.432511 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:18.432434    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6qrf" podUID="c3c750a9-6302-4266-9b0d-5a681ccdf76d"
Sep 10 20:57:18.432584 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:18.432551    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9wtc" podUID="27f82b26-5156-4cbe-ac0c-23a386fa331b"
Sep 10 20:57:18.682784 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:18.682718    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jfw2v" event={"ID":"218e6e53-1d6e-4c3a-adf3-4e9c2d9f9afa","Type":"ContainerStarted","Data":"1bc874a9be6706a9beb33ca0b550491e1a465f6cddd19707434bda271ed29ba7"}
Sep 10 20:57:18.685794 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:18.685761    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc" event={"ID":"86fbadd7-7507-494c-ba3f-0d90f598cf26","Type":"ContainerStarted","Data":"5260e5f55236440e82deae4a457079cce0d5d7a55b34966f11cb71a6816c47af"}
Sep 10 20:57:18.685926 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:18.685842    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc" event={"ID":"86fbadd7-7507-494c-ba3f-0d90f598cf26","Type":"ContainerStarted","Data":"d66fc002b414763ddb907aed101bfe1fd22e2c7880111a6cf81103e1ff558d5a"}
Sep 10 20:57:18.685926 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:18.685891    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc" event={"ID":"86fbadd7-7507-494c-ba3f-0d90f598cf26","Type":"ContainerStarted","Data":"88a24ff72ea9cc2e44de16bfa831cc49cd25593c7de0bbd28cbd3efad929bf46"}
Sep 10 20:57:18.686059 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:18.685933    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc" event={"ID":"86fbadd7-7507-494c-ba3f-0d90f598cf26","Type":"ContainerStarted","Data":"04397fa91ab5a03119499752b04f6510aa93eae5f1658ba1e4abc5eed6a30c97"}
Sep 10 20:57:18.688733 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:18.688712    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7jnhf" event={"ID":"be238cba-4fb7-4176-9a0a-cd247ac53feb","Type":"ContainerStarted","Data":"909ebb21d55f9e8207c15c7741205c334e3a6117782185fdc15223a20365cd03"}
Sep 10 20:57:18.690091 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:18.690066    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v86hl" event={"ID":"33e1772c-e184-4b7d-89a8-05da5870b62c","Type":"ContainerStarted","Data":"48bf3f5a65375760fe0174dfd1cbdf4d8daca53d1b9519d025130a14837f59f8"}
Sep 10 20:57:18.691454 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:18.691434    2572 generic.go:356] "Generic (PLEG): container finished" podID="97856763-2459-4493-9a2e-f34819281bc6" containerID="be72c02ba949d298e46569a35a549ca2d0f0fa23ba3cdd9ba7641497739be730" exitCode=0
Sep 10 20:57:18.691549 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:18.691508    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwvmr" event={"ID":"97856763-2459-4493-9a2e-f34819281bc6","Type":"ContainerDied","Data":"be72c02ba949d298e46569a35a549ca2d0f0fa23ba3cdd9ba7641497739be730"}
Sep 10 20:57:18.693097 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:18.693073    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd" event={"ID":"b6a04231-4e0b-4a89-b8a0-276cc93d6a94","Type":"ContainerStarted","Data":"38522bea2877967e9104068ea04df438a1fd2f9479cbbec303e9c6c45b37c439"}
Sep 10 20:57:18.694519 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:18.694501    2572 generic.go:356] "Generic (PLEG): container finished" podID="5251df6d6d13e895d3869fc5a6e732f6" containerID="9738358521c42cea54b2027cac918ede595dfc104852904dd46cf92c5299bf07" exitCode=0
Sep 10 20:57:18.694607 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:18.694563    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-232.ec2.internal" event={"ID":"5251df6d6d13e895d3869fc5a6e732f6","Type":"ContainerDied","Data":"9738358521c42cea54b2027cac918ede595dfc104852904dd46cf92c5299bf07"}
Sep 10 20:57:18.701744 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:18.701710    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jfw2v" podStartSLOduration=2.265449194 podStartE2EDuration="22.701698947s" totalImagesPullingTime="20.436249753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:18.701328031 +0000 UTC m=+23.913792949" watchObservedRunningTime="2026-09-10 20:57:18.701698947 +0000 UTC m=+23.914163865"
Sep 10 20:57:18.791751 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:18.791711    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v86hl" podStartSLOduration=2.236193165 podStartE2EDuration="22.791699764s" totalImagesPullingTime="20.555506599s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:18.791506761 +0000 UTC m=+24.003971686" watchObservedRunningTime="2026-09-10 20:57:18.791699764 +0000 UTC m=+24.004164683"
Sep 10 20:57:18.791835 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:18.791773    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7jnhf" podStartSLOduration=2.304323139 podStartE2EDuration="22.791768079s" totalImagesPullingTime="20.48744494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:18.765223101 +0000 UTC m=+23.977688019" watchObservedRunningTime="2026-09-10 20:57:18.791768079 +0000 UTC m=+24.004232995"
Sep 10 20:57:19.007434 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:19.007297    2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 20:57:19.095019 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:19.094976    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jfw2v"
Sep 10 20:57:19.160940 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:19.160916    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jfw2v"
Sep 10 20:57:19.161482 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:19.161405    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jfw2v"
Sep 10 20:57:19.438591 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:19.438509    2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T20:57:19.007314632Z","UUID":"edb24e42-0ab7-45f7-ba86-d0a50070d392","Handler":null,"Name":"","Endpoint":""}
Sep 10 20:57:19.441303 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:19.441281    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 20:57:19.441419 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:19.441318    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 20:57:19.699427 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:19.699343    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5xlmr" event={"ID":"b9050703-03d5-49d4-8c74-dd1588f75210","Type":"ContainerStarted","Data":"ae05d2c7a761b82440602d0668e4bd707c30563e6f4f9b4896a974b38fc8f495"}
Sep 10 20:57:19.701965 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:19.701932    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd" event={"ID":"b6a04231-4e0b-4a89-b8a0-276cc93d6a94","Type":"ContainerStarted","Data":"cbeb508e2313624c1a239082bbfda510a791fa530ec2a1e024db2352defb8e97"}
Sep 10 20:57:19.704339 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:19.704314    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-232.ec2.internal" event={"ID":"5251df6d6d13e895d3869fc5a6e732f6","Type":"ContainerStarted","Data":"3817788dd97802a2e213f7adfccb8a6cb98e50920fe6e3d7f08ac8064b946ca2"}
Sep 10 20:57:19.705484 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:19.705463    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jfw2v"
Sep 10 20:57:19.743015 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:19.742949    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5xlmr" podStartSLOduration=3.316125505 podStartE2EDuration="23.742931733s" totalImagesPullingTime="20.426806228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:19.720562354 +0000 UTC m=+24.933027273" watchObservedRunningTime="2026-09-10 20:57:19.742931733 +0000 UTC m=+24.955396645"
Sep 10 20:57:20.432832 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:20.432582    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:57:20.433009 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:20.432862    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6qrf" podUID="c3c750a9-6302-4266-9b0d-5a681ccdf76d"
Sep 10 20:57:20.433009 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:20.432640    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:57:20.433009 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:20.432970    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9wtc" podUID="27f82b26-5156-4cbe-ac0c-23a386fa331b"
Sep 10 20:57:20.433180 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:20.432587    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:57:20.433180 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:20.433079    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm62m" podUID="f720dd2d-8c6a-4365-986a-68bbf39853cb"
Sep 10 20:57:20.709501 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:20.709414    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd" event={"ID":"b6a04231-4e0b-4a89-b8a0-276cc93d6a94","Type":"ContainerStarted","Data":"f05501976a6aac154c4a1bae58818dbf1b874e75cbfc4c104e7e17b3320b5970"}
Sep 10 20:57:20.714319 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:20.714286    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc" event={"ID":"86fbadd7-7507-494c-ba3f-0d90f598cf26","Type":"ContainerStarted","Data":"a816a815e7cd2cad699f8ee55273a5efa453365aff2698d09823516fbe766bcb"}
Sep 10 20:57:20.730700 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:20.730651    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-232.ec2.internal" podStartSLOduration=23.730634468 podStartE2EDuration="23.730634468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:19.767420585 +0000 UTC m=+24.979885505" watchObservedRunningTime="2026-09-10 20:57:20.730634468 +0000 UTC m=+25.943099386"
Sep 10 20:57:20.730932 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:20.730906    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-822jd" podStartSLOduration=1.89990049 podStartE2EDuration="24.730895609s" totalImagesPullingTime="22.830995119s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:56:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:20.730295 +0000 UTC m=+25.942759920" watchObservedRunningTime="2026-09-10 20:57:20.730895609 +0000 UTC m=+25.943360529"
Sep 10 20:57:22.432380 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:22.432348    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:57:22.432803 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:22.432461    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:57:22.432803 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:22.432476    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6qrf" podUID="c3c750a9-6302-4266-9b0d-5a681ccdf76d"
Sep 10 20:57:22.432803 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:22.432498    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:57:22.432803 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:22.432572    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm62m" podUID="f720dd2d-8c6a-4365-986a-68bbf39853cb"
Sep 10 20:57:22.432803 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:22.432652    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9wtc" podUID="27f82b26-5156-4cbe-ac0c-23a386fa331b"
Sep 10 20:57:22.723996 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:22.723925    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc" event={"ID":"86fbadd7-7507-494c-ba3f-0d90f598cf26","Type":"ContainerStarted","Data":"679b348e5851dcf6c0ec7b5b9259a027432a7415b276691bd47499d6b931918a"}
Sep 10 20:57:22.724887 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:22.724780    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:57:22.724887 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:22.724843    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:57:22.742783 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:22.742714    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:57:22.744700 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:22.744609    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:57:22.764006 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:22.763501    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc" podStartSLOduration=6.226277035 podStartE2EDuration="26.763483209s" totalImagesPullingTime="20.537206174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:22.762733122 +0000 UTC m=+27.975198036" watchObservedRunningTime="2026-09-10 20:57:22.763483209 +0000 UTC m=+27.975948129"
Sep 10 20:57:23.728350 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:23.728176    2572 generic.go:356] "Generic (PLEG): container finished" podID="97856763-2459-4493-9a2e-f34819281bc6" containerID="794a44aca928202193b856522f26c8e1ca129aff67eef91e2299566bfdc96659" exitCode=0
Sep 10 20:57:23.728646 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:23.728220    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwvmr" event={"ID":"97856763-2459-4493-9a2e-f34819281bc6","Type":"ContainerDied","Data":"794a44aca928202193b856522f26c8e1ca129aff67eef91e2299566bfdc96659"}
Sep 10 20:57:23.728646 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:23.728499    2572 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 20:57:24.323710 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:24.323674    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:57:24.432136 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:24.432108    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:57:24.432284 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:24.432206    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:57:24.432284 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:24.432213    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9wtc" podUID="27f82b26-5156-4cbe-ac0c-23a386fa331b"
Sep 10 20:57:24.432284 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:24.432237    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:57:24.432449 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:24.432334    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm62m" podUID="f720dd2d-8c6a-4365-986a-68bbf39853cb"
Sep 10 20:57:24.432449 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:24.432422    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6qrf" podUID="c3c750a9-6302-4266-9b0d-5a681ccdf76d"
Sep 10 20:57:24.741217 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:24.741151    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sm62m"]
Sep 10 20:57:24.741938 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:24.741233    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:57:24.741938 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:24.741304    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm62m" podUID="f720dd2d-8c6a-4365-986a-68bbf39853cb"
Sep 10 20:57:24.745068 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:24.745031    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j6qrf"]
Sep 10 20:57:24.745179 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:24.745120    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:57:24.745256 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:24.745236    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6qrf" podUID="c3c750a9-6302-4266-9b0d-5a681ccdf76d"
Sep 10 20:57:24.745665 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:24.745645    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p9wtc"]
Sep 10 20:57:24.745731 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:24.745718    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:57:24.745836 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:24.745796    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9wtc" podUID="27f82b26-5156-4cbe-ac0c-23a386fa331b"
Sep 10 20:57:25.734623 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:25.734581    2572 generic.go:356] "Generic (PLEG): container finished" podID="97856763-2459-4493-9a2e-f34819281bc6" containerID="8b4811e13974a993d3202e0847aa4449086e9da0a62e5b108c1e6608706dec42" exitCode=0
Sep 10 20:57:25.734776 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:25.734668    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwvmr" event={"ID":"97856763-2459-4493-9a2e-f34819281bc6","Type":"ContainerDied","Data":"8b4811e13974a993d3202e0847aa4449086e9da0a62e5b108c1e6608706dec42"}
Sep 10 20:57:26.432812 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:26.432579    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:57:26.433104 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:26.432613    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:57:26.433104 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:26.432920    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm62m" podUID="f720dd2d-8c6a-4365-986a-68bbf39853cb"
Sep 10 20:57:26.433104 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:26.432613    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:57:26.433104 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:26.432957    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6qrf" podUID="c3c750a9-6302-4266-9b0d-5a681ccdf76d"
Sep 10 20:57:26.433229 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:26.433111    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9wtc" podUID="27f82b26-5156-4cbe-ac0c-23a386fa331b"
Sep 10 20:57:26.738952 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:26.738925    2572 generic.go:356] "Generic (PLEG): container finished" podID="97856763-2459-4493-9a2e-f34819281bc6" containerID="a428bdb33e0225d063827c4a33c70ea5b01a61bc3ebfd96a1c51425be0027e52" exitCode=0
Sep 10 20:57:26.739106 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:26.739007    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwvmr" event={"ID":"97856763-2459-4493-9a2e-f34819281bc6","Type":"ContainerDied","Data":"a428bdb33e0225d063827c4a33c70ea5b01a61bc3ebfd96a1c51425be0027e52"}
Sep 10 20:57:28.214094 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:28.214052    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27f82b26-5156-4cbe-ac0c-23a386fa331b-metrics-certs\") pod \"network-metrics-daemon-p9wtc\" (UID: \"27f82b26-5156-4cbe-ac0c-23a386fa331b\") " pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:57:28.214094 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:28.214095    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f720dd2d-8c6a-4365-986a-68bbf39853cb-original-pull-secret\") pod \"global-pull-secret-syncer-sm62m\" (UID: \"f720dd2d-8c6a-4365-986a-68bbf39853cb\") " pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:57:28.214604 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:28.214208    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:28.214604 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:28.214209    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:28.214604 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:28.214277    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f720dd2d-8c6a-4365-986a-68bbf39853cb-original-pull-secret podName:f720dd2d-8c6a-4365-986a-68bbf39853cb nodeName:}" failed. No retries permitted until 2026-09-10 20:58:00.214258548 +0000 UTC m=+65.426723449 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f720dd2d-8c6a-4365-986a-68bbf39853cb-original-pull-secret") pod "global-pull-secret-syncer-sm62m" (UID: "f720dd2d-8c6a-4365-986a-68bbf39853cb") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:28.214604 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:28.214296    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27f82b26-5156-4cbe-ac0c-23a386fa331b-metrics-certs podName:27f82b26-5156-4cbe-ac0c-23a386fa331b nodeName:}" failed. No retries permitted until 2026-09-10 20:58:00.214286383 +0000 UTC m=+65.426751281 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27f82b26-5156-4cbe-ac0c-23a386fa331b-metrics-certs") pod "network-metrics-daemon-p9wtc" (UID: "27f82b26-5156-4cbe-ac0c-23a386fa331b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:28.314956 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:28.314926    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4v6g\" (UniqueName: \"kubernetes.io/projected/c3c750a9-6302-4266-9b0d-5a681ccdf76d-kube-api-access-t4v6g\") pod \"network-check-target-j6qrf\" (UID: \"c3c750a9-6302-4266-9b0d-5a681ccdf76d\") " pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:57:28.315101 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:28.315071    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:57:28.315101 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:28.315088    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:57:28.315101 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:28.315096    2572 projected.go:196] Error preparing data for projected volume kube-api-access-t4v6g for pod openshift-network-diagnostics/network-check-target-j6qrf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:28.315270 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:28.315134    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3c750a9-6302-4266-9b0d-5a681ccdf76d-kube-api-access-t4v6g podName:c3c750a9-6302-4266-9b0d-5a681ccdf76d nodeName:}" failed. No retries permitted until 2026-09-10 20:58:00.315120427 +0000 UTC m=+65.527585324 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-t4v6g" (UniqueName: "kubernetes.io/projected/c3c750a9-6302-4266-9b0d-5a681ccdf76d-kube-api-access-t4v6g") pod "network-check-target-j6qrf" (UID: "c3c750a9-6302-4266-9b0d-5a681ccdf76d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:28.432425 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:28.432399    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:57:28.432425 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:28.432430    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:57:28.432597 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:28.432511    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm62m" podUID="f720dd2d-8c6a-4365-986a-68bbf39853cb"
Sep 10 20:57:28.432597 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:28.432545    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6qrf" podUID="c3c750a9-6302-4266-9b0d-5a681ccdf76d"
Sep 10 20:57:28.432597 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:28.432575    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:57:28.432689 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:28.432629    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9wtc" podUID="27f82b26-5156-4cbe-ac0c-23a386fa331b"
Sep 10 20:57:30.431718 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:30.431687    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:57:30.432240 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:30.431730    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:57:30.432240 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:30.431731    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:57:30.432240 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:30.431809    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm62m" podUID="f720dd2d-8c6a-4365-986a-68bbf39853cb"
Sep 10 20:57:30.432240 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:30.431891    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6qrf" podUID="c3c750a9-6302-4266-9b0d-5a681ccdf76d"
Sep 10 20:57:30.432240 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:30.432010    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9wtc" podUID="27f82b26-5156-4cbe-ac0c-23a386fa331b"
Sep 10 20:57:31.095898 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.095873    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-232.ec2.internal" event="NodeReady"
Sep 10 20:57:31.096012 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.095998    2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 20:57:31.172935 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.172912    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-n29fp"]
Sep 10 20:57:31.192905 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.192878    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lwhlk"]
Sep 10 20:57:31.193037 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.193018    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n29fp"
Sep 10 20:57:31.197236 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.196914    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 20:57:31.197236 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.196941    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 20:57:31.197236 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.197162    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 20:57:31.197471 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.197384    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kgg6r\""
Sep 10 20:57:31.213010 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.212990    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n29fp"]
Sep 10 20:57:31.213127 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.213033    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lwhlk"]
Sep 10 20:57:31.213127 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.213071    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lwhlk"
Sep 10 20:57:31.216017 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.215998    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 20:57:31.216107 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.216040    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jw26g\""
Sep 10 20:57:31.216360 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.216345    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 20:57:31.338034 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.338010    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8mlc\" (UniqueName: \"kubernetes.io/projected/80ad8dbd-fba2-4011-befe-6a1b7aa0c6b0-kube-api-access-w8mlc\") pod \"ingress-canary-n29fp\" (UID: \"80ad8dbd-fba2-4011-befe-6a1b7aa0c6b0\") " pod="openshift-ingress-canary/ingress-canary-n29fp"
Sep 10 20:57:31.338133 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.338044    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7afb1a12-ba19-4437-8444-efd2b3a5de37-config-volume\") pod \"dns-default-lwhlk\" (UID: \"7afb1a12-ba19-4437-8444-efd2b3a5de37\") " pod="openshift-dns/dns-default-lwhlk"
Sep 10 20:57:31.338133 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.338113    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4rdb\" (UniqueName: \"kubernetes.io/projected/7afb1a12-ba19-4437-8444-efd2b3a5de37-kube-api-access-h4rdb\") pod \"dns-default-lwhlk\" (UID: \"7afb1a12-ba19-4437-8444-efd2b3a5de37\") " pod="openshift-dns/dns-default-lwhlk"
Sep 10 20:57:31.338210 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.338166    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80ad8dbd-fba2-4011-befe-6a1b7aa0c6b0-cert\") pod \"ingress-canary-n29fp\" (UID: \"80ad8dbd-fba2-4011-befe-6a1b7aa0c6b0\") " pod="openshift-ingress-canary/ingress-canary-n29fp"
Sep 10 20:57:31.338210 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.338202    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7afb1a12-ba19-4437-8444-efd2b3a5de37-metrics-tls\") pod \"dns-default-lwhlk\" (UID: \"7afb1a12-ba19-4437-8444-efd2b3a5de37\") " pod="openshift-dns/dns-default-lwhlk"
Sep 10 20:57:31.338281 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.338225    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7afb1a12-ba19-4437-8444-efd2b3a5de37-tmp-dir\") pod \"dns-default-lwhlk\" (UID: \"7afb1a12-ba19-4437-8444-efd2b3a5de37\") " pod="openshift-dns/dns-default-lwhlk"
Sep 10 20:57:31.439258 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.439204    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4rdb\" (UniqueName: \"kubernetes.io/projected/7afb1a12-ba19-4437-8444-efd2b3a5de37-kube-api-access-h4rdb\") pod \"dns-default-lwhlk\" (UID: \"7afb1a12-ba19-4437-8444-efd2b3a5de37\") " pod="openshift-dns/dns-default-lwhlk"
Sep 10 20:57:31.439258 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.439233    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80ad8dbd-fba2-4011-befe-6a1b7aa0c6b0-cert\") pod \"ingress-canary-n29fp\" (UID: \"80ad8dbd-fba2-4011-befe-6a1b7aa0c6b0\") " pod="openshift-ingress-canary/ingress-canary-n29fp"
Sep 10 20:57:31.439697 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.439264    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7afb1a12-ba19-4437-8444-efd2b3a5de37-metrics-tls\") pod \"dns-default-lwhlk\" (UID: \"7afb1a12-ba19-4437-8444-efd2b3a5de37\") " pod="openshift-dns/dns-default-lwhlk"
Sep 10 20:57:31.439697 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.439284    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7afb1a12-ba19-4437-8444-efd2b3a5de37-tmp-dir\") pod \"dns-default-lwhlk\" (UID: \"7afb1a12-ba19-4437-8444-efd2b3a5de37\") " pod="openshift-dns/dns-default-lwhlk"
Sep 10 20:57:31.439697 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.439303    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8mlc\" (UniqueName: \"kubernetes.io/projected/80ad8dbd-fba2-4011-befe-6a1b7aa0c6b0-kube-api-access-w8mlc\") pod \"ingress-canary-n29fp\" (UID: \"80ad8dbd-fba2-4011-befe-6a1b7aa0c6b0\") " pod="openshift-ingress-canary/ingress-canary-n29fp"
Sep 10 20:57:31.439697 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.439319    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7afb1a12-ba19-4437-8444-efd2b3a5de37-config-volume\") pod \"dns-default-lwhlk\" (UID: \"7afb1a12-ba19-4437-8444-efd2b3a5de37\") " pod="openshift-dns/dns-default-lwhlk"
Sep 10 20:57:31.439697 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:31.439367    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:57:31.439697 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:31.439446    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7afb1a12-ba19-4437-8444-efd2b3a5de37-metrics-tls podName:7afb1a12-ba19-4437-8444-efd2b3a5de37 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:31.939425733 +0000 UTC m=+37.151890631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7afb1a12-ba19-4437-8444-efd2b3a5de37-metrics-tls") pod "dns-default-lwhlk" (UID: "7afb1a12-ba19-4437-8444-efd2b3a5de37") : secret "dns-default-metrics-tls" not found
Sep 10 20:57:31.439697 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.439649    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7afb1a12-ba19-4437-8444-efd2b3a5de37-tmp-dir\") pod \"dns-default-lwhlk\" (UID: \"7afb1a12-ba19-4437-8444-efd2b3a5de37\") " pod="openshift-dns/dns-default-lwhlk"
Sep 10 20:57:31.439917 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.439835    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7afb1a12-ba19-4437-8444-efd2b3a5de37-config-volume\") pod \"dns-default-lwhlk\" (UID: \"7afb1a12-ba19-4437-8444-efd2b3a5de37\") " pod="openshift-dns/dns-default-lwhlk"
Sep 10 20:57:31.443405 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.443385    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80ad8dbd-fba2-4011-befe-6a1b7aa0c6b0-cert\") pod \"ingress-canary-n29fp\" (UID: \"80ad8dbd-fba2-4011-befe-6a1b7aa0c6b0\") " pod="openshift-ingress-canary/ingress-canary-n29fp"
Sep 10 20:57:31.460303 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.460277    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8mlc\" (UniqueName: \"kubernetes.io/projected/80ad8dbd-fba2-4011-befe-6a1b7aa0c6b0-kube-api-access-w8mlc\") pod \"ingress-canary-n29fp\" (UID: \"80ad8dbd-fba2-4011-befe-6a1b7aa0c6b0\") " pod="openshift-ingress-canary/ingress-canary-n29fp"
Sep 10 20:57:31.460633 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.460615    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4rdb\" (UniqueName: \"kubernetes.io/projected/7afb1a12-ba19-4437-8444-efd2b3a5de37-kube-api-access-h4rdb\") pod \"dns-default-lwhlk\" (UID: \"7afb1a12-ba19-4437-8444-efd2b3a5de37\") " pod="openshift-dns/dns-default-lwhlk"
Sep 10 20:57:31.502683 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.502663    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n29fp"
Sep 10 20:57:31.754544 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.754338    2572 generic.go:356] "Generic (PLEG): container finished" podID="97856763-2459-4493-9a2e-f34819281bc6" containerID="b6a293101eecbefc1e2c2dd71e21693b0957ab26f95d2e8bdfa0d3f8af5e7d4e" exitCode=0
Sep 10 20:57:31.754544 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.754424    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwvmr" event={"ID":"97856763-2459-4493-9a2e-f34819281bc6","Type":"ContainerDied","Data":"b6a293101eecbefc1e2c2dd71e21693b0957ab26f95d2e8bdfa0d3f8af5e7d4e"}
Sep 10 20:57:31.818627 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.818603    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n29fp"]
Sep 10 20:57:31.822055 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:57:31.822023    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ad8dbd_fba2_4011_befe_6a1b7aa0c6b0.slice/crio-b26e97396858d1f9b41054d9bc2f78e1479471ea6803ae4f569448b49e8b8fda WatchSource:0}: Error finding container b26e97396858d1f9b41054d9bc2f78e1479471ea6803ae4f569448b49e8b8fda: Status 404 returned error can't find the container with id b26e97396858d1f9b41054d9bc2f78e1479471ea6803ae4f569448b49e8b8fda
Sep 10 20:57:31.943129 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:31.943104    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7afb1a12-ba19-4437-8444-efd2b3a5de37-metrics-tls\") pod \"dns-default-lwhlk\" (UID: \"7afb1a12-ba19-4437-8444-efd2b3a5de37\") " pod="openshift-dns/dns-default-lwhlk"
Sep 10 20:57:31.943225 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:31.943213    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:57:31.943264 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:31.943259    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7afb1a12-ba19-4437-8444-efd2b3a5de37-metrics-tls podName:7afb1a12-ba19-4437-8444-efd2b3a5de37 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:32.943244796 +0000 UTC m=+38.155709697 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7afb1a12-ba19-4437-8444-efd2b3a5de37-metrics-tls") pod "dns-default-lwhlk" (UID: "7afb1a12-ba19-4437-8444-efd2b3a5de37") : secret "dns-default-metrics-tls" not found
Sep 10 20:57:32.432396 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:32.432366    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:57:32.432560 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:32.432482    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:57:32.432560 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:32.432518    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:57:32.435872 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:32.435853    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 20:57:32.436068 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:32.436043    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zr4bv\""
Sep 10 20:57:32.436829 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:32.436810    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 20:57:32.436929 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:32.436818    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gncgs\""
Sep 10 20:57:32.437101 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:32.437078    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 20:57:32.437197 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:32.437162    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 20:57:32.758220 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:32.758152    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n29fp" event={"ID":"80ad8dbd-fba2-4011-befe-6a1b7aa0c6b0","Type":"ContainerStarted","Data":"b26e97396858d1f9b41054d9bc2f78e1479471ea6803ae4f569448b49e8b8fda"}
Sep 10 20:57:32.761776 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:32.761745    2572 generic.go:356] "Generic (PLEG): container finished" podID="97856763-2459-4493-9a2e-f34819281bc6" containerID="347f6c5345832d5fb2e495100bee2bbb04e5396d9e2d50a530541e7dc308e440" exitCode=0
Sep 10 20:57:32.761905 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:32.761785    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwvmr" event={"ID":"97856763-2459-4493-9a2e-f34819281bc6","Type":"ContainerDied","Data":"347f6c5345832d5fb2e495100bee2bbb04e5396d9e2d50a530541e7dc308e440"}
Sep 10 20:57:32.949055 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:32.949030    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7afb1a12-ba19-4437-8444-efd2b3a5de37-metrics-tls\") pod \"dns-default-lwhlk\" (UID: \"7afb1a12-ba19-4437-8444-efd2b3a5de37\") " pod="openshift-dns/dns-default-lwhlk"
Sep 10 20:57:32.949162 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:32.949144    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:57:32.949213 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:32.949198    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7afb1a12-ba19-4437-8444-efd2b3a5de37-metrics-tls podName:7afb1a12-ba19-4437-8444-efd2b3a5de37 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:34.94918342 +0000 UTC m=+40.161648317 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7afb1a12-ba19-4437-8444-efd2b3a5de37-metrics-tls") pod "dns-default-lwhlk" (UID: "7afb1a12-ba19-4437-8444-efd2b3a5de37") : secret "dns-default-metrics-tls" not found
Sep 10 20:57:33.768373 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:33.768338    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwvmr" event={"ID":"97856763-2459-4493-9a2e-f34819281bc6","Type":"ContainerStarted","Data":"e8fc5250befbb1a20d16e0556585d481b43300fb488fc07af9cfb81fab6719f4"}
Sep 10 20:57:33.828586 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:33.828538    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hwvmr" podStartSLOduration=7.026226691 podStartE2EDuration="37.828519177s" totalImagesPullingTime="30.802292486s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:56:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:33.824312852 +0000 UTC m=+39.036777772" watchObservedRunningTime="2026-09-10 20:57:33.828519177 +0000 UTC m=+39.040984098"
Sep 10 20:57:34.772673 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:34.772639    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n29fp" event={"ID":"80ad8dbd-fba2-4011-befe-6a1b7aa0c6b0","Type":"ContainerStarted","Data":"1c6d091ec30329e2c54d8fe55adf3dc9e61af8f75a41e9fce43140a9db795661"}
Sep 10 20:57:34.798634 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:34.798592    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-n29fp" podStartSLOduration=1.399592543 podStartE2EDuration="3.798577138s" totalImagesPullingTime="2.398984595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:34.794340906 +0000 UTC m=+40.006805803" watchObservedRunningTime="2026-09-10 20:57:34.798577138 +0000 UTC m=+40.011042086"
Sep 10 20:57:34.965249 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:34.965223    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7afb1a12-ba19-4437-8444-efd2b3a5de37-metrics-tls\") pod \"dns-default-lwhlk\" (UID: \"7afb1a12-ba19-4437-8444-efd2b3a5de37\") " pod="openshift-dns/dns-default-lwhlk"
Sep 10 20:57:34.965356 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:34.965307    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:57:34.965356 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:34.965351    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7afb1a12-ba19-4437-8444-efd2b3a5de37-metrics-tls podName:7afb1a12-ba19-4437-8444-efd2b3a5de37 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:38.965339674 +0000 UTC m=+44.177804572 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7afb1a12-ba19-4437-8444-efd2b3a5de37-metrics-tls") pod "dns-default-lwhlk" (UID: "7afb1a12-ba19-4437-8444-efd2b3a5de37") : secret "dns-default-metrics-tls" not found
Sep 10 20:57:38.991185 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:38.991144    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7afb1a12-ba19-4437-8444-efd2b3a5de37-metrics-tls\") pod \"dns-default-lwhlk\" (UID: \"7afb1a12-ba19-4437-8444-efd2b3a5de37\") " pod="openshift-dns/dns-default-lwhlk"
Sep 10 20:57:38.991584 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:38.991274    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:57:38.991584 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:38.991334    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7afb1a12-ba19-4437-8444-efd2b3a5de37-metrics-tls podName:7afb1a12-ba19-4437-8444-efd2b3a5de37 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:46.991316915 +0000 UTC m=+52.203781824 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7afb1a12-ba19-4437-8444-efd2b3a5de37-metrics-tls") pod "dns-default-lwhlk" (UID: "7afb1a12-ba19-4437-8444-efd2b3a5de37") : secret "dns-default-metrics-tls" not found
Sep 10 20:57:41.404835 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.404789    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-dbhxv"]
Sep 10 20:57:41.427319 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.427295    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dbhxv"
Sep 10 20:57:41.440692 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.440669    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:57:41.440803 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.440689    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-m9hd8\""
Sep 10 20:57:41.444263 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.444248    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 10 20:57:41.448454 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.448432    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 10 20:57:41.449757 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.449636    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 10 20:57:41.454378 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.454350    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-dbhxv"]
Sep 10 20:57:41.505495 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.505473    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x229c\" (UniqueName: \"kubernetes.io/projected/4a1bd0d7-bb2f-494d-9553-9f3f89cfa92d-kube-api-access-x229c\") pod \"service-ca-operator-cf7c776b7-dbhxv\" (UID: \"4a1bd0d7-bb2f-494d-9553-9f3f89cfa92d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dbhxv"
Sep 10 20:57:41.505609 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.505512    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a1bd0d7-bb2f-494d-9553-9f3f89cfa92d-config\") pod \"service-ca-operator-cf7c776b7-dbhxv\" (UID: \"4a1bd0d7-bb2f-494d-9553-9f3f89cfa92d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dbhxv"
Sep 10 20:57:41.505609 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.505551    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4a1bd0d7-bb2f-494d-9553-9f3f89cfa92d-serving-cert\") pod \"service-ca-operator-cf7c776b7-dbhxv\" (UID: \"4a1bd0d7-bb2f-494d-9553-9f3f89cfa92d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dbhxv"
Sep 10 20:57:41.606390 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.606361    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4a1bd0d7-bb2f-494d-9553-9f3f89cfa92d-serving-cert\") pod \"service-ca-operator-cf7c776b7-dbhxv\" (UID: \"4a1bd0d7-bb2f-494d-9553-9f3f89cfa92d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dbhxv"
Sep 10 20:57:41.606498 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.606413    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x229c\" (UniqueName: \"kubernetes.io/projected/4a1bd0d7-bb2f-494d-9553-9f3f89cfa92d-kube-api-access-x229c\") pod \"service-ca-operator-cf7c776b7-dbhxv\" (UID: \"4a1bd0d7-bb2f-494d-9553-9f3f89cfa92d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dbhxv"
Sep 10 20:57:41.606498 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.606479    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a1bd0d7-bb2f-494d-9553-9f3f89cfa92d-config\") pod \"service-ca-operator-cf7c776b7-dbhxv\" (UID: \"4a1bd0d7-bb2f-494d-9553-9f3f89cfa92d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dbhxv"
Sep 10 20:57:41.607123 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.607091    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a1bd0d7-bb2f-494d-9553-9f3f89cfa92d-config\") pod \"service-ca-operator-cf7c776b7-dbhxv\" (UID: \"4a1bd0d7-bb2f-494d-9553-9f3f89cfa92d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dbhxv"
Sep 10 20:57:41.609974 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.609939    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4a1bd0d7-bb2f-494d-9553-9f3f89cfa92d-serving-cert\") pod \"service-ca-operator-cf7c776b7-dbhxv\" (UID: \"4a1bd0d7-bb2f-494d-9553-9f3f89cfa92d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dbhxv"
Sep 10 20:57:41.625209 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.625185    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-j49cs"]
Sep 10 20:57:41.638956 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.638922    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x229c\" (UniqueName: \"kubernetes.io/projected/4a1bd0d7-bb2f-494d-9553-9f3f89cfa92d-kube-api-access-x229c\") pod \"service-ca-operator-cf7c776b7-dbhxv\" (UID: \"4a1bd0d7-bb2f-494d-9553-9f3f89cfa92d\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dbhxv"
Sep 10 20:57:41.646071 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.646051    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79994c7475-ddczp"]
Sep 10 20:57:41.646176 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.646161    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-j49cs"
Sep 10 20:57:41.652697 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.652678    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 10 20:57:41.652840 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.652765    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:57:41.653115 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.653100    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7v5gj\""
Sep 10 20:57:41.666671 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.666629    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:41.671924 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.671797    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 20:57:41.671924 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.671899    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 20:57:41.672117 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.672100    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-24mm2\""
Sep 10 20:57:41.672513 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.672498    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 20:57:41.681147 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.681130    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 20:57:41.688391 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.688369    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-j49cs"]
Sep 10 20:57:41.736274 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.736254    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dbhxv"
Sep 10 20:57:41.740085 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.740067    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79994c7475-ddczp"]
Sep 10 20:57:41.808394 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.808367    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/522ae11b-496f-496b-8af2-71b79451d971-trusted-ca\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:41.808514 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.808407    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qdc8\" (UniqueName: \"kubernetes.io/projected/408e4fb2-52fb-4a31-92d8-9f9aeaeacf6d-kube-api-access-8qdc8\") pod \"volume-data-source-validator-86695b7cc7-j49cs\" (UID: \"408e4fb2-52fb-4a31-92d8-9f9aeaeacf6d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-j49cs"
Sep 10 20:57:41.808514 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.808489    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-registry-tls\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:41.808627 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.808513    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7q8r\" (UniqueName: \"kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-kube-api-access-x7q8r\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:41.808627 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.808548    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/522ae11b-496f-496b-8af2-71b79451d971-image-registry-private-configuration\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:41.808627 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.808573    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/522ae11b-496f-496b-8af2-71b79451d971-ca-trust-extracted\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:41.808627 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.808592    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/522ae11b-496f-496b-8af2-71b79451d971-installation-pull-secrets\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:41.808627 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.808624    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/522ae11b-496f-496b-8af2-71b79451d971-registry-certificates\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:41.808876 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.808656    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-bound-sa-token\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:41.908953 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.908929    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qdc8\" (UniqueName: \"kubernetes.io/projected/408e4fb2-52fb-4a31-92d8-9f9aeaeacf6d-kube-api-access-8qdc8\") pod \"volume-data-source-validator-86695b7cc7-j49cs\" (UID: \"408e4fb2-52fb-4a31-92d8-9f9aeaeacf6d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-j49cs"
Sep 10 20:57:41.909090 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.908995    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-registry-tls\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:41.909090 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.909014    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7q8r\" (UniqueName: \"kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-kube-api-access-x7q8r\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:41.909090 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.909043    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/522ae11b-496f-496b-8af2-71b79451d971-image-registry-private-configuration\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:41.909090 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.909058    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/522ae11b-496f-496b-8af2-71b79451d971-ca-trust-extracted\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:41.909090 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.909076    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/522ae11b-496f-496b-8af2-71b79451d971-installation-pull-secrets\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:41.909314 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.909124    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/522ae11b-496f-496b-8af2-71b79451d971-registry-certificates\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:41.909314 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:41.909175    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:57:41.909314 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:41.909221    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79994c7475-ddczp: secret "image-registry-tls" not found
Sep 10 20:57:41.909314 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:41.909286    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-registry-tls podName:522ae11b-496f-496b-8af2-71b79451d971 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:42.409265265 +0000 UTC m=+47.621730182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-registry-tls") pod "image-registry-79994c7475-ddczp" (UID: "522ae11b-496f-496b-8af2-71b79451d971") : secret "image-registry-tls" not found
Sep 10 20:57:41.909314 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.909309    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-bound-sa-token\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:41.909544 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.909351    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/522ae11b-496f-496b-8af2-71b79451d971-trusted-ca\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:41.909544 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.909522    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/522ae11b-496f-496b-8af2-71b79451d971-ca-trust-extracted\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:41.909824 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.909796    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/522ae11b-496f-496b-8af2-71b79451d971-registry-certificates\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:41.910504 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.910485    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/522ae11b-496f-496b-8af2-71b79451d971-trusted-ca\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:41.911629 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.911609    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/522ae11b-496f-496b-8af2-71b79451d971-image-registry-private-configuration\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:41.911946 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.911929    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/522ae11b-496f-496b-8af2-71b79451d971-installation-pull-secrets\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:41.926125 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.926072    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7q8r\" (UniqueName: \"kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-kube-api-access-x7q8r\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:41.934219 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.934204    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-bound-sa-token\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:41.948922 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.948905    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qdc8\" (UniqueName: \"kubernetes.io/projected/408e4fb2-52fb-4a31-92d8-9f9aeaeacf6d-kube-api-access-8qdc8\") pod \"volume-data-source-validator-86695b7cc7-j49cs\" (UID: \"408e4fb2-52fb-4a31-92d8-9f9aeaeacf6d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-j49cs"
Sep 10 20:57:41.954717 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:41.954703    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-j49cs"
Sep 10 20:57:42.036801 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:42.036777    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-dbhxv"]
Sep 10 20:57:42.042077 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:57:42.042054    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1bd0d7_bb2f_494d_9553_9f3f89cfa92d.slice/crio-88aea37d3e6cae51f9615afbe5176744f212b0f7ae2715b75cfb736615141b85 WatchSource:0}: Error finding container 88aea37d3e6cae51f9615afbe5176744f212b0f7ae2715b75cfb736615141b85: Status 404 returned error can't find the container with id 88aea37d3e6cae51f9615afbe5176744f212b0f7ae2715b75cfb736615141b85
Sep 10 20:57:42.274267 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:42.274206    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-j49cs"]
Sep 10 20:57:42.277257 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:57:42.277222    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408e4fb2_52fb_4a31_92d8_9f9aeaeacf6d.slice/crio-b629c9793fb75c9283a6223542f4144cb1e1fe0282c23fee975a2d9b32ec7924 WatchSource:0}: Error finding container b629c9793fb75c9283a6223542f4144cb1e1fe0282c23fee975a2d9b32ec7924: Status 404 returned error can't find the container with id b629c9793fb75c9283a6223542f4144cb1e1fe0282c23fee975a2d9b32ec7924
Sep 10 20:57:42.414184 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:42.414161    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-registry-tls\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:42.414500 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:42.414296    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:57:42.414500 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:42.414311    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79994c7475-ddczp: secret "image-registry-tls" not found
Sep 10 20:57:42.414500 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:42.414358    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-registry-tls podName:522ae11b-496f-496b-8af2-71b79451d971 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:43.41434342 +0000 UTC m=+48.626808334 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-registry-tls") pod "image-registry-79994c7475-ddczp" (UID: "522ae11b-496f-496b-8af2-71b79451d971") : secret "image-registry-tls" not found
Sep 10 20:57:42.793884 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:42.793850    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-j49cs" event={"ID":"408e4fb2-52fb-4a31-92d8-9f9aeaeacf6d","Type":"ContainerStarted","Data":"b629c9793fb75c9283a6223542f4144cb1e1fe0282c23fee975a2d9b32ec7924"}
Sep 10 20:57:42.795199 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:42.795166    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dbhxv" event={"ID":"4a1bd0d7-bb2f-494d-9553-9f3f89cfa92d","Type":"ContainerStarted","Data":"88aea37d3e6cae51f9615afbe5176744f212b0f7ae2715b75cfb736615141b85"}
Sep 10 20:57:43.422778 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:43.422739    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-registry-tls\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:43.423191 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:43.422932    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:57:43.423191 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:43.422951    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79994c7475-ddczp: secret "image-registry-tls" not found
Sep 10 20:57:43.423191 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:43.423035    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-registry-tls podName:522ae11b-496f-496b-8af2-71b79451d971 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:45.423016997 +0000 UTC m=+50.635481903 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-registry-tls") pod "image-registry-79994c7475-ddczp" (UID: "522ae11b-496f-496b-8af2-71b79451d971") : secret "image-registry-tls" not found
Sep 10 20:57:44.801703 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:44.801620    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-j49cs" event={"ID":"408e4fb2-52fb-4a31-92d8-9f9aeaeacf6d","Type":"ContainerStarted","Data":"f7c8d05d7a6ceb8b0aca9623e09d28e6d5ed7c45556ebbb6720b74da34f0e4d7"}
Sep 10 20:57:44.803187 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:44.803159    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dbhxv" event={"ID":"4a1bd0d7-bb2f-494d-9553-9f3f89cfa92d","Type":"ContainerStarted","Data":"eb44257fdf7595e7940f6da5ed59bfc98b46c64b17f83b59cf7a61959e8a6a5d"}
Sep 10 20:57:44.867071 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:44.867034    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-j49cs" podStartSLOduration=1.592353998 podStartE2EDuration="3.867020956s" totalImagesPullingTime="2.274666958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:44.825326046 +0000 UTC m=+50.037790964" watchObservedRunningTime="2026-09-10 20:57:44.867020956 +0000 UTC m=+50.079485875"
Sep 10 20:57:44.867195 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:44.867125    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dbhxv" podStartSLOduration=1.352186327 podStartE2EDuration="3.867121321s" totalImagesPullingTime="2.514934994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:44.865707193 +0000 UTC m=+50.078172124" watchObservedRunningTime="2026-09-10 20:57:44.867121321 +0000 UTC m=+50.079586239"
Sep 10 20:57:45.439483 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:45.439458    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-registry-tls\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:45.439674 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:45.439581    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:57:45.439674 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:45.439596    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79994c7475-ddczp: secret "image-registry-tls" not found
Sep 10 20:57:45.439674 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:45.439649    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-registry-tls podName:522ae11b-496f-496b-8af2-71b79451d971 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:49.43963176 +0000 UTC m=+54.652096676 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-registry-tls") pod "image-registry-79994c7475-ddczp" (UID: "522ae11b-496f-496b-8af2-71b79451d971") : secret "image-registry-tls" not found
Sep 10 20:57:47.050432 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:47.050393    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7afb1a12-ba19-4437-8444-efd2b3a5de37-metrics-tls\") pod \"dns-default-lwhlk\" (UID: \"7afb1a12-ba19-4437-8444-efd2b3a5de37\") " pod="openshift-dns/dns-default-lwhlk"
Sep 10 20:57:47.050886 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:47.050519    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:57:47.050886 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:47.050571    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7afb1a12-ba19-4437-8444-efd2b3a5de37-metrics-tls podName:7afb1a12-ba19-4437-8444-efd2b3a5de37 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:03.050557361 +0000 UTC m=+68.263022257 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7afb1a12-ba19-4437-8444-efd2b3a5de37-metrics-tls") pod "dns-default-lwhlk" (UID: "7afb1a12-ba19-4437-8444-efd2b3a5de37") : secret "dns-default-metrics-tls" not found
Sep 10 20:57:49.468548 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.468501    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-registry-tls\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:49.469036 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:49.468663    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:57:49.469036 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:49.468683    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79994c7475-ddczp: secret "image-registry-tls" not found
Sep 10 20:57:49.469036 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:49.468741    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-registry-tls podName:522ae11b-496f-496b-8af2-71b79451d971 nodeName:}" failed. No retries permitted until 2026-09-10 20:57:57.468725055 +0000 UTC m=+62.681189952 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-registry-tls") pod "image-registry-79994c7475-ddczp" (UID: "522ae11b-496f-496b-8af2-71b79451d971") : secret "image-registry-tls" not found
Sep 10 20:57:49.503458 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.503427    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-r72gv"]
Sep 10 20:57:49.505213 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.505196    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-r72gv"
Sep 10 20:57:49.508566 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.508541    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 10 20:57:49.510811 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.510787    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-r4jnw\""
Sep 10 20:57:49.511681 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.511630    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 10 20:57:49.527485 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.527452    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-r72gv"]
Sep 10 20:57:49.669916 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.669892    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gfhq\" (UniqueName: \"kubernetes.io/projected/cb644a90-87a0-4942-8ec6-53fd20873973-kube-api-access-6gfhq\") pod \"migrator-8b4d697c5-r72gv\" (UID: \"cb644a90-87a0-4942-8ec6-53fd20873973\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-r72gv"
Sep 10 20:57:49.768945 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.768889    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f74sn"]
Sep 10 20:57:49.770344 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.770315    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gfhq\" (UniqueName: \"kubernetes.io/projected/cb644a90-87a0-4942-8ec6-53fd20873973-kube-api-access-6gfhq\") pod \"migrator-8b4d697c5-r72gv\" (UID: \"cb644a90-87a0-4942-8ec6-53fd20873973\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-r72gv"
Sep 10 20:57:49.770724 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.770708    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:57:49.774603 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.774586    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b4777\""
Sep 10 20:57:49.774877 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.774855    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 20:57:49.775500 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.775482    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 20:57:49.775649 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.775484    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 20:57:49.775770 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.775508    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 20:57:49.798476 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.798453    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f74sn"]
Sep 10 20:57:49.803312 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.803289    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gfhq\" (UniqueName: \"kubernetes.io/projected/cb644a90-87a0-4942-8ec6-53fd20873973-kube-api-access-6gfhq\") pod \"migrator-8b4d697c5-r72gv\" (UID: \"cb644a90-87a0-4942-8ec6-53fd20873973\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-r72gv"
Sep 10 20:57:49.813818 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.813794    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-r72gv"
Sep 10 20:57:49.871438 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.871411    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a20b64fb-c649-43e7-ba2c-3238ed96520c-data-volume\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:57:49.871563 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.871444    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a20b64fb-c649-43e7-ba2c-3238ed96520c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:57:49.871563 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.871471    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a20b64fb-c649-43e7-ba2c-3238ed96520c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:57:49.871563 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.871493    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4mrb\" (UniqueName: \"kubernetes.io/projected/a20b64fb-c649-43e7-ba2c-3238ed96520c-kube-api-access-z4mrb\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:57:49.871563 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.871510    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a20b64fb-c649-43e7-ba2c-3238ed96520c-crio-socket\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:57:49.972764 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.972736    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a20b64fb-c649-43e7-ba2c-3238ed96520c-data-volume\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:57:49.972873 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.972770    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a20b64fb-c649-43e7-ba2c-3238ed96520c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:57:49.972873 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.972794    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a20b64fb-c649-43e7-ba2c-3238ed96520c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:57:49.972873 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:49.972863    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:57:49.973048 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.972924    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4mrb\" (UniqueName: \"kubernetes.io/projected/a20b64fb-c649-43e7-ba2c-3238ed96520c-kube-api-access-z4mrb\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:57:49.973048 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:49.972934    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a20b64fb-c649-43e7-ba2c-3238ed96520c-kube-rbac-proxy-cm podName:a20b64fb-c649-43e7-ba2c-3238ed96520c nodeName:}" failed. No retries permitted until 2026-09-10 20:57:50.472918728 +0000 UTC m=+55.685383625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a20b64fb-c649-43e7-ba2c-3238ed96520c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f74sn" (UID: "a20b64fb-c649-43e7-ba2c-3238ed96520c") : configmap "kube-rbac-proxy" not found
Sep 10 20:57:49.973048 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.972971    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a20b64fb-c649-43e7-ba2c-3238ed96520c-crio-socket\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:57:49.973187 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:49.973043    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:57:49.973187 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.973096    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a20b64fb-c649-43e7-ba2c-3238ed96520c-data-volume\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:57:49.973187 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:49.973110    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a20b64fb-c649-43e7-ba2c-3238ed96520c-insights-runtime-extractor-tls podName:a20b64fb-c649-43e7-ba2c-3238ed96520c nodeName:}" failed. No retries permitted until 2026-09-10 20:57:50.473089439 +0000 UTC m=+55.685554341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a20b64fb-c649-43e7-ba2c-3238ed96520c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f74sn" (UID: "a20b64fb-c649-43e7-ba2c-3238ed96520c") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:57:49.973288 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.973244    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a20b64fb-c649-43e7-ba2c-3238ed96520c-crio-socket\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:57:49.988539 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:49.988517    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4mrb\" (UniqueName: \"kubernetes.io/projected/a20b64fb-c649-43e7-ba2c-3238ed96520c-kube-api-access-z4mrb\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:57:50.104399 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:50.104374    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-r72gv"]
Sep 10 20:57:50.110849 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:57:50.110829    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb644a90_87a0_4942_8ec6_53fd20873973.slice/crio-cf196a66a42de24c00d6050ad2d77055315eac332f077bf7ea205dfd7e21ad31 WatchSource:0}: Error finding container cf196a66a42de24c00d6050ad2d77055315eac332f077bf7ea205dfd7e21ad31: Status 404 returned error can't find the container with id cf196a66a42de24c00d6050ad2d77055315eac332f077bf7ea205dfd7e21ad31
Sep 10 20:57:50.477316 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:50.477260    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a20b64fb-c649-43e7-ba2c-3238ed96520c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:57:50.477601 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:50.477352    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a20b64fb-c649-43e7-ba2c-3238ed96520c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:57:50.477601 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:50.477408    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:57:50.477601 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:50.477462    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a20b64fb-c649-43e7-ba2c-3238ed96520c-insights-runtime-extractor-tls podName:a20b64fb-c649-43e7-ba2c-3238ed96520c nodeName:}" failed. No retries permitted until 2026-09-10 20:57:51.477445871 +0000 UTC m=+56.689910768 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a20b64fb-c649-43e7-ba2c-3238ed96520c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f74sn" (UID: "a20b64fb-c649-43e7-ba2c-3238ed96520c") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:57:50.477601 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:50.477413    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:57:50.477601 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:50.477555    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a20b64fb-c649-43e7-ba2c-3238ed96520c-kube-rbac-proxy-cm podName:a20b64fb-c649-43e7-ba2c-3238ed96520c nodeName:}" failed. No retries permitted until 2026-09-10 20:57:51.477522664 +0000 UTC m=+56.689987562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a20b64fb-c649-43e7-ba2c-3238ed96520c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f74sn" (UID: "a20b64fb-c649-43e7-ba2c-3238ed96520c") : configmap "kube-rbac-proxy" not found
Sep 10 20:57:50.819763 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:50.819731    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-r72gv" event={"ID":"cb644a90-87a0-4942-8ec6-53fd20873973","Type":"ContainerStarted","Data":"cf196a66a42de24c00d6050ad2d77055315eac332f077bf7ea205dfd7e21ad31"}
Sep 10 20:57:51.484690 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:51.484658    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a20b64fb-c649-43e7-ba2c-3238ed96520c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:57:51.484690 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:51.484697    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a20b64fb-c649-43e7-ba2c-3238ed96520c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:57:51.485238 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:51.484774    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:57:51.485238 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:51.484799    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:57:51.485238 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:51.484839    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a20b64fb-c649-43e7-ba2c-3238ed96520c-kube-rbac-proxy-cm podName:a20b64fb-c649-43e7-ba2c-3238ed96520c nodeName:}" failed. No retries permitted until 2026-09-10 20:57:53.48482249 +0000 UTC m=+58.697287401 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a20b64fb-c649-43e7-ba2c-3238ed96520c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f74sn" (UID: "a20b64fb-c649-43e7-ba2c-3238ed96520c") : configmap "kube-rbac-proxy" not found
Sep 10 20:57:51.485238 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:51.484853    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a20b64fb-c649-43e7-ba2c-3238ed96520c-insights-runtime-extractor-tls podName:a20b64fb-c649-43e7-ba2c-3238ed96520c nodeName:}" failed. No retries permitted until 2026-09-10 20:57:53.48484722 +0000 UTC m=+58.697312117 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a20b64fb-c649-43e7-ba2c-3238ed96520c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f74sn" (UID: "a20b64fb-c649-43e7-ba2c-3238ed96520c") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:57:51.824428 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:51.824396    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-r72gv" event={"ID":"cb644a90-87a0-4942-8ec6-53fd20873973","Type":"ContainerStarted","Data":"2420cc263f7ceb95260e14519d5b49289f0ff52a0ae666e65c328f4f1e18c66f"}
Sep 10 20:57:51.824527 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:51.824448    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-r72gv" event={"ID":"cb644a90-87a0-4942-8ec6-53fd20873973","Type":"ContainerStarted","Data":"ea166d0525c352901dbe4fd6b0ed7ca66e6285c487ea196cb52fc52e5b2f2ff3"}
Sep 10 20:57:51.853662 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:51.853619    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-r72gv" podStartSLOduration=1.4539307639999999 podStartE2EDuration="2.853605238s" totalImagesPullingTime="1.399674474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:57:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:51.85161005 +0000 UTC m=+57.064074970" watchObservedRunningTime="2026-09-10 20:57:51.853605238 +0000 UTC m=+57.066070156"
Sep 10 20:57:52.645782 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:52.645747    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v86hl_33e1772c-e184-4b7d-89a8-05da5870b62c/dns-node-resolver/0.log"
Sep 10 20:57:53.245461 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:53.245435    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7jnhf_be238cba-4fb7-4176-9a0a-cd247ac53feb/node-ca/0.log"
Sep 10 20:57:53.500160 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:53.500093    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a20b64fb-c649-43e7-ba2c-3238ed96520c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:57:53.500160 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:53.500131    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a20b64fb-c649-43e7-ba2c-3238ed96520c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:57:53.500367 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:53.500234    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:57:53.500367 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:53.500235    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:57:53.500367 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:53.500288    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a20b64fb-c649-43e7-ba2c-3238ed96520c-insights-runtime-extractor-tls podName:a20b64fb-c649-43e7-ba2c-3238ed96520c nodeName:}" failed. No retries permitted until 2026-09-10 20:57:57.500274735 +0000 UTC m=+62.712739631 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a20b64fb-c649-43e7-ba2c-3238ed96520c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f74sn" (UID: "a20b64fb-c649-43e7-ba2c-3238ed96520c") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:57:53.500367 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:53.500303    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a20b64fb-c649-43e7-ba2c-3238ed96520c-kube-rbac-proxy-cm podName:a20b64fb-c649-43e7-ba2c-3238ed96520c nodeName:}" failed. No retries permitted until 2026-09-10 20:57:57.500295262 +0000 UTC m=+62.712760158 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a20b64fb-c649-43e7-ba2c-3238ed96520c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f74sn" (UID: "a20b64fb-c649-43e7-ba2c-3238ed96520c") : configmap "kube-rbac-proxy" not found
Sep 10 20:57:54.242404 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:54.242374    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n29fp_80ad8dbd-fba2-4011-befe-6a1b7aa0c6b0/serve-healthcheck-canary/0.log"
Sep 10 20:57:54.642532 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:54.642511    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-r72gv_cb644a90-87a0-4942-8ec6-53fd20873973/migrator/0.log"
Sep 10 20:57:54.849989 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:54.849964    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-r72gv_cb644a90-87a0-4942-8ec6-53fd20873973/graceful-termination/0.log"
Sep 10 20:57:55.747260 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:55.747232    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r6fkc"
Sep 10 20:57:57.527851 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:57.527812    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-registry-tls\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:57.528345 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:57.527882    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a20b64fb-c649-43e7-ba2c-3238ed96520c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:57:57.528345 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:57.527917    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a20b64fb-c649-43e7-ba2c-3238ed96520c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:57:57.528345 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:57.527955    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:57:57.528345 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:57.528047    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a20b64fb-c649-43e7-ba2c-3238ed96520c-kube-rbac-proxy-cm podName:a20b64fb-c649-43e7-ba2c-3238ed96520c nodeName:}" failed. No retries permitted until 2026-09-10 20:58:05.528029869 +0000 UTC m=+70.740494766 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a20b64fb-c649-43e7-ba2c-3238ed96520c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f74sn" (UID: "a20b64fb-c649-43e7-ba2c-3238ed96520c") : configmap "kube-rbac-proxy" not found
Sep 10 20:57:57.528345 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:57.528056    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:57:57.528345 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:57:57.528108    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a20b64fb-c649-43e7-ba2c-3238ed96520c-insights-runtime-extractor-tls podName:a20b64fb-c649-43e7-ba2c-3238ed96520c nodeName:}" failed. No retries permitted until 2026-09-10 20:58:05.528095084 +0000 UTC m=+70.740559982 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a20b64fb-c649-43e7-ba2c-3238ed96520c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f74sn" (UID: "a20b64fb-c649-43e7-ba2c-3238ed96520c") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:57:57.530604 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:57.530576    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-registry-tls\") pod \"image-registry-79994c7475-ddczp\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") " pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:57.577915 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:57.577891    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-24mm2\""
Sep 10 20:57:57.584977 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:57.584951    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:57.886748 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:57.886719    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79994c7475-ddczp"]
Sep 10 20:57:57.890707 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:57:57.890681    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522ae11b_496f_496b_8af2_71b79451d971.slice/crio-1ef9588b9121ece0bf62d299c07067755026ee83ec88cf6ed119849690b88bde WatchSource:0}: Error finding container 1ef9588b9121ece0bf62d299c07067755026ee83ec88cf6ed119849690b88bde: Status 404 returned error can't find the container with id 1ef9588b9121ece0bf62d299c07067755026ee83ec88cf6ed119849690b88bde
Sep 10 20:57:58.845537 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:58.845499    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79994c7475-ddczp" event={"ID":"522ae11b-496f-496b-8af2-71b79451d971","Type":"ContainerStarted","Data":"84e9db0934c5f38ed24c0bf6d53f7d9e5004c61a5bf107fe803e1e83d85f355e"}
Sep 10 20:57:58.845946 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:58.845560    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79994c7475-ddczp" event={"ID":"522ae11b-496f-496b-8af2-71b79451d971","Type":"ContainerStarted","Data":"1ef9588b9121ece0bf62d299c07067755026ee83ec88cf6ed119849690b88bde"}
Sep 10 20:57:58.845946 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:58.845663    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:57:58.880617 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:57:58.880570    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79994c7475-ddczp" podStartSLOduration=17.880556025 podStartE2EDuration="17.880556025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:58.879303513 +0000 UTC m=+64.091768432" watchObservedRunningTime="2026-09-10 20:57:58.880556025 +0000 UTC m=+64.093020943"
Sep 10 20:58:00.247807 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:00.247766    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27f82b26-5156-4cbe-ac0c-23a386fa331b-metrics-certs\") pod \"network-metrics-daemon-p9wtc\" (UID: \"27f82b26-5156-4cbe-ac0c-23a386fa331b\") " pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:58:00.248249 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:00.247815    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f720dd2d-8c6a-4365-986a-68bbf39853cb-original-pull-secret\") pod \"global-pull-secret-syncer-sm62m\" (UID: \"f720dd2d-8c6a-4365-986a-68bbf39853cb\") " pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:58:00.253295 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:00.253276    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 20:58:00.254090 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:00.254074    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 20:58:00.260740 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:00.260694    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f720dd2d-8c6a-4365-986a-68bbf39853cb-original-pull-secret\") pod \"global-pull-secret-syncer-sm62m\" (UID: \"f720dd2d-8c6a-4365-986a-68bbf39853cb\") " pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:58:00.260844 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:00.260699    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27f82b26-5156-4cbe-ac0c-23a386fa331b-metrics-certs\") pod \"network-metrics-daemon-p9wtc\" (UID: \"27f82b26-5156-4cbe-ac0c-23a386fa331b\") " pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:58:00.345351 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:00.345329    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm62m"
Sep 10 20:58:00.348610 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:00.348593    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4v6g\" (UniqueName: \"kubernetes.io/projected/c3c750a9-6302-4266-9b0d-5a681ccdf76d-kube-api-access-t4v6g\") pod \"network-check-target-j6qrf\" (UID: \"c3c750a9-6302-4266-9b0d-5a681ccdf76d\") " pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:58:00.351913 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:00.351894    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 20:58:00.356588 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:00.356569    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zr4bv\""
Sep 10 20:58:00.360726 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:00.360710    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 20:58:00.365008 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:00.364993    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9wtc"
Sep 10 20:58:00.371233 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:00.371217    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4v6g\" (UniqueName: \"kubernetes.io/projected/c3c750a9-6302-4266-9b0d-5a681ccdf76d-kube-api-access-t4v6g\") pod \"network-check-target-j6qrf\" (UID: \"c3c750a9-6302-4266-9b0d-5a681ccdf76d\") " pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:58:00.651417 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:00.651391    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sm62m"]
Sep 10 20:58:00.654929 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:58:00.654904    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf720dd2d_8c6a_4365_986a_68bbf39853cb.slice/crio-02d275f1648b755b4d3813fda1c213da1abfa43960edaf38865862b7c7a06c22 WatchSource:0}: Error finding container 02d275f1648b755b4d3813fda1c213da1abfa43960edaf38865862b7c7a06c22: Status 404 returned error can't find the container with id 02d275f1648b755b4d3813fda1c213da1abfa43960edaf38865862b7c7a06c22
Sep 10 20:58:00.663142 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:00.663117    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gncgs\""
Sep 10 20:58:00.671191 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:00.671173    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:58:00.680833 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:00.680810    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p9wtc"]
Sep 10 20:58:00.686122 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:58:00.686099    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f82b26_5156_4cbe_ac0c_23a386fa331b.slice/crio-1b13dfd688091a64707500ce16760464b5cd0beaa2ddb545ca8d32a7afa0cb59 WatchSource:0}: Error finding container 1b13dfd688091a64707500ce16760464b5cd0beaa2ddb545ca8d32a7afa0cb59: Status 404 returned error can't find the container with id 1b13dfd688091a64707500ce16760464b5cd0beaa2ddb545ca8d32a7afa0cb59
Sep 10 20:58:00.852598 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:00.852574    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p9wtc" event={"ID":"27f82b26-5156-4cbe-ac0c-23a386fa331b","Type":"ContainerStarted","Data":"1b13dfd688091a64707500ce16760464b5cd0beaa2ddb545ca8d32a7afa0cb59"}
Sep 10 20:58:00.853629 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:00.853611    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sm62m" event={"ID":"f720dd2d-8c6a-4365-986a-68bbf39853cb","Type":"ContainerStarted","Data":"02d275f1648b755b4d3813fda1c213da1abfa43960edaf38865862b7c7a06c22"}
Sep 10 20:58:00.982145 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:00.982096    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j6qrf"]
Sep 10 20:58:00.985545 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:58:00.985519    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c750a9_6302_4266_9b0d_5a681ccdf76d.slice/crio-a51312bd3705e4d1dabb5c5b16d3d4762ba1b786954e863ab3a82cf523ba872b WatchSource:0}: Error finding container a51312bd3705e4d1dabb5c5b16d3d4762ba1b786954e863ab3a82cf523ba872b: Status 404 returned error can't find the container with id a51312bd3705e4d1dabb5c5b16d3d4762ba1b786954e863ab3a82cf523ba872b
Sep 10 20:58:01.859411 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:01.859339    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j6qrf" event={"ID":"c3c750a9-6302-4266-9b0d-5a681ccdf76d","Type":"ContainerStarted","Data":"a51312bd3705e4d1dabb5c5b16d3d4762ba1b786954e863ab3a82cf523ba872b"}
Sep 10 20:58:02.865730 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:02.865696    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p9wtc" event={"ID":"27f82b26-5156-4cbe-ac0c-23a386fa331b","Type":"ContainerStarted","Data":"35635e0dbc3662156d46db23febac1054aec0df33da78ca73ef0f33acab7007f"}
Sep 10 20:58:02.866305 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:02.865753    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p9wtc" event={"ID":"27f82b26-5156-4cbe-ac0c-23a386fa331b","Type":"ContainerStarted","Data":"fdb5302de36fce3c907befd6ddb2bee275f05a0dcc0f752fd62f256fc092a60b"}
Sep 10 20:58:02.895164 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:02.895099    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-p9wtc" podStartSLOduration=65.479286313 podStartE2EDuration="1m6.895080959s" totalImagesPullingTime="1.415794646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:02.893840864 +0000 UTC m=+68.106305790" watchObservedRunningTime="2026-09-10 20:58:02.895080959 +0000 UTC m=+68.107545881"
Sep 10 20:58:03.068864 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:03.068833    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7afb1a12-ba19-4437-8444-efd2b3a5de37-metrics-tls\") pod \"dns-default-lwhlk\" (UID: \"7afb1a12-ba19-4437-8444-efd2b3a5de37\") " pod="openshift-dns/dns-default-lwhlk"
Sep 10 20:58:03.072240 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:03.072216    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7afb1a12-ba19-4437-8444-efd2b3a5de37-metrics-tls\") pod \"dns-default-lwhlk\" (UID: \"7afb1a12-ba19-4437-8444-efd2b3a5de37\") " pod="openshift-dns/dns-default-lwhlk"
Sep 10 20:58:03.327797 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:03.327764    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jw26g\""
Sep 10 20:58:03.335585 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:03.335555    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lwhlk"
Sep 10 20:58:05.589136 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:05.589104    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a20b64fb-c649-43e7-ba2c-3238ed96520c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:58:05.589579 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:05.589167    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a20b64fb-c649-43e7-ba2c-3238ed96520c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:58:05.589579 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:05.589561    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a20b64fb-c649-43e7-ba2c-3238ed96520c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:58:05.591613 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:05.591586    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a20b64fb-c649-43e7-ba2c-3238ed96520c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f74sn\" (UID: \"a20b64fb-c649-43e7-ba2c-3238ed96520c\") " pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:58:05.684931 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:05.684870    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b4777\""
Sep 10 20:58:05.690262 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:05.690233    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f74sn"
Sep 10 20:58:05.879021 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:05.878936    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j6qrf" event={"ID":"c3c750a9-6302-4266-9b0d-5a681ccdf76d","Type":"ContainerStarted","Data":"eac84c0dba4f081312a3d706a3329eaed60267b715b4687b3825f2a51c131fd6"}
Sep 10 20:58:05.879164 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:05.879063    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:58:05.880601 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:05.880573    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sm62m" event={"ID":"f720dd2d-8c6a-4365-986a-68bbf39853cb","Type":"ContainerStarted","Data":"d9cb669cd4faa4c5fad1c30568158c0f627aeb551f23767d8046b484374fc024"}
Sep 10 20:58:05.899055 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:05.899038    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lwhlk"]
Sep 10 20:58:05.901115 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:58:05.901093    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7afb1a12_ba19_4437_8444_efd2b3a5de37.slice/crio-0cc45493cacf83e4c3cd51197559d4d8317c9879925e360b2ccddd7200c3ecf4 WatchSource:0}: Error finding container 0cc45493cacf83e4c3cd51197559d4d8317c9879925e360b2ccddd7200c3ecf4: Status 404 returned error can't find the container with id 0cc45493cacf83e4c3cd51197559d4d8317c9879925e360b2ccddd7200c3ecf4
Sep 10 20:58:05.960352 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:05.960316    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j6qrf" podStartSLOduration=65.329201825 podStartE2EDuration="1m9.96030384s" totalImagesPullingTime="4.631102015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:05.910883225 +0000 UTC m=+71.123348145" watchObservedRunningTime="2026-09-10 20:58:05.96030384 +0000 UTC m=+71.172768777"
Sep 10 20:58:06.027687 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:06.027638    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sm62m" podStartSLOduration=65.060690129 podStartE2EDuration="1m10.027619723s" totalImagesPullingTime="4.966929594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:05.973089491 +0000 UTC m=+71.185554409" watchObservedRunningTime="2026-09-10 20:58:06.027619723 +0000 UTC m=+71.240084645"
Sep 10 20:58:06.027812 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:06.027773    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f74sn"]
Sep 10 20:58:06.887779 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:06.887739    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f74sn" event={"ID":"a20b64fb-c649-43e7-ba2c-3238ed96520c","Type":"ContainerStarted","Data":"551f44fb2d1d48f0084cfcb0062327b53fa30ff9b66b45a7f4f6b3f51a06a7bc"}
Sep 10 20:58:06.888217 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:06.887814    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f74sn" event={"ID":"a20b64fb-c649-43e7-ba2c-3238ed96520c","Type":"ContainerStarted","Data":"167dad50f4bf5d322a51b309b91c9086e5bb87158e1170998cce83d053c3076e"}
Sep 10 20:58:06.889119 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:06.889086    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lwhlk" event={"ID":"7afb1a12-ba19-4437-8444-efd2b3a5de37","Type":"ContainerStarted","Data":"0cc45493cacf83e4c3cd51197559d4d8317c9879925e360b2ccddd7200c3ecf4"}
Sep 10 20:58:07.894389 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:07.894350    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f74sn" event={"ID":"a20b64fb-c649-43e7-ba2c-3238ed96520c","Type":"ContainerStarted","Data":"b0a7f5b778d56e1e1e9fd3327a7b9ddfc0e68af7028e9c76715497b44804489d"}
Sep 10 20:58:07.896530 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:07.896504    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lwhlk" event={"ID":"7afb1a12-ba19-4437-8444-efd2b3a5de37","Type":"ContainerStarted","Data":"b04aa25d13d2e14114b0aab7291f9a0d5cce3e86550be6acbd42857408188a0a"}
Sep 10 20:58:07.896679 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:07.896554    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lwhlk" event={"ID":"7afb1a12-ba19-4437-8444-efd2b3a5de37","Type":"ContainerStarted","Data":"d842bc90c6bfab505eda95e0cec9e9a1508715a06efb8d57260531db881f4fad"}
Sep 10 20:58:07.896679 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:07.896631    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lwhlk"
Sep 10 20:58:07.941052 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:07.941002    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lwhlk" podStartSLOduration=35.37658949 podStartE2EDuration="36.940967523s" totalImagesPullingTime="1.564378033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:07.940551247 +0000 UTC m=+73.153016165" watchObservedRunningTime="2026-09-10 20:58:07.940967523 +0000 UTC m=+73.153432442"
Sep 10 20:58:08.902282 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:08.902247    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f74sn" event={"ID":"a20b64fb-c649-43e7-ba2c-3238ed96520c","Type":"ContainerStarted","Data":"8f67dd957fb5cea4dd02af145b60b70838d380b0f148b7de151b37d09e9f5be8"}
Sep 10 20:58:14.873965 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:14.873910    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f74sn" podStartSLOduration=23.602785076 podStartE2EDuration="25.873893328s" totalImagesPullingTime="2.271108252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:57:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:08.939200789 +0000 UTC m=+74.151665707" watchObservedRunningTime="2026-09-10 20:58:14.873893328 +0000 UTC m=+80.086358246"
Sep 10 20:58:14.874357 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:14.874054    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58885996b8-sllmw"]
Sep 10 20:58:14.877283 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:14.877253    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58885996b8-sllmw"
Sep 10 20:58:14.880720 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:14.880694    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 10 20:58:14.880720 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:14.880706    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 20:58:14.880884 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:14.880739    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 20:58:14.881869 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:14.881852    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 20:58:14.882087 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:14.882073    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-w9jhx\""
Sep 10 20:58:14.906185 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:14.906165    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58885996b8-sllmw"]
Sep 10 20:58:14.949971 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:14.949947    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fd1cfddd-e95d-4b19-90fc-a95c2533f021-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58885996b8-sllmw\" (UID: \"fd1cfddd-e95d-4b19-90fc-a95c2533f021\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58885996b8-sllmw"
Sep 10 20:58:14.950078 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:14.949998    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t676g\" (UniqueName: \"kubernetes.io/projected/fd1cfddd-e95d-4b19-90fc-a95c2533f021-kube-api-access-t676g\") pod \"managed-serviceaccount-addon-agent-58885996b8-sllmw\" (UID: \"fd1cfddd-e95d-4b19-90fc-a95c2533f021\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58885996b8-sllmw"
Sep 10 20:58:14.957929 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:14.957907    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79994c7475-ddczp"]
Sep 10 20:58:14.962357 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:14.962323    2572 patch_prober.go:28] interesting pod/image-registry-79994c7475-ddczp container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 10 20:58:14.962453 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:14.962369    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-79994c7475-ddczp" podUID="522ae11b-496f-496b-8af2-71b79451d971" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 20:58:15.043939 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.043915    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5776c87579-ddfvx"]
Sep 10 20:58:15.046807 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.046792    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.050342 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.050322    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fd1cfddd-e95d-4b19-90fc-a95c2533f021-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58885996b8-sllmw\" (UID: \"fd1cfddd-e95d-4b19-90fc-a95c2533f021\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58885996b8-sllmw"
Sep 10 20:58:15.050445 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.050349    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t676g\" (UniqueName: \"kubernetes.io/projected/fd1cfddd-e95d-4b19-90fc-a95c2533f021-kube-api-access-t676g\") pod \"managed-serviceaccount-addon-agent-58885996b8-sllmw\" (UID: \"fd1cfddd-e95d-4b19-90fc-a95c2533f021\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58885996b8-sllmw"
Sep 10 20:58:15.052861 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.052842    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fd1cfddd-e95d-4b19-90fc-a95c2533f021-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58885996b8-sllmw\" (UID: \"fd1cfddd-e95d-4b19-90fc-a95c2533f021\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58885996b8-sllmw"
Sep 10 20:58:15.077618 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.077594    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t676g\" (UniqueName: \"kubernetes.io/projected/fd1cfddd-e95d-4b19-90fc-a95c2533f021-kube-api-access-t676g\") pod \"managed-serviceaccount-addon-agent-58885996b8-sllmw\" (UID: \"fd1cfddd-e95d-4b19-90fc-a95c2533f021\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58885996b8-sllmw"
Sep 10 20:58:15.110641 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.110614    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5776c87579-ddfvx"]
Sep 10 20:58:15.150622 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.150560    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/16069e45-d22c-453e-96b2-34941e28b05c-bound-sa-token\") pod \"image-registry-5776c87579-ddfvx\" (UID: \"16069e45-d22c-453e-96b2-34941e28b05c\") " pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.150622 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.150598    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mffng\" (UniqueName: \"kubernetes.io/projected/16069e45-d22c-453e-96b2-34941e28b05c-kube-api-access-mffng\") pod \"image-registry-5776c87579-ddfvx\" (UID: \"16069e45-d22c-453e-96b2-34941e28b05c\") " pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.150622 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.150617    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/16069e45-d22c-453e-96b2-34941e28b05c-registry-tls\") pod \"image-registry-5776c87579-ddfvx\" (UID: \"16069e45-d22c-453e-96b2-34941e28b05c\") " pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.150768 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.150671    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/16069e45-d22c-453e-96b2-34941e28b05c-image-registry-private-configuration\") pod \"image-registry-5776c87579-ddfvx\" (UID: \"16069e45-d22c-453e-96b2-34941e28b05c\") " pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.150768 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.150702    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/16069e45-d22c-453e-96b2-34941e28b05c-ca-trust-extracted\") pod \"image-registry-5776c87579-ddfvx\" (UID: \"16069e45-d22c-453e-96b2-34941e28b05c\") " pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.150768 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.150750    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/16069e45-d22c-453e-96b2-34941e28b05c-trusted-ca\") pod \"image-registry-5776c87579-ddfvx\" (UID: \"16069e45-d22c-453e-96b2-34941e28b05c\") " pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.150858 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.150797    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/16069e45-d22c-453e-96b2-34941e28b05c-registry-certificates\") pod \"image-registry-5776c87579-ddfvx\" (UID: \"16069e45-d22c-453e-96b2-34941e28b05c\") " pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.150858 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.150816    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/16069e45-d22c-453e-96b2-34941e28b05c-installation-pull-secrets\") pod \"image-registry-5776c87579-ddfvx\" (UID: \"16069e45-d22c-453e-96b2-34941e28b05c\") " pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.198733 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.198709    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58885996b8-sllmw"
Sep 10 20:58:15.251395 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.251372    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/16069e45-d22c-453e-96b2-34941e28b05c-registry-certificates\") pod \"image-registry-5776c87579-ddfvx\" (UID: \"16069e45-d22c-453e-96b2-34941e28b05c\") " pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.251514 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.251405    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/16069e45-d22c-453e-96b2-34941e28b05c-installation-pull-secrets\") pod \"image-registry-5776c87579-ddfvx\" (UID: \"16069e45-d22c-453e-96b2-34941e28b05c\") " pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.251514 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.251432    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/16069e45-d22c-453e-96b2-34941e28b05c-bound-sa-token\") pod \"image-registry-5776c87579-ddfvx\" (UID: \"16069e45-d22c-453e-96b2-34941e28b05c\") " pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.251514 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.251453    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mffng\" (UniqueName: \"kubernetes.io/projected/16069e45-d22c-453e-96b2-34941e28b05c-kube-api-access-mffng\") pod \"image-registry-5776c87579-ddfvx\" (UID: \"16069e45-d22c-453e-96b2-34941e28b05c\") " pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.251514 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.251483    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/16069e45-d22c-453e-96b2-34941e28b05c-registry-tls\") pod \"image-registry-5776c87579-ddfvx\" (UID: \"16069e45-d22c-453e-96b2-34941e28b05c\") " pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.251706 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.251523    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/16069e45-d22c-453e-96b2-34941e28b05c-image-registry-private-configuration\") pod \"image-registry-5776c87579-ddfvx\" (UID: \"16069e45-d22c-453e-96b2-34941e28b05c\") " pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.251706 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.251545    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/16069e45-d22c-453e-96b2-34941e28b05c-ca-trust-extracted\") pod \"image-registry-5776c87579-ddfvx\" (UID: \"16069e45-d22c-453e-96b2-34941e28b05c\") " pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.251706 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.251574    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/16069e45-d22c-453e-96b2-34941e28b05c-trusted-ca\") pod \"image-registry-5776c87579-ddfvx\" (UID: \"16069e45-d22c-453e-96b2-34941e28b05c\") " pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.252192 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.252159    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/16069e45-d22c-453e-96b2-34941e28b05c-ca-trust-extracted\") pod \"image-registry-5776c87579-ddfvx\" (UID: \"16069e45-d22c-453e-96b2-34941e28b05c\") " pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.252697 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.252667    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/16069e45-d22c-453e-96b2-34941e28b05c-registry-certificates\") pod \"image-registry-5776c87579-ddfvx\" (UID: \"16069e45-d22c-453e-96b2-34941e28b05c\") " pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.252935 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.252898    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/16069e45-d22c-453e-96b2-34941e28b05c-trusted-ca\") pod \"image-registry-5776c87579-ddfvx\" (UID: \"16069e45-d22c-453e-96b2-34941e28b05c\") " pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.257008 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.256941    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/16069e45-d22c-453e-96b2-34941e28b05c-image-registry-private-configuration\") pod \"image-registry-5776c87579-ddfvx\" (UID: \"16069e45-d22c-453e-96b2-34941e28b05c\") " pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.257008 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.256951    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/16069e45-d22c-453e-96b2-34941e28b05c-installation-pull-secrets\") pod \"image-registry-5776c87579-ddfvx\" (UID: \"16069e45-d22c-453e-96b2-34941e28b05c\") " pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.257198 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.257175    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/16069e45-d22c-453e-96b2-34941e28b05c-registry-tls\") pod \"image-registry-5776c87579-ddfvx\" (UID: \"16069e45-d22c-453e-96b2-34941e28b05c\") " pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.270876 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.270849    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mffng\" (UniqueName: \"kubernetes.io/projected/16069e45-d22c-453e-96b2-34941e28b05c-kube-api-access-mffng\") pod \"image-registry-5776c87579-ddfvx\" (UID: \"16069e45-d22c-453e-96b2-34941e28b05c\") " pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.286858 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.286836    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/16069e45-d22c-453e-96b2-34941e28b05c-bound-sa-token\") pod \"image-registry-5776c87579-ddfvx\" (UID: \"16069e45-d22c-453e-96b2-34941e28b05c\") " pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.359208 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.359173    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.557774 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.557744    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58885996b8-sllmw"]
Sep 10 20:58:15.564051 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:58:15.564023    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd1cfddd_e95d_4b19_90fc_a95c2533f021.slice/crio-11a852f709eb98b1afdaa363fe3c2fc4231d068796d22dedf2b4a72636a99c51 WatchSource:0}: Error finding container 11a852f709eb98b1afdaa363fe3c2fc4231d068796d22dedf2b4a72636a99c51: Status 404 returned error can't find the container with id 11a852f709eb98b1afdaa363fe3c2fc4231d068796d22dedf2b4a72636a99c51
Sep 10 20:58:15.714417 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.714328    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5776c87579-ddfvx"]
Sep 10 20:58:15.718027 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:58:15.717971    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16069e45_d22c_453e_96b2_34941e28b05c.slice/crio-0c8c89101a36a8029135953cef4c4a5117c031821767789724537ccb18620930 WatchSource:0}: Error finding container 0c8c89101a36a8029135953cef4c4a5117c031821767789724537ccb18620930: Status 404 returned error can't find the container with id 0c8c89101a36a8029135953cef4c4a5117c031821767789724537ccb18620930
Sep 10 20:58:15.926517 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.926484    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5776c87579-ddfvx" event={"ID":"16069e45-d22c-453e-96b2-34941e28b05c","Type":"ContainerStarted","Data":"4e38393d72f098cff18af19789c23c346f6267a4587490df0b02b6771d7b62e7"}
Sep 10 20:58:15.926891 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.926543    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5776c87579-ddfvx" event={"ID":"16069e45-d22c-453e-96b2-34941e28b05c","Type":"ContainerStarted","Data":"0c8c89101a36a8029135953cef4c4a5117c031821767789724537ccb18620930"}
Sep 10 20:58:15.926891 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.926569    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:15.927793 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.927773    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58885996b8-sllmw" event={"ID":"fd1cfddd-e95d-4b19-90fc-a95c2533f021","Type":"ContainerStarted","Data":"11a852f709eb98b1afdaa363fe3c2fc4231d068796d22dedf2b4a72636a99c51"}
Sep 10 20:58:15.967149 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:15.967070    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5776c87579-ddfvx" podStartSLOduration=0.967054411 podStartE2EDuration="967.054411ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:15.966378793 +0000 UTC m=+81.178843711" watchObservedRunningTime="2026-09-10 20:58:15.967054411 +0000 UTC m=+81.179519329"
Sep 10 20:58:17.904728 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:17.904600    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lwhlk"
Sep 10 20:58:18.941617 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:18.941575    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58885996b8-sllmw" event={"ID":"fd1cfddd-e95d-4b19-90fc-a95c2533f021","Type":"ContainerStarted","Data":"bf7440715da7c4fdde4d537f6ebd7f1985d6085fa8634c90b35b126af8685501"}
Sep 10 20:58:18.971509 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:18.971434    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58885996b8-sllmw" podStartSLOduration=2.477328515 podStartE2EDuration="4.971417731s" totalImagesPullingTime="2.494089216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:18.971123222 +0000 UTC m=+84.183588141" watchObservedRunningTime="2026-09-10 20:58:18.971417731 +0000 UTC m=+84.183882717"
Sep 10 20:58:24.962615 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:24.962580    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:58:27.978425 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:27.978388    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-qhhck"]
Sep 10 20:58:27.984152 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:27.984136    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qhhck"
Sep 10 20:58:27.988642 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:27.988619    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 20:58:27.988642 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:27.988636    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 10 20:58:27.988826 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:27.988731    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-vm6v4\""
Sep 10 20:58:27.989592 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:27.989578    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 10 20:58:27.989843 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:27.989826    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 20:58:27.990015 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:27.989974    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 20:58:28.001716 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.001696    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-qhhck"]
Sep 10 20:58:28.037955 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.037934    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7eac4c07-da69-470e-9285-1dec8f7ae8a9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-qhhck\" (UID: \"7eac4c07-da69-470e-9285-1dec8f7ae8a9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qhhck"
Sep 10 20:58:28.038058 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.037974    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcl9w\" (UniqueName: \"kubernetes.io/projected/7eac4c07-da69-470e-9285-1dec8f7ae8a9-kube-api-access-gcl9w\") pod \"openshift-state-metrics-67dcf8c979-qhhck\" (UID: \"7eac4c07-da69-470e-9285-1dec8f7ae8a9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qhhck"
Sep 10 20:58:28.038099 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.038054    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7eac4c07-da69-470e-9285-1dec8f7ae8a9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-qhhck\" (UID: \"7eac4c07-da69-470e-9285-1dec8f7ae8a9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qhhck"
Sep 10 20:58:28.038099 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.038080    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7eac4c07-da69-470e-9285-1dec8f7ae8a9-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-qhhck\" (UID: \"7eac4c07-da69-470e-9285-1dec8f7ae8a9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qhhck"
Sep 10 20:58:28.043524 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.043498    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-j78zp"]
Sep 10 20:58:28.047184 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.047168    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.051895 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.051858    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 20:58:28.052057 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.052033    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tgnsq\""
Sep 10 20:58:28.052394 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.052374    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 20:58:28.052686 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.052672    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 20:58:28.138561 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.138539    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eae9e839-dfdb-41b5-8657-77744647bb9a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.138669 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.138575    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eae9e839-dfdb-41b5-8657-77744647bb9a-metrics-client-ca\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.138669 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.138609    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7eac4c07-da69-470e-9285-1dec8f7ae8a9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-qhhck\" (UID: \"7eac4c07-da69-470e-9285-1dec8f7ae8a9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qhhck"
Sep 10 20:58:28.138669 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.138632    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtvzg\" (UniqueName: \"kubernetes.io/projected/eae9e839-dfdb-41b5-8657-77744647bb9a-kube-api-access-vtvzg\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.138900 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.138675    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eae9e839-dfdb-41b5-8657-77744647bb9a-node-exporter-textfile\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.138900 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.138706    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eae9e839-dfdb-41b5-8657-77744647bb9a-node-exporter-tls\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.138900 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.138733    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eae9e839-dfdb-41b5-8657-77744647bb9a-node-exporter-accelerators-collector-config\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.138900 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.138805    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcl9w\" (UniqueName: \"kubernetes.io/projected/7eac4c07-da69-470e-9285-1dec8f7ae8a9-kube-api-access-gcl9w\") pod \"openshift-state-metrics-67dcf8c979-qhhck\" (UID: \"7eac4c07-da69-470e-9285-1dec8f7ae8a9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qhhck"
Sep 10 20:58:28.138900 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.138850    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eae9e839-dfdb-41b5-8657-77744647bb9a-node-exporter-wtmp\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.138900 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.138894    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eae9e839-dfdb-41b5-8657-77744647bb9a-sys\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.139226 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.138930    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7eac4c07-da69-470e-9285-1dec8f7ae8a9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-qhhck\" (UID: \"7eac4c07-da69-470e-9285-1dec8f7ae8a9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qhhck"
Sep 10 20:58:28.139226 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.138967    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7eac4c07-da69-470e-9285-1dec8f7ae8a9-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-qhhck\" (UID: \"7eac4c07-da69-470e-9285-1dec8f7ae8a9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qhhck"
Sep 10 20:58:28.139226 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.139012    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eae9e839-dfdb-41b5-8657-77744647bb9a-root\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.139677 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.139650    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7eac4c07-da69-470e-9285-1dec8f7ae8a9-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-qhhck\" (UID: \"7eac4c07-da69-470e-9285-1dec8f7ae8a9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qhhck"
Sep 10 20:58:28.141797 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.141773    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7eac4c07-da69-470e-9285-1dec8f7ae8a9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-qhhck\" (UID: \"7eac4c07-da69-470e-9285-1dec8f7ae8a9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qhhck"
Sep 10 20:58:28.142040 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.142024    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7eac4c07-da69-470e-9285-1dec8f7ae8a9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-qhhck\" (UID: \"7eac4c07-da69-470e-9285-1dec8f7ae8a9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qhhck"
Sep 10 20:58:28.157560 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.157538    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcl9w\" (UniqueName: \"kubernetes.io/projected/7eac4c07-da69-470e-9285-1dec8f7ae8a9-kube-api-access-gcl9w\") pod \"openshift-state-metrics-67dcf8c979-qhhck\" (UID: \"7eac4c07-da69-470e-9285-1dec8f7ae8a9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qhhck"
Sep 10 20:58:28.239602 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.239544    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtvzg\" (UniqueName: \"kubernetes.io/projected/eae9e839-dfdb-41b5-8657-77744647bb9a-kube-api-access-vtvzg\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.239602 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.239585    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eae9e839-dfdb-41b5-8657-77744647bb9a-node-exporter-textfile\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.239747 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.239614    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eae9e839-dfdb-41b5-8657-77744647bb9a-node-exporter-tls\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.239747 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.239641    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eae9e839-dfdb-41b5-8657-77744647bb9a-node-exporter-accelerators-collector-config\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.239747 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.239671    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eae9e839-dfdb-41b5-8657-77744647bb9a-node-exporter-wtmp\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.239747 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.239702    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eae9e839-dfdb-41b5-8657-77744647bb9a-sys\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.239939 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.239745    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eae9e839-dfdb-41b5-8657-77744647bb9a-root\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.239939 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.239769    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eae9e839-dfdb-41b5-8657-77744647bb9a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.239939 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.239800    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eae9e839-dfdb-41b5-8657-77744647bb9a-metrics-client-ca\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.239939 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.239823    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eae9e839-dfdb-41b5-8657-77744647bb9a-sys\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.239939 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.239844    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eae9e839-dfdb-41b5-8657-77744647bb9a-node-exporter-wtmp\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.239939 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.239863    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eae9e839-dfdb-41b5-8657-77744647bb9a-root\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.239939 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.239879    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eae9e839-dfdb-41b5-8657-77744647bb9a-node-exporter-textfile\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.240295 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.240203    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eae9e839-dfdb-41b5-8657-77744647bb9a-node-exporter-accelerators-collector-config\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.240479 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.240387    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eae9e839-dfdb-41b5-8657-77744647bb9a-metrics-client-ca\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.241838 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.241811    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eae9e839-dfdb-41b5-8657-77744647bb9a-node-exporter-tls\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.242033 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.242017    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eae9e839-dfdb-41b5-8657-77744647bb9a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.252737 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.252715    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtvzg\" (UniqueName: \"kubernetes.io/projected/eae9e839-dfdb-41b5-8657-77744647bb9a-kube-api-access-vtvzg\") pod \"node-exporter-j78zp\" (UID: \"eae9e839-dfdb-41b5-8657-77744647bb9a\") " pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.292711 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.292688    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qhhck"
Sep 10 20:58:28.355324 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.355296    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j78zp"
Sep 10 20:58:28.369010 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:58:28.368959    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae9e839_dfdb_41b5_8657_77744647bb9a.slice/crio-8634ecad7af9700a0e717a4a56ea43b9a7eba0aaae92fa1607832e8ab937d408 WatchSource:0}: Error finding container 8634ecad7af9700a0e717a4a56ea43b9a7eba0aaae92fa1607832e8ab937d408: Status 404 returned error can't find the container with id 8634ecad7af9700a0e717a4a56ea43b9a7eba0aaae92fa1607832e8ab937d408
Sep 10 20:58:28.582787 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.582762    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-qhhck"]
Sep 10 20:58:28.584517 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:58:28.584493    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eac4c07_da69_470e_9285_1dec8f7ae8a9.slice/crio-bb8a83c06b66b5ed72ea1e9d48a23f8f3ca664ec8ad0fca07d08f9d7b88dd347 WatchSource:0}: Error finding container bb8a83c06b66b5ed72ea1e9d48a23f8f3ca664ec8ad0fca07d08f9d7b88dd347: Status 404 returned error can't find the container with id bb8a83c06b66b5ed72ea1e9d48a23f8f3ca664ec8ad0fca07d08f9d7b88dd347
Sep 10 20:58:28.982650 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.982572    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qhhck" event={"ID":"7eac4c07-da69-470e-9285-1dec8f7ae8a9","Type":"ContainerStarted","Data":"251c811f5a1980d86cba52e2fc2830f395ca22dd652094b43344fb50878e4b7d"}
Sep 10 20:58:28.982650 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.982629    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qhhck" event={"ID":"7eac4c07-da69-470e-9285-1dec8f7ae8a9","Type":"ContainerStarted","Data":"9d0c8e88a99e1f18971340668b18f671353ae3151fa54d0c12284d9b2c49ea0d"}
Sep 10 20:58:28.983088 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.982661    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qhhck" event={"ID":"7eac4c07-da69-470e-9285-1dec8f7ae8a9","Type":"ContainerStarted","Data":"bb8a83c06b66b5ed72ea1e9d48a23f8f3ca664ec8ad0fca07d08f9d7b88dd347"}
Sep 10 20:58:28.983757 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:28.983734    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j78zp" event={"ID":"eae9e839-dfdb-41b5-8657-77744647bb9a","Type":"ContainerStarted","Data":"8634ecad7af9700a0e717a4a56ea43b9a7eba0aaae92fa1607832e8ab937d408"}
Sep 10 20:58:29.200918 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.200880    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 20:58:29.205345 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.205313    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.219966 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.219942    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 20:58:29.220166 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.219942    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 20:58:29.220166 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.219943    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 20:58:29.220166 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.219943    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 20:58:29.220166 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.219943    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 20:58:29.220514 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.220254    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 20:58:29.220514 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.220353    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 20:58:29.220514 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.220466    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9xv52\""
Sep 10 20:58:29.220674 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.220550    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 20:58:29.231649 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.231632    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 20:58:29.246157 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.246109    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/59c135ab-d11e-421c-b90f-d79c0416f909-config-out\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.246157 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.246146    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59c135ab-d11e-421c-b90f-d79c0416f909-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.246305 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.246179    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59c135ab-d11e-421c-b90f-d79c0416f909-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.246305 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.246284    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/59c135ab-d11e-421c-b90f-d79c0416f909-tls-assets\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.246418 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.246330    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.246418 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.246371    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-config-volume\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.246521 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.246429    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-web-config\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.246521 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.246463    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/59c135ab-d11e-421c-b90f-d79c0416f909-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.246521 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.246485    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr8dv\" (UniqueName: \"kubernetes.io/projected/59c135ab-d11e-421c-b90f-d79c0416f909-kube-api-access-cr8dv\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.246521 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.246513    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.246709 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.246541    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.246709 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.246572    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.246709 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.246606    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.337894 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.337865    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 20:58:29.347167 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.347135    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-config-volume\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.347288 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.347184    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-web-config\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.347288 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.347216    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/59c135ab-d11e-421c-b90f-d79c0416f909-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.347288 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.347238    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr8dv\" (UniqueName: \"kubernetes.io/projected/59c135ab-d11e-421c-b90f-d79c0416f909-kube-api-access-cr8dv\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.347288 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.347267    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.347507 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.347294    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.347507 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.347326    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.347507 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.347356    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.347507 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.347385    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/59c135ab-d11e-421c-b90f-d79c0416f909-config-out\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.347507 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.347406    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59c135ab-d11e-421c-b90f-d79c0416f909-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.347507 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.347439    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59c135ab-d11e-421c-b90f-d79c0416f909-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.347815 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.347525    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/59c135ab-d11e-421c-b90f-d79c0416f909-tls-assets\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.347815 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.347567    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.348312 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:58:29.348286    2572 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 10 20:58:29.348409 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:58:29.348373    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-main-tls podName:59c135ab-d11e-421c-b90f-d79c0416f909 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:29.848351787 +0000 UTC m=+95.060816704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "59c135ab-d11e-421c-b90f-d79c0416f909") : secret "alertmanager-main-tls" not found
Sep 10 20:58:29.356787 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.355886    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/59c135ab-d11e-421c-b90f-d79c0416f909-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.356787 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.356438    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59c135ab-d11e-421c-b90f-d79c0416f909-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.356787 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.356506    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59c135ab-d11e-421c-b90f-d79c0416f909-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.362238 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.359107    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/59c135ab-d11e-421c-b90f-d79c0416f909-tls-assets\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.362238 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.361402    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-web-config\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.364732 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.364148    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.364732 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.364185    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-config-volume\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.366555 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.366492    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.366555 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.366528    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.368825 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.368786    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/59c135ab-d11e-421c-b90f-d79c0416f909-config-out\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.368931 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.368912    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.388107 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.388083    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr8dv\" (UniqueName: \"kubernetes.io/projected/59c135ab-d11e-421c-b90f-d79c0416f909-kube-api-access-cr8dv\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.851287 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.851257    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.853501 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.853478    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:29.988796 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.988766    2572 generic.go:356] "Generic (PLEG): container finished" podID="eae9e839-dfdb-41b5-8657-77744647bb9a" containerID="561c40dd5760c42381c93e3d7db51398587678c171b4d7234f3a92628cffbd49" exitCode=0
Sep 10 20:58:29.989178 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.988851    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j78zp" event={"ID":"eae9e839-dfdb-41b5-8657-77744647bb9a","Type":"ContainerDied","Data":"561c40dd5760c42381c93e3d7db51398587678c171b4d7234f3a92628cffbd49"}
Sep 10 20:58:29.991378 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:29.991340    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qhhck" event={"ID":"7eac4c07-da69-470e-9285-1dec8f7ae8a9","Type":"ContainerStarted","Data":"1b626cb98792376cef7cf07a66eceebb964e320102fb17c6c27972e305779b39"}
Sep 10 20:58:30.063960 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:30.063918    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-qhhck" podStartSLOduration=1.941580678 podStartE2EDuration="3.063902949s" totalImagesPullingTime="1.122322271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:30.063079946 +0000 UTC m=+95.275544866" watchObservedRunningTime="2026-09-10 20:58:30.063902949 +0000 UTC m=+95.276367867"
Sep 10 20:58:30.116897 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:30.116876    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:58:30.432766 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:30.432744    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 20:58:30.434928 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:58:30.434899    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c135ab_d11e_421c_b90f_d79c0416f909.slice/crio-8b5224e45253717da33f0687250e09edd7166843eb29e76dfd00cc44bdd6f0d0 WatchSource:0}: Error finding container 8b5224e45253717da33f0687250e09edd7166843eb29e76dfd00cc44bdd6f0d0: Status 404 returned error can't find the container with id 8b5224e45253717da33f0687250e09edd7166843eb29e76dfd00cc44bdd6f0d0
Sep 10 20:58:30.999260 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:30.999231    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59c135ab-d11e-421c-b90f-d79c0416f909","Type":"ContainerStarted","Data":"8b5224e45253717da33f0687250e09edd7166843eb29e76dfd00cc44bdd6f0d0"}
Sep 10 20:58:31.001615 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:31.001590    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j78zp" event={"ID":"eae9e839-dfdb-41b5-8657-77744647bb9a","Type":"ContainerStarted","Data":"534743ba96681de3036eda20750eb6818830acdba156c3e29eccc71cce4a308d"}
Sep 10 20:58:31.001718 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:31.001639    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j78zp" event={"ID":"eae9e839-dfdb-41b5-8657-77744647bb9a","Type":"ContainerStarted","Data":"0ba7d5916ec230a33701a35721de527ee4ae3ca5d34c3884bc2d1adc57b92b24"}
Sep 10 20:58:31.050191 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:31.050146    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-j78zp" podStartSLOduration=2.561107992 podStartE2EDuration="4.050125709s" totalImagesPullingTime="1.415731208s" totalInitContainerRuntime="73.286509ms" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:31.049275507 +0000 UTC m=+96.261740427" watchObservedRunningTime="2026-09-10 20:58:31.050125709 +0000 UTC m=+96.262590628"
Sep 10 20:58:32.007609 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:32.007531    2572 generic.go:356] "Generic (PLEG): container finished" podID="59c135ab-d11e-421c-b90f-d79c0416f909" containerID="2c6cb3af368d818c1f2d134599721da9e9fa054d835b4865f6a943eac6b0b25a" exitCode=0
Sep 10 20:58:32.008031 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:32.007628    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59c135ab-d11e-421c-b90f-d79c0416f909","Type":"ContainerDied","Data":"2c6cb3af368d818c1f2d134599721da9e9fa054d835b4865f6a943eac6b0b25a"}
Sep 10 20:58:32.568266 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:32.568236    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-pd69z"]
Sep 10 20:58:32.571568 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:32.571547    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pd69z"
Sep 10 20:58:32.582582 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:32.582561    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-wnn5f\""
Sep 10 20:58:32.582977 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:32.582961    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 10 20:58:32.591487 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:32.591463    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-pd69z"]
Sep 10 20:58:32.672727 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:32.672698    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a7abb943-7fef-4fcd-9454-728be3c193ae-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-pd69z\" (UID: \"a7abb943-7fef-4fcd-9454-728be3c193ae\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pd69z"
Sep 10 20:58:32.672854 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:32.672769    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbmxc\" (UniqueName: \"kubernetes.io/projected/a7abb943-7fef-4fcd-9454-728be3c193ae-kube-api-access-kbmxc\") pod \"monitoring-plugin-5d85778b9d-pd69z\" (UID: \"a7abb943-7fef-4fcd-9454-728be3c193ae\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pd69z"
Sep 10 20:58:32.773763 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:32.773731    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbmxc\" (UniqueName: \"kubernetes.io/projected/a7abb943-7fef-4fcd-9454-728be3c193ae-kube-api-access-kbmxc\") pod \"monitoring-plugin-5d85778b9d-pd69z\" (UID: \"a7abb943-7fef-4fcd-9454-728be3c193ae\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pd69z"
Sep 10 20:58:32.773899 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:32.773801    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a7abb943-7fef-4fcd-9454-728be3c193ae-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-pd69z\" (UID: \"a7abb943-7fef-4fcd-9454-728be3c193ae\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pd69z"
Sep 10 20:58:32.776535 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:32.776507    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a7abb943-7fef-4fcd-9454-728be3c193ae-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-pd69z\" (UID: \"a7abb943-7fef-4fcd-9454-728be3c193ae\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pd69z"
Sep 10 20:58:32.796355 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:32.796332    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbmxc\" (UniqueName: \"kubernetes.io/projected/a7abb943-7fef-4fcd-9454-728be3c193ae-kube-api-access-kbmxc\") pod \"monitoring-plugin-5d85778b9d-pd69z\" (UID: \"a7abb943-7fef-4fcd-9454-728be3c193ae\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pd69z"
Sep 10 20:58:32.883319 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:32.883263    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pd69z"
Sep 10 20:58:33.179028 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:33.178933    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-pd69z"]
Sep 10 20:58:33.288701 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:58:33.288666    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7abb943_7fef_4fcd_9454_728be3c193ae.slice/crio-5467888526a77c3bc0e248312eb5f1636e06d0beed76bb354f57ed19bc30e45b WatchSource:0}: Error finding container 5467888526a77c3bc0e248312eb5f1636e06d0beed76bb354f57ed19bc30e45b: Status 404 returned error can't find the container with id 5467888526a77c3bc0e248312eb5f1636e06d0beed76bb354f57ed19bc30e45b
Sep 10 20:58:34.017438 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:34.017400    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pd69z" event={"ID":"a7abb943-7fef-4fcd-9454-728be3c193ae","Type":"ContainerStarted","Data":"5467888526a77c3bc0e248312eb5f1636e06d0beed76bb354f57ed19bc30e45b"}
Sep 10 20:58:34.021057 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:34.021029    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59c135ab-d11e-421c-b90f-d79c0416f909","Type":"ContainerStarted","Data":"1b9877a8a758e906b032561e784b8581d29ed1b00f707d39063a148760952c7f"}
Sep 10 20:58:34.021196 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:34.021084    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59c135ab-d11e-421c-b90f-d79c0416f909","Type":"ContainerStarted","Data":"f88498262d80cde8890c38237d30553cf27fd489136b532c7d8a717ab5ae3d54"}
Sep 10 20:58:34.021196 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:34.021117    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59c135ab-d11e-421c-b90f-d79c0416f909","Type":"ContainerStarted","Data":"724baaf284d888b95097d3efa7e2ce8bd849a080eb5c243e1ce9520ac69452a0"}
Sep 10 20:58:34.021196 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:34.021145    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59c135ab-d11e-421c-b90f-d79c0416f909","Type":"ContainerStarted","Data":"3508ec7c7fe73333ef89f15bc3e06a6f809dd104794658fb4bfb8d725cb8bee1"}
Sep 10 20:58:34.021196 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:34.021174    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59c135ab-d11e-421c-b90f-d79c0416f909","Type":"ContainerStarted","Data":"c300ceae13478b6f92ff8aced29fb2b780d7b9fecc8e0353beec1af02675f213"}
Sep 10 20:58:36.035239 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:36.035203    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59c135ab-d11e-421c-b90f-d79c0416f909","Type":"ContainerStarted","Data":"13909b481bbc1bc738abe832ed39943890ea6c4552c3ce8b775eb872184f5ad2"}
Sep 10 20:58:36.036760 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:36.036737    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pd69z" event={"ID":"a7abb943-7fef-4fcd-9454-728be3c193ae","Type":"ContainerStarted","Data":"efe97ee92cd6436174e6866658c122c6b5b33465ef9e0760f1d14684d104ac1f"}
Sep 10 20:58:36.036904 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:36.036786    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pd69z"
Sep 10 20:58:36.041511 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:36.041489    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pd69z"
Sep 10 20:58:36.077277 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:36.077235    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.932930711 podStartE2EDuration="7.077222442s" totalImagesPullingTime="4.128886418s" totalInitContainerRuntime="15.405313ms" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:36.075603556 +0000 UTC m=+101.288068479" watchObservedRunningTime="2026-09-10 20:58:36.077222442 +0000 UTC m=+101.289687360"
Sep 10 20:58:36.101497 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:36.101444    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-pd69z" podStartSLOduration=2.153853832 podStartE2EDuration="4.101427539s" totalImagesPullingTime="1.947573707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:36.100658823 +0000 UTC m=+101.313123742" watchObservedRunningTime="2026-09-10 20:58:36.101427539 +0000 UTC m=+101.313892459"
Sep 10 20:58:36.437159 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:36.437097    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-f5qj7"]
Sep 10 20:58:36.467759 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:36.467734    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-f5qj7"]
Sep 10 20:58:36.467882 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:36.467803    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-f5qj7"
Sep 10 20:58:36.470362 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:36.470339    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 20:58:36.470362 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:36.470346    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 20:58:36.470663 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:36.470645    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-szmdj\""
Sep 10 20:58:36.604651 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:36.604625    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc8qm\" (UniqueName: \"kubernetes.io/projected/f0cb65f4-5aa1-41bd-9584-e3d440e4cb79-kube-api-access-hc8qm\") pod \"downloads-76f99584f4-f5qj7\" (UID: \"f0cb65f4-5aa1-41bd-9584-e3d440e4cb79\") " pod="openshift-console/downloads-76f99584f4-f5qj7"
Sep 10 20:58:36.706018 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:36.705924    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc8qm\" (UniqueName: \"kubernetes.io/projected/f0cb65f4-5aa1-41bd-9584-e3d440e4cb79-kube-api-access-hc8qm\") pod \"downloads-76f99584f4-f5qj7\" (UID: \"f0cb65f4-5aa1-41bd-9584-e3d440e4cb79\") " pod="openshift-console/downloads-76f99584f4-f5qj7"
Sep 10 20:58:36.715464 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:36.715437    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc8qm\" (UniqueName: \"kubernetes.io/projected/f0cb65f4-5aa1-41bd-9584-e3d440e4cb79-kube-api-access-hc8qm\") pod \"downloads-76f99584f4-f5qj7\" (UID: \"f0cb65f4-5aa1-41bd-9584-e3d440e4cb79\") " pod="openshift-console/downloads-76f99584f4-f5qj7"
Sep 10 20:58:36.777214 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:36.777185    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-f5qj7"
Sep 10 20:58:36.892025 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:36.892002    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j6qrf"
Sep 10 20:58:36.936139 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:36.936112    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5776c87579-ddfvx"
Sep 10 20:58:37.142908 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:37.142882    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-f5qj7"]
Sep 10 20:58:37.145962 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:58:37.145935    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0cb65f4_5aa1_41bd_9584_e3d440e4cb79.slice/crio-53e344f2faf3d4b8f65f5d3f45989231cacd4137301033f4d5b4e409d3c00338 WatchSource:0}: Error finding container 53e344f2faf3d4b8f65f5d3f45989231cacd4137301033f4d5b4e409d3c00338: Status 404 returned error can't find the container with id 53e344f2faf3d4b8f65f5d3f45989231cacd4137301033f4d5b4e409d3c00338
Sep 10 20:58:38.045203 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:38.045163    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-f5qj7" event={"ID":"f0cb65f4-5aa1-41bd-9584-e3d440e4cb79","Type":"ContainerStarted","Data":"53e344f2faf3d4b8f65f5d3f45989231cacd4137301033f4d5b4e409d3c00338"}
Sep 10 20:58:39.977143 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:39.977098    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-79994c7475-ddczp" podUID="522ae11b-496f-496b-8af2-71b79451d971" containerName="registry" containerID="cri-o://84e9db0934c5f38ed24c0bf6d53f7d9e5004c61a5bf107fe803e1e83d85f355e" gracePeriod=30
Sep 10 20:58:40.233952 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.233897    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:58:40.335342 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.335319    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-registry-tls\") pod \"522ae11b-496f-496b-8af2-71b79451d971\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") "
Sep 10 20:58:40.335503 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.335356    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-bound-sa-token\") pod \"522ae11b-496f-496b-8af2-71b79451d971\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") "
Sep 10 20:58:40.335503 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.335391    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/522ae11b-496f-496b-8af2-71b79451d971-trusted-ca\") pod \"522ae11b-496f-496b-8af2-71b79451d971\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") "
Sep 10 20:58:40.335503 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.335420    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/522ae11b-496f-496b-8af2-71b79451d971-ca-trust-extracted\") pod \"522ae11b-496f-496b-8af2-71b79451d971\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") "
Sep 10 20:58:40.335503 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.335496    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/522ae11b-496f-496b-8af2-71b79451d971-image-registry-private-configuration\") pod \"522ae11b-496f-496b-8af2-71b79451d971\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") "
Sep 10 20:58:40.335664 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.335529    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7q8r\" (UniqueName: \"kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-kube-api-access-x7q8r\") pod \"522ae11b-496f-496b-8af2-71b79451d971\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") "
Sep 10 20:58:40.335664 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.335564    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/522ae11b-496f-496b-8af2-71b79451d971-installation-pull-secrets\") pod \"522ae11b-496f-496b-8af2-71b79451d971\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") "
Sep 10 20:58:40.335664 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.335591    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/522ae11b-496f-496b-8af2-71b79451d971-registry-certificates\") pod \"522ae11b-496f-496b-8af2-71b79451d971\" (UID: \"522ae11b-496f-496b-8af2-71b79451d971\") "
Sep 10 20:58:40.336999 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.336559    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522ae11b-496f-496b-8af2-71b79451d971-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "522ae11b-496f-496b-8af2-71b79451d971" (UID: "522ae11b-496f-496b-8af2-71b79451d971"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:58:40.336999 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.336600    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522ae11b-496f-496b-8af2-71b79451d971-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "522ae11b-496f-496b-8af2-71b79451d971" (UID: "522ae11b-496f-496b-8af2-71b79451d971"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:58:40.339176 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.339111    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "522ae11b-496f-496b-8af2-71b79451d971" (UID: "522ae11b-496f-496b-8af2-71b79451d971"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:58:40.339176 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.339160    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/522ae11b-496f-496b-8af2-71b79451d971-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "522ae11b-496f-496b-8af2-71b79451d971" (UID: "522ae11b-496f-496b-8af2-71b79451d971"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:58:40.339717 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.339690    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/522ae11b-496f-496b-8af2-71b79451d971-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "522ae11b-496f-496b-8af2-71b79451d971" (UID: "522ae11b-496f-496b-8af2-71b79451d971"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:58:40.339864 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.339841    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "522ae11b-496f-496b-8af2-71b79451d971" (UID: "522ae11b-496f-496b-8af2-71b79451d971"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:58:40.340611 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.340574    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-kube-api-access-x7q8r" (OuterVolumeSpecName: "kube-api-access-x7q8r") pod "522ae11b-496f-496b-8af2-71b79451d971" (UID: "522ae11b-496f-496b-8af2-71b79451d971"). InnerVolumeSpecName "kube-api-access-x7q8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:58:40.346459 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.346431    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522ae11b-496f-496b-8af2-71b79451d971-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "522ae11b-496f-496b-8af2-71b79451d971" (UID: "522ae11b-496f-496b-8af2-71b79451d971"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:58:40.437175 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.437150    2572 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/522ae11b-496f-496b-8af2-71b79451d971-image-registry-private-configuration\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:58:40.437175 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.437176    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7q8r\" (UniqueName: \"kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-kube-api-access-x7q8r\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:58:40.437338 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.437187    2572 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/522ae11b-496f-496b-8af2-71b79451d971-installation-pull-secrets\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:58:40.437338 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.437195    2572 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/522ae11b-496f-496b-8af2-71b79451d971-registry-certificates\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:58:40.437338 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.437204    2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-registry-tls\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:58:40.437338 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.437213    2572 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/522ae11b-496f-496b-8af2-71b79451d971-bound-sa-token\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:58:40.437338 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.437220    2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/522ae11b-496f-496b-8af2-71b79451d971-trusted-ca\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:58:40.437338 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:40.437227    2572 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/522ae11b-496f-496b-8af2-71b79451d971-ca-trust-extracted\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:58:41.059524 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:41.059490    2572 generic.go:356] "Generic (PLEG): container finished" podID="522ae11b-496f-496b-8af2-71b79451d971" containerID="84e9db0934c5f38ed24c0bf6d53f7d9e5004c61a5bf107fe803e1e83d85f355e" exitCode=0
Sep 10 20:58:41.059956 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:41.059554    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79994c7475-ddczp"
Sep 10 20:58:41.059956 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:41.059584    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79994c7475-ddczp" event={"ID":"522ae11b-496f-496b-8af2-71b79451d971","Type":"ContainerDied","Data":"84e9db0934c5f38ed24c0bf6d53f7d9e5004c61a5bf107fe803e1e83d85f355e"}
Sep 10 20:58:41.059956 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:41.059638    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79994c7475-ddczp" event={"ID":"522ae11b-496f-496b-8af2-71b79451d971","Type":"ContainerDied","Data":"1ef9588b9121ece0bf62d299c07067755026ee83ec88cf6ed119849690b88bde"}
Sep 10 20:58:41.059956 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:41.059665    2572 scope.go:117] "RemoveContainer" containerID="84e9db0934c5f38ed24c0bf6d53f7d9e5004c61a5bf107fe803e1e83d85f355e"
Sep 10 20:58:41.072743 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:41.071109    2572 scope.go:117] "RemoveContainer" containerID="84e9db0934c5f38ed24c0bf6d53f7d9e5004c61a5bf107fe803e1e83d85f355e"
Sep 10 20:58:41.072743 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:58:41.071411    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e9db0934c5f38ed24c0bf6d53f7d9e5004c61a5bf107fe803e1e83d85f355e\": container with ID starting with 84e9db0934c5f38ed24c0bf6d53f7d9e5004c61a5bf107fe803e1e83d85f355e not found: ID does not exist" containerID="84e9db0934c5f38ed24c0bf6d53f7d9e5004c61a5bf107fe803e1e83d85f355e"
Sep 10 20:58:41.072743 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:41.071450    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e9db0934c5f38ed24c0bf6d53f7d9e5004c61a5bf107fe803e1e83d85f355e"} err="failed to get container status \"84e9db0934c5f38ed24c0bf6d53f7d9e5004c61a5bf107fe803e1e83d85f355e\": rpc error: code = NotFound desc = could not find container \"84e9db0934c5f38ed24c0bf6d53f7d9e5004c61a5bf107fe803e1e83d85f355e\": container with ID starting with 84e9db0934c5f38ed24c0bf6d53f7d9e5004c61a5bf107fe803e1e83d85f355e not found: ID does not exist"
Sep 10 20:58:41.095792 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:41.095766    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79994c7475-ddczp"]
Sep 10 20:58:41.099261 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:41.099242    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-79994c7475-ddczp"]
Sep 10 20:58:41.437001 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:41.436917    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="522ae11b-496f-496b-8af2-71b79451d971" path="/var/lib/kubelet/pods/522ae11b-496f-496b-8af2-71b79451d971/volumes"
Sep 10 20:58:45.845282 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:45.845194    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d9f9fb68c-t8spw"]
Sep 10 20:58:45.845855 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:45.845827    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="522ae11b-496f-496b-8af2-71b79451d971" containerName="registry"
Sep 10 20:58:45.845930 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:45.845858    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="522ae11b-496f-496b-8af2-71b79451d971" containerName="registry"
Sep 10 20:58:45.845930 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:45.845924    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="522ae11b-496f-496b-8af2-71b79451d971" containerName="registry"
Sep 10 20:58:45.849346 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:45.849324    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9f9fb68c-t8spw"
Sep 10 20:58:45.854574 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:45.854548    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 20:58:45.854956 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:45.854932    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 20:58:45.855088 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:45.854968    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 20:58:45.855594 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:45.855575    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 20:58:45.877515 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:45.877495    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 20:58:45.878122 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:45.878099    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kb4m6\""
Sep 10 20:58:45.978784 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:45.978746    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-console-oauth-config\") pod \"console-5d9f9fb68c-t8spw\" (UID: \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\") " pod="openshift-console/console-5d9f9fb68c-t8spw"
Sep 10 20:58:45.978932 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:45.978798    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2prq\" (UniqueName: \"kubernetes.io/projected/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-kube-api-access-s2prq\") pod \"console-5d9f9fb68c-t8spw\" (UID: \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\") " pod="openshift-console/console-5d9f9fb68c-t8spw"
Sep 10 20:58:45.978932 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:45.978817    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-service-ca\") pod \"console-5d9f9fb68c-t8spw\" (UID: \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\") " pod="openshift-console/console-5d9f9fb68c-t8spw"
Sep 10 20:58:45.978932 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:45.978865    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-console-serving-cert\") pod \"console-5d9f9fb68c-t8spw\" (UID: \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\") " pod="openshift-console/console-5d9f9fb68c-t8spw"
Sep 10 20:58:45.979088 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:45.978931    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-console-config\") pod \"console-5d9f9fb68c-t8spw\" (UID: \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\") " pod="openshift-console/console-5d9f9fb68c-t8spw"
Sep 10 20:58:45.979088 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:45.978965    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-oauth-serving-cert\") pod \"console-5d9f9fb68c-t8spw\" (UID: \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\") " pod="openshift-console/console-5d9f9fb68c-t8spw"
Sep 10 20:58:45.984607 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:45.984582    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d9f9fb68c-t8spw"]
Sep 10 20:58:46.079453 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:46.079426    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-console-serving-cert\") pod \"console-5d9f9fb68c-t8spw\" (UID: \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\") " pod="openshift-console/console-5d9f9fb68c-t8spw"
Sep 10 20:58:46.079599 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:46.079483    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-console-config\") pod \"console-5d9f9fb68c-t8spw\" (UID: \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\") " pod="openshift-console/console-5d9f9fb68c-t8spw"
Sep 10 20:58:46.079599 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:46.079514    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-oauth-serving-cert\") pod \"console-5d9f9fb68c-t8spw\" (UID: \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\") " pod="openshift-console/console-5d9f9fb68c-t8spw"
Sep 10 20:58:46.079599 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:46.079577    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-console-oauth-config\") pod \"console-5d9f9fb68c-t8spw\" (UID: \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\") " pod="openshift-console/console-5d9f9fb68c-t8spw"
Sep 10 20:58:46.079758 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:46.079615    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2prq\" (UniqueName: \"kubernetes.io/projected/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-kube-api-access-s2prq\") pod \"console-5d9f9fb68c-t8spw\" (UID: \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\") " pod="openshift-console/console-5d9f9fb68c-t8spw"
Sep 10 20:58:46.079758 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:46.079641    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-service-ca\") pod \"console-5d9f9fb68c-t8spw\" (UID: \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\") " pod="openshift-console/console-5d9f9fb68c-t8spw"
Sep 10 20:58:46.080455 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:46.080396    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-oauth-serving-cert\") pod \"console-5d9f9fb68c-t8spw\" (UID: \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\") " pod="openshift-console/console-5d9f9fb68c-t8spw"
Sep 10 20:58:46.080455 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:46.080395    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-service-ca\") pod \"console-5d9f9fb68c-t8spw\" (UID: \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\") " pod="openshift-console/console-5d9f9fb68c-t8spw"
Sep 10 20:58:46.080621 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:46.080523    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-console-config\") pod \"console-5d9f9fb68c-t8spw\" (UID: \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\") " pod="openshift-console/console-5d9f9fb68c-t8spw"
Sep 10 20:58:46.082492 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:46.082468    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-console-serving-cert\") pod \"console-5d9f9fb68c-t8spw\" (UID: \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\") " pod="openshift-console/console-5d9f9fb68c-t8spw"
Sep 10 20:58:46.082823 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:46.082798    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-console-oauth-config\") pod \"console-5d9f9fb68c-t8spw\" (UID: \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\") " pod="openshift-console/console-5d9f9fb68c-t8spw"
Sep 10 20:58:46.101112 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:46.101065    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2prq\" (UniqueName: \"kubernetes.io/projected/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-kube-api-access-s2prq\") pod \"console-5d9f9fb68c-t8spw\" (UID: \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\") " pod="openshift-console/console-5d9f9fb68c-t8spw"
Sep 10 20:58:46.159931 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:46.159910    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9f9fb68c-t8spw"
Sep 10 20:58:53.114146 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:53.114114    2572 generic.go:356] "Generic (PLEG): container finished" podID="4a1bd0d7-bb2f-494d-9553-9f3f89cfa92d" containerID="eb44257fdf7595e7940f6da5ed59bfc98b46c64b17f83b59cf7a61959e8a6a5d" exitCode=0
Sep 10 20:58:53.114508 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:53.114182    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dbhxv" event={"ID":"4a1bd0d7-bb2f-494d-9553-9f3f89cfa92d","Type":"ContainerDied","Data":"eb44257fdf7595e7940f6da5ed59bfc98b46c64b17f83b59cf7a61959e8a6a5d"}
Sep 10 20:58:53.119139 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:53.114899    2572 scope.go:117] "RemoveContainer" containerID="eb44257fdf7595e7940f6da5ed59bfc98b46c64b17f83b59cf7a61959e8a6a5d"
Sep 10 20:58:53.371674 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:53.371607    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d9f9fb68c-t8spw"]
Sep 10 20:58:54.123836 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:54.123135    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dbhxv" event={"ID":"4a1bd0d7-bb2f-494d-9553-9f3f89cfa92d","Type":"ContainerStarted","Data":"6b3ee5a7b52e61789f01823e0bbeb4c643c457c556af48cba53ce5109873c63e"}
Sep 10 20:58:54.126515 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:54.126488    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-f5qj7" event={"ID":"f0cb65f4-5aa1-41bd-9584-e3d440e4cb79","Type":"ContainerStarted","Data":"abaf764b42064445d7305b21a2c25981886ad509ec301a26663d7a6379483b67"}
Sep 10 20:58:54.127026 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:54.127001    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-f5qj7"
Sep 10 20:58:54.130608 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:54.130587    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9f9fb68c-t8spw" event={"ID":"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e","Type":"ContainerStarted","Data":"73ebf608019b56159d7927d3d10f64fa66736d3347ebb61c5404dcf9cbacb841"}
Sep 10 20:58:54.139819 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:54.139794    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-f5qj7"
Sep 10 20:58:54.201561 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:54.201513    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-f5qj7" podStartSLOduration=2.158011935 podStartE2EDuration="18.201496811s" totalImagesPullingTime="16.043484876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:54.200230923 +0000 UTC m=+119.412695846" watchObservedRunningTime="2026-09-10 20:58:54.201496811 +0000 UTC m=+119.413961732"
Sep 10 20:58:58.150915 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:58.150875    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9f9fb68c-t8spw" event={"ID":"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e","Type":"ContainerStarted","Data":"8cd8cfb396ab9889596f0694a222ea2e2759718041f79d092cf6632bd8dc5fe3"}
Sep 10 20:58:58.203551 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:58:58.203501    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d9f9fb68c-t8spw" podStartSLOduration=9.260065585 podStartE2EDuration="13.203488053s" totalImagesPullingTime="3.943422468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:58.202183074 +0000 UTC m=+123.414647995" watchObservedRunningTime="2026-09-10 20:58:58.203488053 +0000 UTC m=+123.415952972"
Sep 10 20:59:03.508221 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:03.508177    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d9f9fb68c-t8spw"]
Sep 10 20:59:06.161136 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:06.161104    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d9f9fb68c-t8spw"
Sep 10 20:59:10.040123 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:10.040070    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n29fp_80ad8dbd-fba2-4011-befe-6a1b7aa0c6b0/serve-healthcheck-canary/0.log"
Sep 10 20:59:28.527491 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:28.527437    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d9f9fb68c-t8spw" podUID="e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e" containerName="console" containerID="cri-o://8cd8cfb396ab9889596f0694a222ea2e2759718041f79d092cf6632bd8dc5fe3" gracePeriod=15
Sep 10 20:59:28.802765 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:28.802737    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9f9fb68c-t8spw"
Sep 10 20:59:28.995151 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:28.995125    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-console-serving-cert\") pod \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\" (UID: \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\") "
Sep 10 20:59:28.995287 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:28.995163    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-console-oauth-config\") pod \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\" (UID: \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\") "
Sep 10 20:59:28.995287 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:28.995203    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-service-ca\") pod \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\" (UID: \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\") "
Sep 10 20:59:28.995287 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:28.995226    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2prq\" (UniqueName: \"kubernetes.io/projected/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-kube-api-access-s2prq\") pod \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\" (UID: \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\") "
Sep 10 20:59:28.995287 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:28.995245    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-console-config\") pod \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\" (UID: \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\") "
Sep 10 20:59:28.995287 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:28.995260    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-oauth-serving-cert\") pod \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\" (UID: \"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e\") "
Sep 10 20:59:28.995718 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:28.995685    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-service-ca" (OuterVolumeSpecName: "service-ca") pod "e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e" (UID: "e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:59:28.995718 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:28.995703    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-console-config" (OuterVolumeSpecName: "console-config") pod "e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e" (UID: "e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:59:28.996060 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:28.995691    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e" (UID: "e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:59:28.997624 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:28.997588    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e" (UID: "e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:59:28.997728 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:28.997646    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e" (UID: "e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:59:28.997728 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:28.997684    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-kube-api-access-s2prq" (OuterVolumeSpecName: "kube-api-access-s2prq") pod "e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e" (UID: "e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e"). InnerVolumeSpecName "kube-api-access-s2prq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:59:29.096625 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:29.096573    2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-service-ca\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:59:29.096625 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:29.096595    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2prq\" (UniqueName: \"kubernetes.io/projected/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-kube-api-access-s2prq\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:59:29.096625 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:29.096605    2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-console-config\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:59:29.096625 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:29.096614    2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-oauth-serving-cert\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:59:29.096625 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:29.096623    2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-console-serving-cert\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:59:29.096823 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:29.096632    2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e-console-oauth-config\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:59:29.283602 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:29.283565    2572 generic.go:356] "Generic (PLEG): container finished" podID="e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e" containerID="8cd8cfb396ab9889596f0694a222ea2e2759718041f79d092cf6632bd8dc5fe3" exitCode=0
Sep 10 20:59:29.283718 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:29.283615    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9f9fb68c-t8spw"
Sep 10 20:59:29.283718 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:29.283658    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9f9fb68c-t8spw" event={"ID":"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e","Type":"ContainerDied","Data":"8cd8cfb396ab9889596f0694a222ea2e2759718041f79d092cf6632bd8dc5fe3"}
Sep 10 20:59:29.283718 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:29.283717    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9f9fb68c-t8spw" event={"ID":"e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e","Type":"ContainerDied","Data":"73ebf608019b56159d7927d3d10f64fa66736d3347ebb61c5404dcf9cbacb841"}
Sep 10 20:59:29.283851 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:29.283740    2572 scope.go:117] "RemoveContainer" containerID="8cd8cfb396ab9889596f0694a222ea2e2759718041f79d092cf6632bd8dc5fe3"
Sep 10 20:59:29.296216 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:29.296195    2572 scope.go:117] "RemoveContainer" containerID="8cd8cfb396ab9889596f0694a222ea2e2759718041f79d092cf6632bd8dc5fe3"
Sep 10 20:59:29.296526 ip-10-0-138-232 kubenswrapper[2572]: E0910 20:59:29.296500    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd8cfb396ab9889596f0694a222ea2e2759718041f79d092cf6632bd8dc5fe3\": container with ID starting with 8cd8cfb396ab9889596f0694a222ea2e2759718041f79d092cf6632bd8dc5fe3 not found: ID does not exist" containerID="8cd8cfb396ab9889596f0694a222ea2e2759718041f79d092cf6632bd8dc5fe3"
Sep 10 20:59:29.296597 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:29.296533    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd8cfb396ab9889596f0694a222ea2e2759718041f79d092cf6632bd8dc5fe3"} err="failed to get container status \"8cd8cfb396ab9889596f0694a222ea2e2759718041f79d092cf6632bd8dc5fe3\": rpc error: code = NotFound desc = could not find container \"8cd8cfb396ab9889596f0694a222ea2e2759718041f79d092cf6632bd8dc5fe3\": container with ID starting with 8cd8cfb396ab9889596f0694a222ea2e2759718041f79d092cf6632bd8dc5fe3 not found: ID does not exist"
Sep 10 20:59:29.320288 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:29.320260    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d9f9fb68c-t8spw"]
Sep 10 20:59:29.335435 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:29.335408    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d9f9fb68c-t8spw"]
Sep 10 20:59:29.437239 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:29.437186    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e" path="/var/lib/kubelet/pods/e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e/volumes"
Sep 10 20:59:49.197330 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:49.197233    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 20:59:49.197889 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:49.197546    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="alertmanager" containerID="cri-o://c300ceae13478b6f92ff8aced29fb2b780d7b9fecc8e0353beec1af02675f213" gracePeriod=120
Sep 10 20:59:49.197889 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:49.197621    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="kube-rbac-proxy-metric" containerID="cri-o://1b9877a8a758e906b032561e784b8581d29ed1b00f707d39063a148760952c7f" gracePeriod=120
Sep 10 20:59:49.197889 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:49.197763    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="kube-rbac-proxy-web" containerID="cri-o://724baaf284d888b95097d3efa7e2ce8bd849a080eb5c243e1ce9520ac69452a0" gracePeriod=120
Sep 10 20:59:49.199766 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:49.197890    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="prom-label-proxy" containerID="cri-o://13909b481bbc1bc738abe832ed39943890ea6c4552c3ce8b775eb872184f5ad2" gracePeriod=120
Sep 10 20:59:49.199766 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:49.197902    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="config-reloader" containerID="cri-o://3508ec7c7fe73333ef89f15bc3e06a6f809dd104794658fb4bfb8d725cb8bee1" gracePeriod=120
Sep 10 20:59:49.199766 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:49.198009    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="kube-rbac-proxy" containerID="cri-o://f88498262d80cde8890c38237d30553cf27fd489136b532c7d8a717ab5ae3d54" gracePeriod=120
Sep 10 20:59:49.369997 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:49.369957    2572 generic.go:356] "Generic (PLEG): container finished" podID="59c135ab-d11e-421c-b90f-d79c0416f909" containerID="13909b481bbc1bc738abe832ed39943890ea6c4552c3ce8b775eb872184f5ad2" exitCode=0
Sep 10 20:59:49.370093 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:49.369998    2572 generic.go:356] "Generic (PLEG): container finished" podID="59c135ab-d11e-421c-b90f-d79c0416f909" containerID="f88498262d80cde8890c38237d30553cf27fd489136b532c7d8a717ab5ae3d54" exitCode=0
Sep 10 20:59:49.370093 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:49.370009    2572 generic.go:356] "Generic (PLEG): container finished" podID="59c135ab-d11e-421c-b90f-d79c0416f909" containerID="3508ec7c7fe73333ef89f15bc3e06a6f809dd104794658fb4bfb8d725cb8bee1" exitCode=0
Sep 10 20:59:49.370093 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:49.370017    2572 generic.go:356] "Generic (PLEG): container finished" podID="59c135ab-d11e-421c-b90f-d79c0416f909" containerID="c300ceae13478b6f92ff8aced29fb2b780d7b9fecc8e0353beec1af02675f213" exitCode=0
Sep 10 20:59:49.370093 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:49.370023    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59c135ab-d11e-421c-b90f-d79c0416f909","Type":"ContainerDied","Data":"13909b481bbc1bc738abe832ed39943890ea6c4552c3ce8b775eb872184f5ad2"}
Sep 10 20:59:49.370093 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:49.370082    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59c135ab-d11e-421c-b90f-d79c0416f909","Type":"ContainerDied","Data":"f88498262d80cde8890c38237d30553cf27fd489136b532c7d8a717ab5ae3d54"}
Sep 10 20:59:49.370245 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:49.370110    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59c135ab-d11e-421c-b90f-d79c0416f909","Type":"ContainerDied","Data":"3508ec7c7fe73333ef89f15bc3e06a6f809dd104794658fb4bfb8d725cb8bee1"}
Sep 10 20:59:49.370245 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:49.370136    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59c135ab-d11e-421c-b90f-d79c0416f909","Type":"ContainerDied","Data":"c300ceae13478b6f92ff8aced29fb2b780d7b9fecc8e0353beec1af02675f213"}
Sep 10 20:59:50.386696 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.386552    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59c135ab-d11e-421c-b90f-d79c0416f909","Type":"ContainerDied","Data":"1b9877a8a758e906b032561e784b8581d29ed1b00f707d39063a148760952c7f"}
Sep 10 20:59:50.387228 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.387184    2572 generic.go:356] "Generic (PLEG): container finished" podID="59c135ab-d11e-421c-b90f-d79c0416f909" containerID="1b9877a8a758e906b032561e784b8581d29ed1b00f707d39063a148760952c7f" exitCode=0
Sep 10 20:59:50.387494 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.387351    2572 generic.go:356] "Generic (PLEG): container finished" podID="59c135ab-d11e-421c-b90f-d79c0416f909" containerID="724baaf284d888b95097d3efa7e2ce8bd849a080eb5c243e1ce9520ac69452a0" exitCode=0
Sep 10 20:59:50.387494 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.387418    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59c135ab-d11e-421c-b90f-d79c0416f909","Type":"ContainerDied","Data":"724baaf284d888b95097d3efa7e2ce8bd849a080eb5c243e1ce9520ac69452a0"}
Sep 10 20:59:50.446005 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.445967    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:50.540743 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.540717    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-kube-rbac-proxy-metric\") pod \"59c135ab-d11e-421c-b90f-d79c0416f909\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") "
Sep 10 20:59:50.540859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.540757    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-config-volume\") pod \"59c135ab-d11e-421c-b90f-d79c0416f909\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") "
Sep 10 20:59:50.540859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.540788    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-main-tls\") pod \"59c135ab-d11e-421c-b90f-d79c0416f909\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") "
Sep 10 20:59:50.540859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.540813    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/59c135ab-d11e-421c-b90f-d79c0416f909-tls-assets\") pod \"59c135ab-d11e-421c-b90f-d79c0416f909\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") "
Sep 10 20:59:50.540859 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.540843    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59c135ab-d11e-421c-b90f-d79c0416f909-metrics-client-ca\") pod \"59c135ab-d11e-421c-b90f-d79c0416f909\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") "
Sep 10 20:59:50.541104 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.540887    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-kube-rbac-proxy\") pod \"59c135ab-d11e-421c-b90f-d79c0416f909\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") "
Sep 10 20:59:50.541104 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.540929    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-kube-rbac-proxy-web\") pod \"59c135ab-d11e-421c-b90f-d79c0416f909\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") "
Sep 10 20:59:50.541104 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.540955    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-cluster-tls-config\") pod \"59c135ab-d11e-421c-b90f-d79c0416f909\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") "
Sep 10 20:59:50.541104 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.541008    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cr8dv\" (UniqueName: \"kubernetes.io/projected/59c135ab-d11e-421c-b90f-d79c0416f909-kube-api-access-cr8dv\") pod \"59c135ab-d11e-421c-b90f-d79c0416f909\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") "
Sep 10 20:59:50.541104 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.541035    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-web-config\") pod \"59c135ab-d11e-421c-b90f-d79c0416f909\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") "
Sep 10 20:59:50.541104 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.541077    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/59c135ab-d11e-421c-b90f-d79c0416f909-config-out\") pod \"59c135ab-d11e-421c-b90f-d79c0416f909\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") "
Sep 10 20:59:50.541402 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.541104    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59c135ab-d11e-421c-b90f-d79c0416f909-alertmanager-trusted-ca-bundle\") pod \"59c135ab-d11e-421c-b90f-d79c0416f909\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") "
Sep 10 20:59:50.541402 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.541131    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/59c135ab-d11e-421c-b90f-d79c0416f909-alertmanager-main-db\") pod \"59c135ab-d11e-421c-b90f-d79c0416f909\" (UID: \"59c135ab-d11e-421c-b90f-d79c0416f909\") "
Sep 10 20:59:50.541402 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.541260    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c135ab-d11e-421c-b90f-d79c0416f909-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "59c135ab-d11e-421c-b90f-d79c0416f909" (UID: "59c135ab-d11e-421c-b90f-d79c0416f909"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:59:50.541724 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.541508    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c135ab-d11e-421c-b90f-d79c0416f909-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "59c135ab-d11e-421c-b90f-d79c0416f909" (UID: "59c135ab-d11e-421c-b90f-d79c0416f909"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:59:50.541724 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.541664    2572 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/59c135ab-d11e-421c-b90f-d79c0416f909-alertmanager-main-db\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:59:50.541724 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.541685    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59c135ab-d11e-421c-b90f-d79c0416f909-metrics-client-ca\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:59:50.543844 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.543804    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "59c135ab-d11e-421c-b90f-d79c0416f909" (UID: "59c135ab-d11e-421c-b90f-d79c0416f909"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:59:50.545002 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.544947    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c135ab-d11e-421c-b90f-d79c0416f909-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "59c135ab-d11e-421c-b90f-d79c0416f909" (UID: "59c135ab-d11e-421c-b90f-d79c0416f909"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:59:50.545747 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.545721    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-config-volume" (OuterVolumeSpecName: "config-volume") pod "59c135ab-d11e-421c-b90f-d79c0416f909" (UID: "59c135ab-d11e-421c-b90f-d79c0416f909"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:59:50.545843 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.545764    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c135ab-d11e-421c-b90f-d79c0416f909-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "59c135ab-d11e-421c-b90f-d79c0416f909" (UID: "59c135ab-d11e-421c-b90f-d79c0416f909"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:59:50.545843 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.545788    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "59c135ab-d11e-421c-b90f-d79c0416f909" (UID: "59c135ab-d11e-421c-b90f-d79c0416f909"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:59:50.545843 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.545800    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "59c135ab-d11e-421c-b90f-d79c0416f909" (UID: "59c135ab-d11e-421c-b90f-d79c0416f909"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:59:50.545843 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.545813    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "59c135ab-d11e-421c-b90f-d79c0416f909" (UID: "59c135ab-d11e-421c-b90f-d79c0416f909"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:59:50.546091 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.545861    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c135ab-d11e-421c-b90f-d79c0416f909-kube-api-access-cr8dv" (OuterVolumeSpecName: "kube-api-access-cr8dv") pod "59c135ab-d11e-421c-b90f-d79c0416f909" (UID: "59c135ab-d11e-421c-b90f-d79c0416f909"). InnerVolumeSpecName "kube-api-access-cr8dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:59:50.546460 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.546443    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c135ab-d11e-421c-b90f-d79c0416f909-config-out" (OuterVolumeSpecName: "config-out") pod "59c135ab-d11e-421c-b90f-d79c0416f909" (UID: "59c135ab-d11e-421c-b90f-d79c0416f909"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:59:50.549498 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.549475    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "59c135ab-d11e-421c-b90f-d79c0416f909" (UID: "59c135ab-d11e-421c-b90f-d79c0416f909"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:59:50.557555 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.557534    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-web-config" (OuterVolumeSpecName: "web-config") pod "59c135ab-d11e-421c-b90f-d79c0416f909" (UID: "59c135ab-d11e-421c-b90f-d79c0416f909"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:59:50.642852 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.642816    2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:59:50.642852 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.642849    2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:59:50.642852 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.642861    2572 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-cluster-tls-config\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:59:50.643030 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.642870    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr8dv\" (UniqueName: \"kubernetes.io/projected/59c135ab-d11e-421c-b90f-d79c0416f909-kube-api-access-cr8dv\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:59:50.643030 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.642880    2572 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-web-config\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:59:50.643030 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.642888    2572 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/59c135ab-d11e-421c-b90f-d79c0416f909-config-out\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:59:50.643030 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.642897    2572 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59c135ab-d11e-421c-b90f-d79c0416f909-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:59:50.643030 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.642905    2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:59:50.643030 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.642913    2572 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-config-volume\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:59:50.643030 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.642921    2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/59c135ab-d11e-421c-b90f-d79c0416f909-secret-alertmanager-main-tls\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:59:50.643030 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:50.642928    2572 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/59c135ab-d11e-421c-b90f-d79c0416f909-tls-assets\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 20:59:51.396209 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.396176    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59c135ab-d11e-421c-b90f-d79c0416f909","Type":"ContainerDied","Data":"8b5224e45253717da33f0687250e09edd7166843eb29e76dfd00cc44bdd6f0d0"}
Sep 10 20:59:51.396508 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.396216    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.396508 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.396282    2572 scope.go:117] "RemoveContainer" containerID="13909b481bbc1bc738abe832ed39943890ea6c4552c3ce8b775eb872184f5ad2"
Sep 10 20:59:51.407316 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.407079    2572 scope.go:117] "RemoveContainer" containerID="1b9877a8a758e906b032561e784b8581d29ed1b00f707d39063a148760952c7f"
Sep 10 20:59:51.415700 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.415679    2572 scope.go:117] "RemoveContainer" containerID="f88498262d80cde8890c38237d30553cf27fd489136b532c7d8a717ab5ae3d54"
Sep 10 20:59:51.422866 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.422849    2572 scope.go:117] "RemoveContainer" containerID="724baaf284d888b95097d3efa7e2ce8bd849a080eb5c243e1ce9520ac69452a0"
Sep 10 20:59:51.428294 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.428270    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 20:59:51.430184 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.430160    2572 scope.go:117] "RemoveContainer" containerID="3508ec7c7fe73333ef89f15bc3e06a6f809dd104794658fb4bfb8d725cb8bee1"
Sep 10 20:59:51.437834 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.437811    2572 scope.go:117] "RemoveContainer" containerID="c300ceae13478b6f92ff8aced29fb2b780d7b9fecc8e0353beec1af02675f213"
Sep 10 20:59:51.439366 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.439342    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 20:59:51.446500 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.446482    2572 scope.go:117] "RemoveContainer" containerID="2c6cb3af368d818c1f2d134599721da9e9fa054d835b4865f6a943eac6b0b25a"
Sep 10 20:59:51.488906 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.488880    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 20:59:51.489172 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.489158    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="kube-rbac-proxy"
Sep 10 20:59:51.489220 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.489174    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="kube-rbac-proxy"
Sep 10 20:59:51.489220 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.489189    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="init-config-reloader"
Sep 10 20:59:51.489220 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.489194    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="init-config-reloader"
Sep 10 20:59:51.489220 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.489201    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="kube-rbac-proxy-web"
Sep 10 20:59:51.489220 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.489207    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="kube-rbac-proxy-web"
Sep 10 20:59:51.489220 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.489214    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="alertmanager"
Sep 10 20:59:51.489220 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.489219    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="alertmanager"
Sep 10 20:59:51.489419 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.489227    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="config-reloader"
Sep 10 20:59:51.489419 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.489232    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="config-reloader"
Sep 10 20:59:51.489419 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.489238    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="prom-label-proxy"
Sep 10 20:59:51.489419 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.489243    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="prom-label-proxy"
Sep 10 20:59:51.489419 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.489252    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="kube-rbac-proxy-metric"
Sep 10 20:59:51.489419 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.489257    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="kube-rbac-proxy-metric"
Sep 10 20:59:51.489419 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.489264    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e" containerName="console"
Sep 10 20:59:51.489419 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.489269    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e" containerName="console"
Sep 10 20:59:51.489419 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.489311    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="config-reloader"
Sep 10 20:59:51.489419 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.489318    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="prom-label-proxy"
Sep 10 20:59:51.489419 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.489324    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8a705ee-1fbf-4593-aa63-a1ae1bd50b6e" containerName="console"
Sep 10 20:59:51.489419 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.489331    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="kube-rbac-proxy-metric"
Sep 10 20:59:51.489419 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.489338    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="kube-rbac-proxy"
Sep 10 20:59:51.489419 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.489343    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="alertmanager"
Sep 10 20:59:51.489419 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.489348    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" containerName="kube-rbac-proxy-web"
Sep 10 20:59:51.492277 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.492262    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.498841 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.498819    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 20:59:51.499172 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.499156    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 20:59:51.499357 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.499341    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 20:59:51.499446 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.499362    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 20:59:51.499494 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.499447    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9xv52\""
Sep 10 20:59:51.499601 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.499585    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 20:59:51.499640 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.499585    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 20:59:51.499640 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.499600    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 20:59:51.499823 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.499809    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 20:59:51.527486 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.527444    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 20:59:51.533420 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.533401    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 20:59:51.549294 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.549276    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.549372 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.549304    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.549372 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.549330    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zblrg\" (UniqueName: \"kubernetes.io/projected/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-kube-api-access-zblrg\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.549372 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.549354    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.549480 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.549380    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-config-volume\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.549480 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.549396    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-web-config\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.549480 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.549411    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.549480 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.549430    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.549603 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.549500    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.549603 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.549523    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.549603 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.549553    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-config-out\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.549603 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.549587    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.549747 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.549609    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.649900 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.649874    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.650008 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.649905    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.650008 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.649924    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.650076 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.650019    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.650076 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.650055    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zblrg\" (UniqueName: \"kubernetes.io/projected/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-kube-api-access-zblrg\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.650076 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.650073    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.650201 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.650096    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-config-volume\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.650201 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.650119    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-web-config\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.650201 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.650145    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.650201 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.650172    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.650201 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.650200    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.650430 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.650223    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.650430 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.650267    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-config-out\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.651270 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.651241    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.651394 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.651370    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.653313 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.653216    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.653313 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.653222    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-web-config\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.653313 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.653273    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-config-volume\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.653943 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.653513    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.653943 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.653575    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.653943 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.653732    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.653943 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.653819    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-config-out\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.654232 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.653965    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.654232 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.654067    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.655301 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.655283    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.660403 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.660381    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zblrg\" (UniqueName: \"kubernetes.io/projected/eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3-kube-api-access-zblrg\") pod \"alertmanager-main-0\" (UID: \"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:51.801617 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:51.801598    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 20:59:52.095611 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:52.095535    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 20:59:52.099626 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:59:52.099598    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeeefc09_7bf6_4e2f_8b6e_b6daa0d412c3.slice/crio-ef942c05a72ad23a0211822501cfa2f94bed0748e74f0abb5aed039d6b1acf2f WatchSource:0}: Error finding container ef942c05a72ad23a0211822501cfa2f94bed0748e74f0abb5aed039d6b1acf2f: Status 404 returned error can't find the container with id ef942c05a72ad23a0211822501cfa2f94bed0748e74f0abb5aed039d6b1acf2f
Sep 10 20:59:52.402096 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:52.402028    2572 generic.go:356] "Generic (PLEG): container finished" podID="eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3" containerID="a541ae8319442db861cc76d2ca1bd9b8348ad1a70692d0a4ccaa06a84b46578b" exitCode=0
Sep 10 20:59:52.402446 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:52.402105    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3","Type":"ContainerDied","Data":"a541ae8319442db861cc76d2ca1bd9b8348ad1a70692d0a4ccaa06a84b46578b"}
Sep 10 20:59:52.402446 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:52.402155    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3","Type":"ContainerStarted","Data":"ef942c05a72ad23a0211822501cfa2f94bed0748e74f0abb5aed039d6b1acf2f"}
Sep 10 20:59:53.409650 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.409617    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3","Type":"ContainerStarted","Data":"f1bf587496a6b424917681ceea122850352257a3e2cdf5457bb0bc1a246f7670"}
Sep 10 20:59:53.410014 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.409692    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3","Type":"ContainerStarted","Data":"88eddf15f6db3db52a650d16e9c265686a9e276df38927eb2683a7474a00338a"}
Sep 10 20:59:53.410014 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.409721    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3","Type":"ContainerStarted","Data":"11e13547cc9efe49b032eca3f177d9f9fb6d43e869d3e4033d6e8d1c33ec9d3c"}
Sep 10 20:59:53.410014 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.409745    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3","Type":"ContainerStarted","Data":"9a85806fc3f0a3b9219358af55f40349b1dbea910a68d1150318f76e4cc69d21"}
Sep 10 20:59:53.410014 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.409769    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3","Type":"ContainerStarted","Data":"14893bc4567b0695d4f2e33703cb0b6b77d968bf914c115537ffdf63297529e3"}
Sep 10 20:59:53.410014 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.409793    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3","Type":"ContainerStarted","Data":"c50877970b308afd8b272c984d00715d89b7c949e0c2293762d6480d1325419e"}
Sep 10 20:59:53.437135 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.437109    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59c135ab-d11e-421c-b90f-d79c0416f909" path="/var/lib/kubelet/pods/59c135ab-d11e-421c-b90f-d79c0416f909/volumes"
Sep 10 20:59:53.467885 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.467846    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.448698812 podStartE2EDuration="2.467828429s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.129617ms" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:53.464645233 +0000 UTC m=+178.677110154" watchObservedRunningTime="2026-09-10 20:59:53.467828429 +0000 UTC m=+178.680293349"
Sep 10 20:59:53.503430 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.503404    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"]
Sep 10 20:59:53.511686 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.511662    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.518962 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.518936    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 10 20:59:53.519085 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.519044    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-xk467\""
Sep 10 20:59:53.520858 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.520841    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 10 20:59:53.520921 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.520841    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 10 20:59:53.520961 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.520940    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 10 20:59:53.521190 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.521173    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 10 20:59:53.561891 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.561861    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 10 20:59:53.562905 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.562889    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a9145cc2-5aef-4ab8-af32-675e626ed7c4-telemeter-client-tls\") pod \"telemeter-client-7d786f5d47-9ph4f\" (UID: \"a9145cc2-5aef-4ab8-af32-675e626ed7c4\") " pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.562957 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.562916    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9145cc2-5aef-4ab8-af32-675e626ed7c4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7d786f5d47-9ph4f\" (UID: \"a9145cc2-5aef-4ab8-af32-675e626ed7c4\") " pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.562957 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.562936    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a9145cc2-5aef-4ab8-af32-675e626ed7c4-federate-client-tls\") pod \"telemeter-client-7d786f5d47-9ph4f\" (UID: \"a9145cc2-5aef-4ab8-af32-675e626ed7c4\") " pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.563060 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.562954    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9145cc2-5aef-4ab8-af32-675e626ed7c4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7d786f5d47-9ph4f\" (UID: \"a9145cc2-5aef-4ab8-af32-675e626ed7c4\") " pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.563098 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.563059    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a9145cc2-5aef-4ab8-af32-675e626ed7c4-secret-telemeter-client\") pod \"telemeter-client-7d786f5d47-9ph4f\" (UID: \"a9145cc2-5aef-4ab8-af32-675e626ed7c4\") " pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.563098 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.563088    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9145cc2-5aef-4ab8-af32-675e626ed7c4-metrics-client-ca\") pod \"telemeter-client-7d786f5d47-9ph4f\" (UID: \"a9145cc2-5aef-4ab8-af32-675e626ed7c4\") " pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.563171 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.563103    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d24g6\" (UniqueName: \"kubernetes.io/projected/a9145cc2-5aef-4ab8-af32-675e626ed7c4-kube-api-access-d24g6\") pod \"telemeter-client-7d786f5d47-9ph4f\" (UID: \"a9145cc2-5aef-4ab8-af32-675e626ed7c4\") " pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.563171 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.563119    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9145cc2-5aef-4ab8-af32-675e626ed7c4-serving-certs-ca-bundle\") pod \"telemeter-client-7d786f5d47-9ph4f\" (UID: \"a9145cc2-5aef-4ab8-af32-675e626ed7c4\") " pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.593680 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.593652    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"]
Sep 10 20:59:53.664079 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.664025    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a9145cc2-5aef-4ab8-af32-675e626ed7c4-secret-telemeter-client\") pod \"telemeter-client-7d786f5d47-9ph4f\" (UID: \"a9145cc2-5aef-4ab8-af32-675e626ed7c4\") " pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.664079 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.664058    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9145cc2-5aef-4ab8-af32-675e626ed7c4-metrics-client-ca\") pod \"telemeter-client-7d786f5d47-9ph4f\" (UID: \"a9145cc2-5aef-4ab8-af32-675e626ed7c4\") " pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.664079 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.664074    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d24g6\" (UniqueName: \"kubernetes.io/projected/a9145cc2-5aef-4ab8-af32-675e626ed7c4-kube-api-access-d24g6\") pod \"telemeter-client-7d786f5d47-9ph4f\" (UID: \"a9145cc2-5aef-4ab8-af32-675e626ed7c4\") " pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.664267 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.664090    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9145cc2-5aef-4ab8-af32-675e626ed7c4-serving-certs-ca-bundle\") pod \"telemeter-client-7d786f5d47-9ph4f\" (UID: \"a9145cc2-5aef-4ab8-af32-675e626ed7c4\") " pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.664267 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.664116    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a9145cc2-5aef-4ab8-af32-675e626ed7c4-telemeter-client-tls\") pod \"telemeter-client-7d786f5d47-9ph4f\" (UID: \"a9145cc2-5aef-4ab8-af32-675e626ed7c4\") " pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.664267 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.664132    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9145cc2-5aef-4ab8-af32-675e626ed7c4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7d786f5d47-9ph4f\" (UID: \"a9145cc2-5aef-4ab8-af32-675e626ed7c4\") " pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.664267 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.664163    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a9145cc2-5aef-4ab8-af32-675e626ed7c4-federate-client-tls\") pod \"telemeter-client-7d786f5d47-9ph4f\" (UID: \"a9145cc2-5aef-4ab8-af32-675e626ed7c4\") " pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.664455 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.664324    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9145cc2-5aef-4ab8-af32-675e626ed7c4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7d786f5d47-9ph4f\" (UID: \"a9145cc2-5aef-4ab8-af32-675e626ed7c4\") " pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.664839 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.664810    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9145cc2-5aef-4ab8-af32-675e626ed7c4-serving-certs-ca-bundle\") pod \"telemeter-client-7d786f5d47-9ph4f\" (UID: \"a9145cc2-5aef-4ab8-af32-675e626ed7c4\") " pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.664974 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.664953    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9145cc2-5aef-4ab8-af32-675e626ed7c4-metrics-client-ca\") pod \"telemeter-client-7d786f5d47-9ph4f\" (UID: \"a9145cc2-5aef-4ab8-af32-675e626ed7c4\") " pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.665104 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.665076    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9145cc2-5aef-4ab8-af32-675e626ed7c4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7d786f5d47-9ph4f\" (UID: \"a9145cc2-5aef-4ab8-af32-675e626ed7c4\") " pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.666707 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.666682    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a9145cc2-5aef-4ab8-af32-675e626ed7c4-telemeter-client-tls\") pod \"telemeter-client-7d786f5d47-9ph4f\" (UID: \"a9145cc2-5aef-4ab8-af32-675e626ed7c4\") " pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.666803 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.666735    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a9145cc2-5aef-4ab8-af32-675e626ed7c4-secret-telemeter-client\") pod \"telemeter-client-7d786f5d47-9ph4f\" (UID: \"a9145cc2-5aef-4ab8-af32-675e626ed7c4\") " pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.667036 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.667021    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9145cc2-5aef-4ab8-af32-675e626ed7c4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7d786f5d47-9ph4f\" (UID: \"a9145cc2-5aef-4ab8-af32-675e626ed7c4\") " pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.667135 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.667119    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a9145cc2-5aef-4ab8-af32-675e626ed7c4-federate-client-tls\") pod \"telemeter-client-7d786f5d47-9ph4f\" (UID: \"a9145cc2-5aef-4ab8-af32-675e626ed7c4\") " pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.678371 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.678349    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d24g6\" (UniqueName: \"kubernetes.io/projected/a9145cc2-5aef-4ab8-af32-675e626ed7c4-kube-api-access-d24g6\") pod \"telemeter-client-7d786f5d47-9ph4f\" (UID: \"a9145cc2-5aef-4ab8-af32-675e626ed7c4\") " pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:53.820914 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:53.820893    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"
Sep 10 20:59:54.123877 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:54.123855    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7d786f5d47-9ph4f"]
Sep 10 20:59:54.126254 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:59:54.126228    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9145cc2_5aef_4ab8_af32_675e626ed7c4.slice/crio-fdc872b8f9caad1d1140e03a6a641c2fb6c967f8f61f9b0ea645dd69be3cdff1 WatchSource:0}: Error finding container fdc872b8f9caad1d1140e03a6a641c2fb6c967f8f61f9b0ea645dd69be3cdff1: Status 404 returned error can't find the container with id fdc872b8f9caad1d1140e03a6a641c2fb6c967f8f61f9b0ea645dd69be3cdff1
Sep 10 20:59:54.414656 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:54.414586    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f" event={"ID":"a9145cc2-5aef-4ab8-af32-675e626ed7c4","Type":"ContainerStarted","Data":"fdc872b8f9caad1d1140e03a6a641c2fb6c967f8f61f9b0ea645dd69be3cdff1"}
Sep 10 20:59:56.424697 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.424648    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f" event={"ID":"a9145cc2-5aef-4ab8-af32-675e626ed7c4","Type":"ContainerStarted","Data":"a552a5ebca13b73b87cc2d7d9b351f6f3fca125cd1c394f4fbd4be4cc1cb81dc"}
Sep 10 20:59:56.425083 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.424736    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f" event={"ID":"a9145cc2-5aef-4ab8-af32-675e626ed7c4","Type":"ContainerStarted","Data":"7c093d9ec6c3f4f97ce3f26d66c040a872e9f2f5df83f3257da1a5c089d6009f"}
Sep 10 20:59:56.425083 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.424759    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f" event={"ID":"a9145cc2-5aef-4ab8-af32-675e626ed7c4","Type":"ContainerStarted","Data":"2cfce8ec28cbb85551e07b9009e61ba7ec77ff5748b7ea606a91c469af4f43a5"}
Sep 10 20:59:56.457143 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.457084    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7d786f5d47-9ph4f" podStartSLOduration=1.710987499 podStartE2EDuration="3.457064202s" totalImagesPullingTime="1.746076703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:56.456535253 +0000 UTC m=+181.669000172" watchObservedRunningTime="2026-09-10 20:59:56.457064202 +0000 UTC m=+181.669529122"
Sep 10 20:59:56.653822 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.653790    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fc8c8b47c-b7nbc"]
Sep 10 20:59:56.655961 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.655942    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 20:59:56.662492 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.662450    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 20:59:56.662639 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.662523    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 20:59:56.662639 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.662617    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 20:59:56.662914 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.662794    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 20:59:56.663562 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.663537    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 20:59:56.663665 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.663595    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kb4m6\""
Sep 10 20:59:56.669121 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.669097    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 20:59:56.687113 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.687049    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dacefb4d-f6eb-4381-ba78-4d7c52f73220-console-serving-cert\") pod \"console-5fc8c8b47c-b7nbc\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") " pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 20:59:56.687113 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.687086    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pzkd\" (UniqueName: \"kubernetes.io/projected/dacefb4d-f6eb-4381-ba78-4d7c52f73220-kube-api-access-5pzkd\") pod \"console-5fc8c8b47c-b7nbc\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") " pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 20:59:56.687251 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.687118    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dacefb4d-f6eb-4381-ba78-4d7c52f73220-service-ca\") pod \"console-5fc8c8b47c-b7nbc\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") " pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 20:59:56.687251 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.687141    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dacefb4d-f6eb-4381-ba78-4d7c52f73220-oauth-serving-cert\") pod \"console-5fc8c8b47c-b7nbc\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") " pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 20:59:56.687251 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.687175    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dacefb4d-f6eb-4381-ba78-4d7c52f73220-console-oauth-config\") pod \"console-5fc8c8b47c-b7nbc\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") " pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 20:59:56.687251 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.687209    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dacefb4d-f6eb-4381-ba78-4d7c52f73220-console-config\") pod \"console-5fc8c8b47c-b7nbc\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") " pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 20:59:56.687392 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.687269    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dacefb4d-f6eb-4381-ba78-4d7c52f73220-trusted-ca-bundle\") pod \"console-5fc8c8b47c-b7nbc\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") " pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 20:59:56.730406 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.730378    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fc8c8b47c-b7nbc"]
Sep 10 20:59:56.788162 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.788139    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dacefb4d-f6eb-4381-ba78-4d7c52f73220-trusted-ca-bundle\") pod \"console-5fc8c8b47c-b7nbc\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") " pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 20:59:56.788246 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.788186    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dacefb4d-f6eb-4381-ba78-4d7c52f73220-console-serving-cert\") pod \"console-5fc8c8b47c-b7nbc\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") " pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 20:59:56.788246 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.788211    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pzkd\" (UniqueName: \"kubernetes.io/projected/dacefb4d-f6eb-4381-ba78-4d7c52f73220-kube-api-access-5pzkd\") pod \"console-5fc8c8b47c-b7nbc\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") " pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 20:59:56.788246 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.788240    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dacefb4d-f6eb-4381-ba78-4d7c52f73220-service-ca\") pod \"console-5fc8c8b47c-b7nbc\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") " pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 20:59:56.788380 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.788262    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dacefb4d-f6eb-4381-ba78-4d7c52f73220-oauth-serving-cert\") pod \"console-5fc8c8b47c-b7nbc\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") " pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 20:59:56.788380 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.788296    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dacefb4d-f6eb-4381-ba78-4d7c52f73220-console-oauth-config\") pod \"console-5fc8c8b47c-b7nbc\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") " pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 20:59:56.788479 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.788411    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dacefb4d-f6eb-4381-ba78-4d7c52f73220-console-config\") pod \"console-5fc8c8b47c-b7nbc\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") " pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 20:59:56.789101 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.789067    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dacefb4d-f6eb-4381-ba78-4d7c52f73220-service-ca\") pod \"console-5fc8c8b47c-b7nbc\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") " pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 20:59:56.789248 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.789226    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dacefb4d-f6eb-4381-ba78-4d7c52f73220-oauth-serving-cert\") pod \"console-5fc8c8b47c-b7nbc\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") " pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 20:59:56.789290 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.789248    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dacefb4d-f6eb-4381-ba78-4d7c52f73220-trusted-ca-bundle\") pod \"console-5fc8c8b47c-b7nbc\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") " pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 20:59:56.789328 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.789248    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dacefb4d-f6eb-4381-ba78-4d7c52f73220-console-config\") pod \"console-5fc8c8b47c-b7nbc\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") " pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 20:59:56.790835 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.790817    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dacefb4d-f6eb-4381-ba78-4d7c52f73220-console-serving-cert\") pod \"console-5fc8c8b47c-b7nbc\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") " pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 20:59:56.790921 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.790906    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dacefb4d-f6eb-4381-ba78-4d7c52f73220-console-oauth-config\") pod \"console-5fc8c8b47c-b7nbc\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") " pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 20:59:56.799265 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.799242    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pzkd\" (UniqueName: \"kubernetes.io/projected/dacefb4d-f6eb-4381-ba78-4d7c52f73220-kube-api-access-5pzkd\") pod \"console-5fc8c8b47c-b7nbc\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") " pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 20:59:56.966276 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:56.966209    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 20:59:57.249159 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:57.249133    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fc8c8b47c-b7nbc"]
Sep 10 20:59:57.250996 ip-10-0-138-232 kubenswrapper[2572]: W0910 20:59:57.250954    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddacefb4d_f6eb_4381_ba78_4d7c52f73220.slice/crio-b7aaec2e426221a3f376c590aa6886d1ea02ba7aae83b54e4202d0222520574d WatchSource:0}: Error finding container b7aaec2e426221a3f376c590aa6886d1ea02ba7aae83b54e4202d0222520574d: Status 404 returned error can't find the container with id b7aaec2e426221a3f376c590aa6886d1ea02ba7aae83b54e4202d0222520574d
Sep 10 20:59:57.437553 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:57.437525    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fc8c8b47c-b7nbc" event={"ID":"dacefb4d-f6eb-4381-ba78-4d7c52f73220","Type":"ContainerStarted","Data":"29bbbd7fc90cf8fb442446fc78014e3d597b34092ddb3247b4919798223a0276"}
Sep 10 20:59:57.437862 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:57.437568    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fc8c8b47c-b7nbc" event={"ID":"dacefb4d-f6eb-4381-ba78-4d7c52f73220","Type":"ContainerStarted","Data":"b7aaec2e426221a3f376c590aa6886d1ea02ba7aae83b54e4202d0222520574d"}
Sep 10 20:59:57.468580 ip-10-0-138-232 kubenswrapper[2572]: I0910 20:59:57.468536    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5fc8c8b47c-b7nbc" podStartSLOduration=1.468521028 podStartE2EDuration="1.468521028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:57.467177717 +0000 UTC m=+182.679642637" watchObservedRunningTime="2026-09-10 20:59:57.468521028 +0000 UTC m=+182.680985946"
Sep 10 21:00:06.966783 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:00:06.966745    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 21:00:06.967261 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:00:06.966809    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 21:00:06.971696 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:00:06.971669    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 21:00:07.483475 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:00:07.483450    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 21:01:36.526417 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.526385    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79f8755c64-p4llk"]
Sep 10 21:01:36.529458 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.529443    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:36.562056 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.562022    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79f8755c64-p4llk"]
Sep 10 21:01:36.600618 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.600581    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8ed697a-aa47-48c6-bbf6-561a2e969401-console-config\") pod \"console-79f8755c64-p4llk\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") " pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:36.600729 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.600628    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb5qz\" (UniqueName: \"kubernetes.io/projected/d8ed697a-aa47-48c6-bbf6-561a2e969401-kube-api-access-gb5qz\") pod \"console-79f8755c64-p4llk\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") " pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:36.600729 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.600654    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8ed697a-aa47-48c6-bbf6-561a2e969401-service-ca\") pod \"console-79f8755c64-p4llk\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") " pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:36.600729 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.600670    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8ed697a-aa47-48c6-bbf6-561a2e969401-console-serving-cert\") pod \"console-79f8755c64-p4llk\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") " pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:36.600729 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.600698    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8ed697a-aa47-48c6-bbf6-561a2e969401-oauth-serving-cert\") pod \"console-79f8755c64-p4llk\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") " pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:36.600863 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.600756    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8ed697a-aa47-48c6-bbf6-561a2e969401-console-oauth-config\") pod \"console-79f8755c64-p4llk\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") " pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:36.600863 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.600778    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8ed697a-aa47-48c6-bbf6-561a2e969401-trusted-ca-bundle\") pod \"console-79f8755c64-p4llk\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") " pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:36.701096 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.701065    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8ed697a-aa47-48c6-bbf6-561a2e969401-console-serving-cert\") pod \"console-79f8755c64-p4llk\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") " pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:36.701096 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.701101    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8ed697a-aa47-48c6-bbf6-561a2e969401-oauth-serving-cert\") pod \"console-79f8755c64-p4llk\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") " pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:36.701233 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.701126    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8ed697a-aa47-48c6-bbf6-561a2e969401-console-oauth-config\") pod \"console-79f8755c64-p4llk\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") " pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:36.701269 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.701250    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8ed697a-aa47-48c6-bbf6-561a2e969401-trusted-ca-bundle\") pod \"console-79f8755c64-p4llk\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") " pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:36.701323 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.701307    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8ed697a-aa47-48c6-bbf6-561a2e969401-console-config\") pod \"console-79f8755c64-p4llk\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") " pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:36.701375 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.701348    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb5qz\" (UniqueName: \"kubernetes.io/projected/d8ed697a-aa47-48c6-bbf6-561a2e969401-kube-api-access-gb5qz\") pod \"console-79f8755c64-p4llk\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") " pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:36.701424 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.701403    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8ed697a-aa47-48c6-bbf6-561a2e969401-service-ca\") pod \"console-79f8755c64-p4llk\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") " pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:36.701871 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.701852    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8ed697a-aa47-48c6-bbf6-561a2e969401-oauth-serving-cert\") pod \"console-79f8755c64-p4llk\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") " pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:36.702115 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.702094    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8ed697a-aa47-48c6-bbf6-561a2e969401-service-ca\") pod \"console-79f8755c64-p4llk\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") " pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:36.702169 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.702095    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8ed697a-aa47-48c6-bbf6-561a2e969401-console-config\") pod \"console-79f8755c64-p4llk\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") " pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:36.702557 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.702360    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8ed697a-aa47-48c6-bbf6-561a2e969401-trusted-ca-bundle\") pod \"console-79f8755c64-p4llk\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") " pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:36.704052 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.704028    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8ed697a-aa47-48c6-bbf6-561a2e969401-console-oauth-config\") pod \"console-79f8755c64-p4llk\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") " pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:36.709361 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.709334    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8ed697a-aa47-48c6-bbf6-561a2e969401-console-serving-cert\") pod \"console-79f8755c64-p4llk\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") " pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:36.712013 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.711973    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb5qz\" (UniqueName: \"kubernetes.io/projected/d8ed697a-aa47-48c6-bbf6-561a2e969401-kube-api-access-gb5qz\") pod \"console-79f8755c64-p4llk\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") " pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:36.838189 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:36.838168    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:37.118272 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:37.118115    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79f8755c64-p4llk"]
Sep 10 21:01:37.120471 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:01:37.120446    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ed697a_aa47_48c6_bbf6_561a2e969401.slice/crio-c6ed84f0f4bbede85cba658f702c28ba4a422cab9db2496dc94c55a18172c8bf WatchSource:0}: Error finding container c6ed84f0f4bbede85cba658f702c28ba4a422cab9db2496dc94c55a18172c8bf: Status 404 returned error can't find the container with id c6ed84f0f4bbede85cba658f702c28ba4a422cab9db2496dc94c55a18172c8bf
Sep 10 21:01:37.839157 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:37.839123    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f8755c64-p4llk" event={"ID":"d8ed697a-aa47-48c6-bbf6-561a2e969401","Type":"ContainerStarted","Data":"e8d69d673015ddd720f7ed141b38824fc228bf02854209e249394d30060c6d55"}
Sep 10 21:01:37.839500 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:37.839178    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f8755c64-p4llk" event={"ID":"d8ed697a-aa47-48c6-bbf6-561a2e969401","Type":"ContainerStarted","Data":"c6ed84f0f4bbede85cba658f702c28ba4a422cab9db2496dc94c55a18172c8bf"}
Sep 10 21:01:37.861733 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:37.861681    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79f8755c64-p4llk" podStartSLOduration=1.8616658849999999 podStartE2EDuration="1.861665885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:01:37.860151749 +0000 UTC m=+283.072616669" watchObservedRunningTime="2026-09-10 21:01:37.861665885 +0000 UTC m=+283.074130905"
Sep 10 21:01:46.838451 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:46.838407    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:46.838848 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:46.838481    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:46.843315 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:46.843292    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:46.877473 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:46.877448    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:01:46.927223 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:46.927193    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fc8c8b47c-b7nbc"]
Sep 10 21:01:55.389708 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:01:55.389681    2572 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 21:02:11.948879 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:11.948801    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5fc8c8b47c-b7nbc" podUID="dacefb4d-f6eb-4381-ba78-4d7c52f73220" containerName="console" containerID="cri-o://29bbbd7fc90cf8fb442446fc78014e3d597b34092ddb3247b4919798223a0276" gracePeriod=15
Sep 10 21:02:12.200672 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.200615    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 21:02:12.365696 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.365671    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dacefb4d-f6eb-4381-ba78-4d7c52f73220-trusted-ca-bundle\") pod \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") "
Sep 10 21:02:12.365838 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.365710    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dacefb4d-f6eb-4381-ba78-4d7c52f73220-console-oauth-config\") pod \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") "
Sep 10 21:02:12.365838 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.365728    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dacefb4d-f6eb-4381-ba78-4d7c52f73220-console-serving-cert\") pod \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") "
Sep 10 21:02:12.365838 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.365781    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dacefb4d-f6eb-4381-ba78-4d7c52f73220-console-config\") pod \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") "
Sep 10 21:02:12.365838 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.365808    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dacefb4d-f6eb-4381-ba78-4d7c52f73220-service-ca\") pod \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") "
Sep 10 21:02:12.366088 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.365848    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pzkd\" (UniqueName: \"kubernetes.io/projected/dacefb4d-f6eb-4381-ba78-4d7c52f73220-kube-api-access-5pzkd\") pod \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") "
Sep 10 21:02:12.366088 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.365871    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dacefb4d-f6eb-4381-ba78-4d7c52f73220-oauth-serving-cert\") pod \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\" (UID: \"dacefb4d-f6eb-4381-ba78-4d7c52f73220\") "
Sep 10 21:02:12.366341 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.366309    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dacefb4d-f6eb-4381-ba78-4d7c52f73220-service-ca" (OuterVolumeSpecName: "service-ca") pod "dacefb4d-f6eb-4381-ba78-4d7c52f73220" (UID: "dacefb4d-f6eb-4381-ba78-4d7c52f73220"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:02:12.366341 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.366329    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dacefb4d-f6eb-4381-ba78-4d7c52f73220-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dacefb4d-f6eb-4381-ba78-4d7c52f73220" (UID: "dacefb4d-f6eb-4381-ba78-4d7c52f73220"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:02:12.366480 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.366345    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dacefb4d-f6eb-4381-ba78-4d7c52f73220-console-config" (OuterVolumeSpecName: "console-config") pod "dacefb4d-f6eb-4381-ba78-4d7c52f73220" (UID: "dacefb4d-f6eb-4381-ba78-4d7c52f73220"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:02:12.366480 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.366377    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dacefb4d-f6eb-4381-ba78-4d7c52f73220-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dacefb4d-f6eb-4381-ba78-4d7c52f73220" (UID: "dacefb4d-f6eb-4381-ba78-4d7c52f73220"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:02:12.368408 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.368383    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dacefb4d-f6eb-4381-ba78-4d7c52f73220-kube-api-access-5pzkd" (OuterVolumeSpecName: "kube-api-access-5pzkd") pod "dacefb4d-f6eb-4381-ba78-4d7c52f73220" (UID: "dacefb4d-f6eb-4381-ba78-4d7c52f73220"). InnerVolumeSpecName "kube-api-access-5pzkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:02:12.368799 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.368770    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dacefb4d-f6eb-4381-ba78-4d7c52f73220-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dacefb4d-f6eb-4381-ba78-4d7c52f73220" (UID: "dacefb4d-f6eb-4381-ba78-4d7c52f73220"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:02:12.368799 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.368781    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dacefb4d-f6eb-4381-ba78-4d7c52f73220-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dacefb4d-f6eb-4381-ba78-4d7c52f73220" (UID: "dacefb4d-f6eb-4381-ba78-4d7c52f73220"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:02:12.466899 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.466850    2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dacefb4d-f6eb-4381-ba78-4d7c52f73220-console-config\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:02:12.466899 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.466870    2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dacefb4d-f6eb-4381-ba78-4d7c52f73220-service-ca\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:02:12.466899 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.466880    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pzkd\" (UniqueName: \"kubernetes.io/projected/dacefb4d-f6eb-4381-ba78-4d7c52f73220-kube-api-access-5pzkd\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:02:12.466899 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.466889    2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dacefb4d-f6eb-4381-ba78-4d7c52f73220-oauth-serving-cert\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:02:12.467102 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.466909    2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dacefb4d-f6eb-4381-ba78-4d7c52f73220-trusted-ca-bundle\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:02:12.467102 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.466919    2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dacefb4d-f6eb-4381-ba78-4d7c52f73220-console-oauth-config\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:02:12.467102 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.466927    2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dacefb4d-f6eb-4381-ba78-4d7c52f73220-console-serving-cert\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:02:12.979142 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.979108    2572 generic.go:356] "Generic (PLEG): container finished" podID="dacefb4d-f6eb-4381-ba78-4d7c52f73220" containerID="29bbbd7fc90cf8fb442446fc78014e3d597b34092ddb3247b4919798223a0276" exitCode=0
Sep 10 21:02:12.979530 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.979167    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fc8c8b47c-b7nbc"
Sep 10 21:02:12.979530 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.979206    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fc8c8b47c-b7nbc" event={"ID":"dacefb4d-f6eb-4381-ba78-4d7c52f73220","Type":"ContainerDied","Data":"29bbbd7fc90cf8fb442446fc78014e3d597b34092ddb3247b4919798223a0276"}
Sep 10 21:02:12.979530 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.979272    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fc8c8b47c-b7nbc" event={"ID":"dacefb4d-f6eb-4381-ba78-4d7c52f73220","Type":"ContainerDied","Data":"b7aaec2e426221a3f376c590aa6886d1ea02ba7aae83b54e4202d0222520574d"}
Sep 10 21:02:12.979530 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.979295    2572 scope.go:117] "RemoveContainer" containerID="29bbbd7fc90cf8fb442446fc78014e3d597b34092ddb3247b4919798223a0276"
Sep 10 21:02:12.988353 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.988333    2572 scope.go:117] "RemoveContainer" containerID="29bbbd7fc90cf8fb442446fc78014e3d597b34092ddb3247b4919798223a0276"
Sep 10 21:02:12.988650 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:02:12.988627    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29bbbd7fc90cf8fb442446fc78014e3d597b34092ddb3247b4919798223a0276\": container with ID starting with 29bbbd7fc90cf8fb442446fc78014e3d597b34092ddb3247b4919798223a0276 not found: ID does not exist" containerID="29bbbd7fc90cf8fb442446fc78014e3d597b34092ddb3247b4919798223a0276"
Sep 10 21:02:12.988724 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:12.988659    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29bbbd7fc90cf8fb442446fc78014e3d597b34092ddb3247b4919798223a0276"} err="failed to get container status \"29bbbd7fc90cf8fb442446fc78014e3d597b34092ddb3247b4919798223a0276\": rpc error: code = NotFound desc = could not find container \"29bbbd7fc90cf8fb442446fc78014e3d597b34092ddb3247b4919798223a0276\": container with ID starting with 29bbbd7fc90cf8fb442446fc78014e3d597b34092ddb3247b4919798223a0276 not found: ID does not exist"
Sep 10 21:02:13.009058 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:13.009031    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fc8c8b47c-b7nbc"]
Sep 10 21:02:13.015895 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:13.015873    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5fc8c8b47c-b7nbc"]
Sep 10 21:02:13.437791 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:13.437756    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dacefb4d-f6eb-4381-ba78-4d7c52f73220" path="/var/lib/kubelet/pods/dacefb4d-f6eb-4381-ba78-4d7c52f73220/volumes"
Sep 10 21:02:14.286594 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:14.286562    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-b99rl"]
Sep 10 21:02:14.286941 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:14.286878    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dacefb4d-f6eb-4381-ba78-4d7c52f73220" containerName="console"
Sep 10 21:02:14.286941 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:14.286890    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="dacefb4d-f6eb-4381-ba78-4d7c52f73220" containerName="console"
Sep 10 21:02:14.287071 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:14.286962    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="dacefb4d-f6eb-4381-ba78-4d7c52f73220" containerName="console"
Sep 10 21:02:14.291440 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:14.291421    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-b99rl"
Sep 10 21:02:14.295063 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:14.295042    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 21:02:14.295167 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:14.295041    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 21:02:14.295566 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:14.295548    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-rwx6g\""
Sep 10 21:02:14.302493 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:14.302465    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-b99rl"]
Sep 10 21:02:14.381540 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:14.381513    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ba3d08b-6049-44f1-857e-af6e8230abfc-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-b99rl\" (UID: \"6ba3d08b-6049-44f1-857e-af6e8230abfc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-b99rl"
Sep 10 21:02:14.381646 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:14.381553    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr9wq\" (UniqueName: \"kubernetes.io/projected/6ba3d08b-6049-44f1-857e-af6e8230abfc-kube-api-access-lr9wq\") pod \"cert-manager-webhook-6486ffd997-b99rl\" (UID: \"6ba3d08b-6049-44f1-857e-af6e8230abfc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-b99rl"
Sep 10 21:02:14.482946 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:14.482903    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ba3d08b-6049-44f1-857e-af6e8230abfc-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-b99rl\" (UID: \"6ba3d08b-6049-44f1-857e-af6e8230abfc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-b99rl"
Sep 10 21:02:14.483083 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:14.482966    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr9wq\" (UniqueName: \"kubernetes.io/projected/6ba3d08b-6049-44f1-857e-af6e8230abfc-kube-api-access-lr9wq\") pod \"cert-manager-webhook-6486ffd997-b99rl\" (UID: \"6ba3d08b-6049-44f1-857e-af6e8230abfc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-b99rl"
Sep 10 21:02:14.492039 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:14.491976    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ba3d08b-6049-44f1-857e-af6e8230abfc-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-b99rl\" (UID: \"6ba3d08b-6049-44f1-857e-af6e8230abfc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-b99rl"
Sep 10 21:02:14.492602 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:14.492582    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr9wq\" (UniqueName: \"kubernetes.io/projected/6ba3d08b-6049-44f1-857e-af6e8230abfc-kube-api-access-lr9wq\") pod \"cert-manager-webhook-6486ffd997-b99rl\" (UID: \"6ba3d08b-6049-44f1-857e-af6e8230abfc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-b99rl"
Sep 10 21:02:14.601721 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:14.601658    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-b99rl"
Sep 10 21:02:14.879735 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:14.879710    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-b99rl"]
Sep 10 21:02:14.882055 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:02:14.882025    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ba3d08b_6049_44f1_857e_af6e8230abfc.slice/crio-35bc42d7a0a4064f059c5139c49c50b7bbdd1e4fd20cd84fd0244e907feade25 WatchSource:0}: Error finding container 35bc42d7a0a4064f059c5139c49c50b7bbdd1e4fd20cd84fd0244e907feade25: Status 404 returned error can't find the container with id 35bc42d7a0a4064f059c5139c49c50b7bbdd1e4fd20cd84fd0244e907feade25
Sep 10 21:02:14.884194 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:14.884175    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:02:14.990319 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:14.990291    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-b99rl" event={"ID":"6ba3d08b-6049-44f1-857e-af6e8230abfc","Type":"ContainerStarted","Data":"35bc42d7a0a4064f059c5139c49c50b7bbdd1e4fd20cd84fd0244e907feade25"}
Sep 10 21:02:19.010506 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:19.010468    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-b99rl" event={"ID":"6ba3d08b-6049-44f1-857e-af6e8230abfc","Type":"ContainerStarted","Data":"d3530683e6f49d7694dbeb5ca347067dda159d747ece17a044e13bf5c93300e7"}
Sep 10 21:02:19.010894 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:19.010607    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-b99rl"
Sep 10 21:02:19.030939 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:19.030889    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-b99rl" podStartSLOduration=1.806470561 podStartE2EDuration="5.03087495s" totalImagesPullingTime="3.224404389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:02:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:02:19.029242356 +0000 UTC m=+324.241707276" watchObservedRunningTime="2026-09-10 21:02:19.03087495 +0000 UTC m=+324.243339868"
Sep 10 21:02:25.016780 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:25.016749    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-b99rl"
Sep 10 21:02:25.398375 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:25.398339    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vsscb"]
Sep 10 21:02:25.401734 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:25.401716    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-vsscb"
Sep 10 21:02:25.419202 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:25.419180    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-q28nc\""
Sep 10 21:02:25.438584 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:25.438559    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vsscb"]
Sep 10 21:02:25.567429 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:25.567405    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb8fb\" (UniqueName: \"kubernetes.io/projected/b315fa97-0df4-4e0f-9629-3a1b79ae22e5-kube-api-access-sb8fb\") pod \"cert-manager-55b4d954d5-vsscb\" (UID: \"b315fa97-0df4-4e0f-9629-3a1b79ae22e5\") " pod="cert-manager/cert-manager-55b4d954d5-vsscb"
Sep 10 21:02:25.567535 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:25.567442    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b315fa97-0df4-4e0f-9629-3a1b79ae22e5-bound-sa-token\") pod \"cert-manager-55b4d954d5-vsscb\" (UID: \"b315fa97-0df4-4e0f-9629-3a1b79ae22e5\") " pod="cert-manager/cert-manager-55b4d954d5-vsscb"
Sep 10 21:02:25.667878 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:25.667823    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb8fb\" (UniqueName: \"kubernetes.io/projected/b315fa97-0df4-4e0f-9629-3a1b79ae22e5-kube-api-access-sb8fb\") pod \"cert-manager-55b4d954d5-vsscb\" (UID: \"b315fa97-0df4-4e0f-9629-3a1b79ae22e5\") " pod="cert-manager/cert-manager-55b4d954d5-vsscb"
Sep 10 21:02:25.667878 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:25.667858    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b315fa97-0df4-4e0f-9629-3a1b79ae22e5-bound-sa-token\") pod \"cert-manager-55b4d954d5-vsscb\" (UID: \"b315fa97-0df4-4e0f-9629-3a1b79ae22e5\") " pod="cert-manager/cert-manager-55b4d954d5-vsscb"
Sep 10 21:02:25.676840 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:25.676808    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb8fb\" (UniqueName: \"kubernetes.io/projected/b315fa97-0df4-4e0f-9629-3a1b79ae22e5-kube-api-access-sb8fb\") pod \"cert-manager-55b4d954d5-vsscb\" (UID: \"b315fa97-0df4-4e0f-9629-3a1b79ae22e5\") " pod="cert-manager/cert-manager-55b4d954d5-vsscb"
Sep 10 21:02:25.676945 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:25.676867    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b315fa97-0df4-4e0f-9629-3a1b79ae22e5-bound-sa-token\") pod \"cert-manager-55b4d954d5-vsscb\" (UID: \"b315fa97-0df4-4e0f-9629-3a1b79ae22e5\") " pod="cert-manager/cert-manager-55b4d954d5-vsscb"
Sep 10 21:02:25.710878 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:25.710843    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-vsscb"
Sep 10 21:02:25.981935 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:25.981911    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vsscb"]
Sep 10 21:02:25.984294 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:02:25.984262    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb315fa97_0df4_4e0f_9629_3a1b79ae22e5.slice/crio-4d5789649f80604dbb3b5c5e422b99d2b5600c300159d9220850542796379c01 WatchSource:0}: Error finding container 4d5789649f80604dbb3b5c5e422b99d2b5600c300159d9220850542796379c01: Status 404 returned error can't find the container with id 4d5789649f80604dbb3b5c5e422b99d2b5600c300159d9220850542796379c01
Sep 10 21:02:26.040222 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:26.040192    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vsscb" event={"ID":"b315fa97-0df4-4e0f-9629-3a1b79ae22e5","Type":"ContainerStarted","Data":"4d5789649f80604dbb3b5c5e422b99d2b5600c300159d9220850542796379c01"}
Sep 10 21:02:27.053380 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:27.053324    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vsscb" event={"ID":"b315fa97-0df4-4e0f-9629-3a1b79ae22e5","Type":"ContainerStarted","Data":"2c1360dc6d0b6b7bdfdca4a2e28d8b9d19603d4a9365bb5c6e431aea30dae071"}
Sep 10 21:02:27.071768 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:27.071713    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-vsscb" podStartSLOduration=2.071698053 podStartE2EDuration="2.071698053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:02:27.071009608 +0000 UTC m=+332.283474523" watchObservedRunningTime="2026-09-10 21:02:27.071698053 +0000 UTC m=+332.284162972"
Sep 10 21:02:48.771614 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:48.771528    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr"]
Sep 10 21:02:48.774937 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:48.774916    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr"
Sep 10 21:02:48.779762 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:48.779718    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wgnwx\""
Sep 10 21:02:48.779926 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:48.779910    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 21:02:48.780019 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:48.779997    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 21:02:48.780087 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:48.780051    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 21:02:48.780146 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:48.780052    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 21:02:48.802864 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:48.802832    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr"]
Sep 10 21:02:48.834678 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:48.834654    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqc4j\" (UniqueName: \"kubernetes.io/projected/0556d7c8-57d9-4775-8532-1e260ea10416-kube-api-access-rqc4j\") pod \"opendatahub-operator-controller-manager-769bdd6b4-mfblr\" (UID: \"0556d7c8-57d9-4775-8532-1e260ea10416\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr"
Sep 10 21:02:48.834774 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:48.834704    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0556d7c8-57d9-4775-8532-1e260ea10416-apiservice-cert\") pod \"opendatahub-operator-controller-manager-769bdd6b4-mfblr\" (UID: \"0556d7c8-57d9-4775-8532-1e260ea10416\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr"
Sep 10 21:02:48.834774 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:48.834754    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0556d7c8-57d9-4775-8532-1e260ea10416-tmp\") pod \"opendatahub-operator-controller-manager-769bdd6b4-mfblr\" (UID: \"0556d7c8-57d9-4775-8532-1e260ea10416\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr"
Sep 10 21:02:48.834864 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:48.834786    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0556d7c8-57d9-4775-8532-1e260ea10416-manifests\") pod \"opendatahub-operator-controller-manager-769bdd6b4-mfblr\" (UID: \"0556d7c8-57d9-4775-8532-1e260ea10416\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr"
Sep 10 21:02:48.834864 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:48.834836    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0556d7c8-57d9-4775-8532-1e260ea10416-webhook-cert\") pod \"opendatahub-operator-controller-manager-769bdd6b4-mfblr\" (UID: \"0556d7c8-57d9-4775-8532-1e260ea10416\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr"
Sep 10 21:02:48.935779 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:48.935754    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqc4j\" (UniqueName: \"kubernetes.io/projected/0556d7c8-57d9-4775-8532-1e260ea10416-kube-api-access-rqc4j\") pod \"opendatahub-operator-controller-manager-769bdd6b4-mfblr\" (UID: \"0556d7c8-57d9-4775-8532-1e260ea10416\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr"
Sep 10 21:02:48.935875 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:48.935799    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0556d7c8-57d9-4775-8532-1e260ea10416-apiservice-cert\") pod \"opendatahub-operator-controller-manager-769bdd6b4-mfblr\" (UID: \"0556d7c8-57d9-4775-8532-1e260ea10416\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr"
Sep 10 21:02:48.935875 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:48.935820    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0556d7c8-57d9-4775-8532-1e260ea10416-tmp\") pod \"opendatahub-operator-controller-manager-769bdd6b4-mfblr\" (UID: \"0556d7c8-57d9-4775-8532-1e260ea10416\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr"
Sep 10 21:02:48.936105 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:48.936085    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0556d7c8-57d9-4775-8532-1e260ea10416-manifests\") pod \"opendatahub-operator-controller-manager-769bdd6b4-mfblr\" (UID: \"0556d7c8-57d9-4775-8532-1e260ea10416\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr"
Sep 10 21:02:48.936173 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:48.936141    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0556d7c8-57d9-4775-8532-1e260ea10416-webhook-cert\") pod \"opendatahub-operator-controller-manager-769bdd6b4-mfblr\" (UID: \"0556d7c8-57d9-4775-8532-1e260ea10416\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr"
Sep 10 21:02:48.936417 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:48.936387    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0556d7c8-57d9-4775-8532-1e260ea10416-manifests\") pod \"opendatahub-operator-controller-manager-769bdd6b4-mfblr\" (UID: \"0556d7c8-57d9-4775-8532-1e260ea10416\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr"
Sep 10 21:02:48.938235 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:48.938214    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0556d7c8-57d9-4775-8532-1e260ea10416-tmp\") pod \"opendatahub-operator-controller-manager-769bdd6b4-mfblr\" (UID: \"0556d7c8-57d9-4775-8532-1e260ea10416\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr"
Sep 10 21:02:48.938443 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:48.938421    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0556d7c8-57d9-4775-8532-1e260ea10416-apiservice-cert\") pod \"opendatahub-operator-controller-manager-769bdd6b4-mfblr\" (UID: \"0556d7c8-57d9-4775-8532-1e260ea10416\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr"
Sep 10 21:02:48.938691 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:48.938673    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0556d7c8-57d9-4775-8532-1e260ea10416-webhook-cert\") pod \"opendatahub-operator-controller-manager-769bdd6b4-mfblr\" (UID: \"0556d7c8-57d9-4775-8532-1e260ea10416\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr"
Sep 10 21:02:48.958196 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:48.958174    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqc4j\" (UniqueName: \"kubernetes.io/projected/0556d7c8-57d9-4775-8532-1e260ea10416-kube-api-access-rqc4j\") pod \"opendatahub-operator-controller-manager-769bdd6b4-mfblr\" (UID: \"0556d7c8-57d9-4775-8532-1e260ea10416\") " pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr"
Sep 10 21:02:49.086108 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:49.086074    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr"
Sep 10 21:02:49.368437 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:49.368415    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr"]
Sep 10 21:02:49.370148 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:02:49.370112    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0556d7c8_57d9_4775_8532_1e260ea10416.slice/crio-c526f7008ef997831fc9d95215b6519013ff116cdfcdf96092234bf6bd98e9f0 WatchSource:0}: Error finding container c526f7008ef997831fc9d95215b6519013ff116cdfcdf96092234bf6bd98e9f0: Status 404 returned error can't find the container with id c526f7008ef997831fc9d95215b6519013ff116cdfcdf96092234bf6bd98e9f0
Sep 10 21:02:50.153023 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:50.152967    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr" event={"ID":"0556d7c8-57d9-4775-8532-1e260ea10416","Type":"ContainerStarted","Data":"c526f7008ef997831fc9d95215b6519013ff116cdfcdf96092234bf6bd98e9f0"}
Sep 10 21:02:53.174269 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:53.174236    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr" event={"ID":"0556d7c8-57d9-4775-8532-1e260ea10416","Type":"ContainerStarted","Data":"100319175b59c9f533958320b3701f2f7c7e22b27274e81be4e4fed6f137546a"}
Sep 10 21:02:54.180392 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:54.180360    2572 generic.go:356] "Generic (PLEG): container finished" podID="0556d7c8-57d9-4775-8532-1e260ea10416" containerID="100319175b59c9f533958320b3701f2f7c7e22b27274e81be4e4fed6f137546a" exitCode=0
Sep 10 21:02:54.180733 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:54.180446    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr" event={"ID":"0556d7c8-57d9-4775-8532-1e260ea10416","Type":"ContainerDied","Data":"100319175b59c9f533958320b3701f2f7c7e22b27274e81be4e4fed6f137546a"}
Sep 10 21:02:55.187243 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:55.187209    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr" event={"ID":"0556d7c8-57d9-4775-8532-1e260ea10416","Type":"ContainerStarted","Data":"8852845064c595fff6fa346ac39746d8e3275a8aab6bd0f9816991f873b40906"}
Sep 10 21:02:55.187639 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:55.187335    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr"
Sep 10 21:02:55.214239 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:02:55.214189    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr" podStartSLOduration=2.796607508 podStartE2EDuration="7.214170716s" totalImagesPullingTime="3.343026549s" totalInitContainerRuntime="1.074536659s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:02:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:02:55.21211272 +0000 UTC m=+360.424577642" watchObservedRunningTime="2026-09-10 21:02:55.214170716 +0000 UTC m=+360.426635636"
Sep 10 21:03:06.194057 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:06.194027    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-769bdd6b4-mfblr"
Sep 10 21:03:08.400034 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.400000    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-98c4dbc44-qkwf9"]
Sep 10 21:03:08.415933 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.415904    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-98c4dbc44-qkwf9"
Sep 10 21:03:08.418402 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.418376    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 21:03:08.418577 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.418511    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fh2kc\""
Sep 10 21:03:08.419270 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.419245    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 21:03:08.419432 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.419409    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-98c4dbc44-qkwf9"]
Sep 10 21:03:08.419760 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.419702    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 21:03:08.419760 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.419711    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 21:03:08.480299 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.480260    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgghf\" (UniqueName: \"kubernetes.io/projected/b5e5879b-760c-457b-88d4-6a1734f02c17-kube-api-access-dgghf\") pod \"kube-auth-proxy-98c4dbc44-qkwf9\" (UID: \"b5e5879b-760c-457b-88d4-6a1734f02c17\") " pod="openshift-ingress/kube-auth-proxy-98c4dbc44-qkwf9"
Sep 10 21:03:08.480466 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.480344    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5e5879b-760c-457b-88d4-6a1734f02c17-tmp\") pod \"kube-auth-proxy-98c4dbc44-qkwf9\" (UID: \"b5e5879b-760c-457b-88d4-6a1734f02c17\") " pod="openshift-ingress/kube-auth-proxy-98c4dbc44-qkwf9"
Sep 10 21:03:08.480466 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.480446    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5e5879b-760c-457b-88d4-6a1734f02c17-tls-certs\") pod \"kube-auth-proxy-98c4dbc44-qkwf9\" (UID: \"b5e5879b-760c-457b-88d4-6a1734f02c17\") " pod="openshift-ingress/kube-auth-proxy-98c4dbc44-qkwf9"
Sep 10 21:03:08.581521 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.581494    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5e5879b-760c-457b-88d4-6a1734f02c17-tls-certs\") pod \"kube-auth-proxy-98c4dbc44-qkwf9\" (UID: \"b5e5879b-760c-457b-88d4-6a1734f02c17\") " pod="openshift-ingress/kube-auth-proxy-98c4dbc44-qkwf9"
Sep 10 21:03:08.581654 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.581530    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgghf\" (UniqueName: \"kubernetes.io/projected/b5e5879b-760c-457b-88d4-6a1734f02c17-kube-api-access-dgghf\") pod \"kube-auth-proxy-98c4dbc44-qkwf9\" (UID: \"b5e5879b-760c-457b-88d4-6a1734f02c17\") " pod="openshift-ingress/kube-auth-proxy-98c4dbc44-qkwf9"
Sep 10 21:03:08.581654 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.581575    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5e5879b-760c-457b-88d4-6a1734f02c17-tmp\") pod \"kube-auth-proxy-98c4dbc44-qkwf9\" (UID: \"b5e5879b-760c-457b-88d4-6a1734f02c17\") " pod="openshift-ingress/kube-auth-proxy-98c4dbc44-qkwf9"
Sep 10 21:03:08.581654 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:03:08.581646    2572 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 10 21:03:08.581753 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:03:08.581728    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5e5879b-760c-457b-88d4-6a1734f02c17-tls-certs podName:b5e5879b-760c-457b-88d4-6a1734f02c17 nodeName:}" failed. No retries permitted until 2026-09-10 21:03:09.081708339 +0000 UTC m=+374.294173237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b5e5879b-760c-457b-88d4-6a1734f02c17-tls-certs") pod "kube-auth-proxy-98c4dbc44-qkwf9" (UID: "b5e5879b-760c-457b-88d4-6a1734f02c17") : secret "kube-auth-proxy-tls" not found
Sep 10 21:03:08.583762 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.583743    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5e5879b-760c-457b-88d4-6a1734f02c17-tmp\") pod \"kube-auth-proxy-98c4dbc44-qkwf9\" (UID: \"b5e5879b-760c-457b-88d4-6a1734f02c17\") " pod="openshift-ingress/kube-auth-proxy-98c4dbc44-qkwf9"
Sep 10 21:03:08.599475 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.599451    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgghf\" (UniqueName: \"kubernetes.io/projected/b5e5879b-760c-457b-88d4-6a1734f02c17-kube-api-access-dgghf\") pod \"kube-auth-proxy-98c4dbc44-qkwf9\" (UID: \"b5e5879b-760c-457b-88d4-6a1734f02c17\") " pod="openshift-ingress/kube-auth-proxy-98c4dbc44-qkwf9"
Sep 10 21:03:08.797737 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.797702    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-799dfcfb97-kdnsc"]
Sep 10 21:03:08.800597 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.800580    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kdnsc"
Sep 10 21:03:08.803237 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.803202    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 21:03:08.803356 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.803246    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-lwtnn\""
Sep 10 21:03:08.803356 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.803264    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 21:03:08.803516 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.803502    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 21:03:08.803516 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.803512    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 21:03:08.803613 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.803504    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 21:03:08.813737 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.813718    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-799dfcfb97-kdnsc"]
Sep 10 21:03:08.884034 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.884014    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8f7828f0-e34a-46df-aaff-59857058ebc3-manager-config\") pod \"lws-controller-manager-799dfcfb97-kdnsc\" (UID: \"8f7828f0-e34a-46df-aaff-59857058ebc3\") " pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kdnsc"
Sep 10 21:03:08.884153 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.884056    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f7828f0-e34a-46df-aaff-59857058ebc3-metrics-cert\") pod \"lws-controller-manager-799dfcfb97-kdnsc\" (UID: \"8f7828f0-e34a-46df-aaff-59857058ebc3\") " pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kdnsc"
Sep 10 21:03:08.884153 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.884091    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f7828f0-e34a-46df-aaff-59857058ebc3-cert\") pod \"lws-controller-manager-799dfcfb97-kdnsc\" (UID: \"8f7828f0-e34a-46df-aaff-59857058ebc3\") " pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kdnsc"
Sep 10 21:03:08.884153 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.884138    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p67mq\" (UniqueName: \"kubernetes.io/projected/8f7828f0-e34a-46df-aaff-59857058ebc3-kube-api-access-p67mq\") pod \"lws-controller-manager-799dfcfb97-kdnsc\" (UID: \"8f7828f0-e34a-46df-aaff-59857058ebc3\") " pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kdnsc"
Sep 10 21:03:08.984977 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.984950    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f7828f0-e34a-46df-aaff-59857058ebc3-cert\") pod \"lws-controller-manager-799dfcfb97-kdnsc\" (UID: \"8f7828f0-e34a-46df-aaff-59857058ebc3\") " pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kdnsc"
Sep 10 21:03:08.984977 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.984996    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p67mq\" (UniqueName: \"kubernetes.io/projected/8f7828f0-e34a-46df-aaff-59857058ebc3-kube-api-access-p67mq\") pod \"lws-controller-manager-799dfcfb97-kdnsc\" (UID: \"8f7828f0-e34a-46df-aaff-59857058ebc3\") " pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kdnsc"
Sep 10 21:03:08.985179 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.985157    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8f7828f0-e34a-46df-aaff-59857058ebc3-manager-config\") pod \"lws-controller-manager-799dfcfb97-kdnsc\" (UID: \"8f7828f0-e34a-46df-aaff-59857058ebc3\") " pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kdnsc"
Sep 10 21:03:08.985240 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.985226    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f7828f0-e34a-46df-aaff-59857058ebc3-metrics-cert\") pod \"lws-controller-manager-799dfcfb97-kdnsc\" (UID: \"8f7828f0-e34a-46df-aaff-59857058ebc3\") " pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kdnsc"
Sep 10 21:03:08.985773 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.985754    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8f7828f0-e34a-46df-aaff-59857058ebc3-manager-config\") pod \"lws-controller-manager-799dfcfb97-kdnsc\" (UID: \"8f7828f0-e34a-46df-aaff-59857058ebc3\") " pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kdnsc"
Sep 10 21:03:08.987544 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.987523    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f7828f0-e34a-46df-aaff-59857058ebc3-cert\") pod \"lws-controller-manager-799dfcfb97-kdnsc\" (UID: \"8f7828f0-e34a-46df-aaff-59857058ebc3\") " pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kdnsc"
Sep 10 21:03:08.987630 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.987600    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f7828f0-e34a-46df-aaff-59857058ebc3-metrics-cert\") pod \"lws-controller-manager-799dfcfb97-kdnsc\" (UID: \"8f7828f0-e34a-46df-aaff-59857058ebc3\") " pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kdnsc"
Sep 10 21:03:08.994826 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:08.994802    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p67mq\" (UniqueName: \"kubernetes.io/projected/8f7828f0-e34a-46df-aaff-59857058ebc3-kube-api-access-p67mq\") pod \"lws-controller-manager-799dfcfb97-kdnsc\" (UID: \"8f7828f0-e34a-46df-aaff-59857058ebc3\") " pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kdnsc"
Sep 10 21:03:09.086172 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:09.086104    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5e5879b-760c-457b-88d4-6a1734f02c17-tls-certs\") pod \"kube-auth-proxy-98c4dbc44-qkwf9\" (UID: \"b5e5879b-760c-457b-88d4-6a1734f02c17\") " pod="openshift-ingress/kube-auth-proxy-98c4dbc44-qkwf9"
Sep 10 21:03:09.088432 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:09.088414    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5e5879b-760c-457b-88d4-6a1734f02c17-tls-certs\") pod \"kube-auth-proxy-98c4dbc44-qkwf9\" (UID: \"b5e5879b-760c-457b-88d4-6a1734f02c17\") " pod="openshift-ingress/kube-auth-proxy-98c4dbc44-qkwf9"
Sep 10 21:03:09.110432 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:09.110413    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kdnsc"
Sep 10 21:03:09.326910 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:09.326871    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-98c4dbc44-qkwf9"
Sep 10 21:03:09.389723 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:09.389701    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-799dfcfb97-kdnsc"]
Sep 10 21:03:09.391656 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:03:09.391628    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7828f0_e34a_46df_aaff_59857058ebc3.slice/crio-4aa37318691215c05f56ef4b2ca1af4c0176d11cfd17964d4c78f293df0afc74 WatchSource:0}: Error finding container 4aa37318691215c05f56ef4b2ca1af4c0176d11cfd17964d4c78f293df0afc74: Status 404 returned error can't find the container with id 4aa37318691215c05f56ef4b2ca1af4c0176d11cfd17964d4c78f293df0afc74
Sep 10 21:03:09.600665 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:09.600643    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-98c4dbc44-qkwf9"]
Sep 10 21:03:09.603010 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:03:09.602960    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e5879b_760c_457b_88d4_6a1734f02c17.slice/crio-f49a6ea40c72169df7c1f37d12cf93614ced50b22af8e24327e6915117ba5f4d WatchSource:0}: Error finding container f49a6ea40c72169df7c1f37d12cf93614ced50b22af8e24327e6915117ba5f4d: Status 404 returned error can't find the container with id f49a6ea40c72169df7c1f37d12cf93614ced50b22af8e24327e6915117ba5f4d
Sep 10 21:03:10.259011 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:10.258927    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kdnsc" event={"ID":"8f7828f0-e34a-46df-aaff-59857058ebc3","Type":"ContainerStarted","Data":"4aa37318691215c05f56ef4b2ca1af4c0176d11cfd17964d4c78f293df0afc74"}
Sep 10 21:03:10.261236 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:10.261133    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-98c4dbc44-qkwf9" event={"ID":"b5e5879b-760c-457b-88d4-6a1734f02c17","Type":"ContainerStarted","Data":"f49a6ea40c72169df7c1f37d12cf93614ced50b22af8e24327e6915117ba5f4d"}
Sep 10 21:03:13.277785 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:13.277704    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kdnsc" event={"ID":"8f7828f0-e34a-46df-aaff-59857058ebc3","Type":"ContainerStarted","Data":"d92af75650b44d75af0b1c7abe0c57b5c0f815dc53cd1757109a5f43f695498a"}
Sep 10 21:03:13.278192 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:13.277818    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kdnsc"
Sep 10 21:03:13.282022 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:13.281970    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-98c4dbc44-qkwf9" event={"ID":"b5e5879b-760c-457b-88d4-6a1734f02c17","Type":"ContainerStarted","Data":"80ce56bb9d863156e2640fda57795c9d391c996006ce59084a0877b8cc3355c0"}
Sep 10 21:03:13.295942 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:13.295902    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kdnsc" podStartSLOduration=1.665130799 podStartE2EDuration="5.295891673s" totalImagesPullingTime="3.630760874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:13.295201762 +0000 UTC m=+378.507666680" watchObservedRunningTime="2026-09-10 21:03:13.295891673 +0000 UTC m=+378.508356589"
Sep 10 21:03:13.312129 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:13.312091    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-98c4dbc44-qkwf9" podStartSLOduration=1.8481033409999998 podStartE2EDuration="5.312077312s" totalImagesPullingTime="3.463973971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:13.311052194 +0000 UTC m=+378.523517112" watchObservedRunningTime="2026-09-10 21:03:13.312077312 +0000 UTC m=+378.524542231"
Sep 10 21:03:24.289765 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:03:24.289727    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-799dfcfb97-kdnsc"
Sep 10 21:04:05.708581 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:05.708544    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79f8755c64-p4llk"]
Sep 10 21:04:30.729677 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:30.729579    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-79f8755c64-p4llk" podUID="d8ed697a-aa47-48c6-bbf6-561a2e969401" containerName="console" containerID="cri-o://e8d69d673015ddd720f7ed141b38824fc228bf02854209e249394d30060c6d55" gracePeriod=15
Sep 10 21:04:30.980379 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:30.980331    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:04:31.081714 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.081689    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8ed697a-aa47-48c6-bbf6-561a2e969401-console-serving-cert\") pod \"d8ed697a-aa47-48c6-bbf6-561a2e969401\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") "
Sep 10 21:04:31.081854 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.081726    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8ed697a-aa47-48c6-bbf6-561a2e969401-console-config\") pod \"d8ed697a-aa47-48c6-bbf6-561a2e969401\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") "
Sep 10 21:04:31.081854 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.081747    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8ed697a-aa47-48c6-bbf6-561a2e969401-trusted-ca-bundle\") pod \"d8ed697a-aa47-48c6-bbf6-561a2e969401\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") "
Sep 10 21:04:31.081854 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.081771    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8ed697a-aa47-48c6-bbf6-561a2e969401-console-oauth-config\") pod \"d8ed697a-aa47-48c6-bbf6-561a2e969401\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") "
Sep 10 21:04:31.081854 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.081796    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb5qz\" (UniqueName: \"kubernetes.io/projected/d8ed697a-aa47-48c6-bbf6-561a2e969401-kube-api-access-gb5qz\") pod \"d8ed697a-aa47-48c6-bbf6-561a2e969401\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") "
Sep 10 21:04:31.081854 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.081818    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8ed697a-aa47-48c6-bbf6-561a2e969401-service-ca\") pod \"d8ed697a-aa47-48c6-bbf6-561a2e969401\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") "
Sep 10 21:04:31.081854 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.081848    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8ed697a-aa47-48c6-bbf6-561a2e969401-oauth-serving-cert\") pod \"d8ed697a-aa47-48c6-bbf6-561a2e969401\" (UID: \"d8ed697a-aa47-48c6-bbf6-561a2e969401\") "
Sep 10 21:04:31.082206 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.082112    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8ed697a-aa47-48c6-bbf6-561a2e969401-console-config" (OuterVolumeSpecName: "console-config") pod "d8ed697a-aa47-48c6-bbf6-561a2e969401" (UID: "d8ed697a-aa47-48c6-bbf6-561a2e969401"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:04:31.082318 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.082292    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8ed697a-aa47-48c6-bbf6-561a2e969401-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d8ed697a-aa47-48c6-bbf6-561a2e969401" (UID: "d8ed697a-aa47-48c6-bbf6-561a2e969401"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:04:31.082385 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.082360    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8ed697a-aa47-48c6-bbf6-561a2e969401-service-ca" (OuterVolumeSpecName: "service-ca") pod "d8ed697a-aa47-48c6-bbf6-561a2e969401" (UID: "d8ed697a-aa47-48c6-bbf6-561a2e969401"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:04:31.082428 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.082393    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8ed697a-aa47-48c6-bbf6-561a2e969401-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d8ed697a-aa47-48c6-bbf6-561a2e969401" (UID: "d8ed697a-aa47-48c6-bbf6-561a2e969401"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:04:31.084089 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.084071    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8ed697a-aa47-48c6-bbf6-561a2e969401-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d8ed697a-aa47-48c6-bbf6-561a2e969401" (UID: "d8ed697a-aa47-48c6-bbf6-561a2e969401"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:04:31.084656 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.084640    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8ed697a-aa47-48c6-bbf6-561a2e969401-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d8ed697a-aa47-48c6-bbf6-561a2e969401" (UID: "d8ed697a-aa47-48c6-bbf6-561a2e969401"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:04:31.084714 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.084681    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ed697a-aa47-48c6-bbf6-561a2e969401-kube-api-access-gb5qz" (OuterVolumeSpecName: "kube-api-access-gb5qz") pod "d8ed697a-aa47-48c6-bbf6-561a2e969401" (UID: "d8ed697a-aa47-48c6-bbf6-561a2e969401"). InnerVolumeSpecName "kube-api-access-gb5qz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:04:31.182682 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.182653    2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8ed697a-aa47-48c6-bbf6-561a2e969401-console-config\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:04:31.182682 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.182678    2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8ed697a-aa47-48c6-bbf6-561a2e969401-trusted-ca-bundle\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:04:31.182821 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.182693    2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8ed697a-aa47-48c6-bbf6-561a2e969401-console-oauth-config\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:04:31.182821 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.182707    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb5qz\" (UniqueName: \"kubernetes.io/projected/d8ed697a-aa47-48c6-bbf6-561a2e969401-kube-api-access-gb5qz\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:04:31.182821 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.182720    2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8ed697a-aa47-48c6-bbf6-561a2e969401-service-ca\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:04:31.182821 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.182732    2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8ed697a-aa47-48c6-bbf6-561a2e969401-oauth-serving-cert\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:04:31.182821 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.182744    2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8ed697a-aa47-48c6-bbf6-561a2e969401-console-serving-cert\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:04:31.633554 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.633516    2572 generic.go:356] "Generic (PLEG): container finished" podID="d8ed697a-aa47-48c6-bbf6-561a2e969401" containerID="e8d69d673015ddd720f7ed141b38824fc228bf02854209e249394d30060c6d55" exitCode=0
Sep 10 21:04:31.633672 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.633584    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f8755c64-p4llk"
Sep 10 21:04:31.633672 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.633601    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f8755c64-p4llk" event={"ID":"d8ed697a-aa47-48c6-bbf6-561a2e969401","Type":"ContainerDied","Data":"e8d69d673015ddd720f7ed141b38824fc228bf02854209e249394d30060c6d55"}
Sep 10 21:04:31.633769 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.633678    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f8755c64-p4llk" event={"ID":"d8ed697a-aa47-48c6-bbf6-561a2e969401","Type":"ContainerDied","Data":"c6ed84f0f4bbede85cba658f702c28ba4a422cab9db2496dc94c55a18172c8bf"}
Sep 10 21:04:31.633769 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.633707    2572 scope.go:117] "RemoveContainer" containerID="e8d69d673015ddd720f7ed141b38824fc228bf02854209e249394d30060c6d55"
Sep 10 21:04:31.642896 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.642878    2572 scope.go:117] "RemoveContainer" containerID="e8d69d673015ddd720f7ed141b38824fc228bf02854209e249394d30060c6d55"
Sep 10 21:04:31.643253 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:04:31.643232    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d69d673015ddd720f7ed141b38824fc228bf02854209e249394d30060c6d55\": container with ID starting with e8d69d673015ddd720f7ed141b38824fc228bf02854209e249394d30060c6d55 not found: ID does not exist" containerID="e8d69d673015ddd720f7ed141b38824fc228bf02854209e249394d30060c6d55"
Sep 10 21:04:31.643326 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.643262    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d69d673015ddd720f7ed141b38824fc228bf02854209e249394d30060c6d55"} err="failed to get container status \"e8d69d673015ddd720f7ed141b38824fc228bf02854209e249394d30060c6d55\": rpc error: code = NotFound desc = could not find container \"e8d69d673015ddd720f7ed141b38824fc228bf02854209e249394d30060c6d55\": container with ID starting with e8d69d673015ddd720f7ed141b38824fc228bf02854209e249394d30060c6d55 not found: ID does not exist"
Sep 10 21:04:31.651044 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.651021    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79f8755c64-p4llk"]
Sep 10 21:04:31.655135 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:31.655112    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79f8755c64-p4llk"]
Sep 10 21:04:33.439283 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:33.439243    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8ed697a-aa47-48c6-bbf6-561a2e969401" path="/var/lib/kubelet/pods/d8ed697a-aa47-48c6-bbf6-561a2e969401/volumes"
Sep 10 21:04:52.472392 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:52.472352    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9v9sk"]
Sep 10 21:04:52.472869 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:52.472672    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8ed697a-aa47-48c6-bbf6-561a2e969401" containerName="console"
Sep 10 21:04:52.472869 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:52.472685    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8ed697a-aa47-48c6-bbf6-561a2e969401" containerName="console"
Sep 10 21:04:52.472869 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:52.472740    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8ed697a-aa47-48c6-bbf6-561a2e969401" containerName="console"
Sep 10 21:04:52.475619 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:52.475593    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9v9sk"
Sep 10 21:04:52.479561 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:52.479530    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 10 21:04:52.479686 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:52.479544    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 21:04:52.479686 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:52.479624    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 10 21:04:52.479686 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:52.479584    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6w8c5\""
Sep 10 21:04:52.480152 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:52.480130    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 21:04:52.501229 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:52.501198    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9v9sk"]
Sep 10 21:04:52.547621 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:52.547596    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/58f8cb50-eaad-47f3-878c-2930d88a8d06-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9v9sk\" (UID: \"58f8cb50-eaad-47f3-878c-2930d88a8d06\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9v9sk"
Sep 10 21:04:52.548111 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:52.548092    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/58f8cb50-eaad-47f3-878c-2930d88a8d06-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9v9sk\" (UID: \"58f8cb50-eaad-47f3-878c-2930d88a8d06\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9v9sk"
Sep 10 21:04:52.548235 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:52.548217    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c748\" (UniqueName: \"kubernetes.io/projected/58f8cb50-eaad-47f3-878c-2930d88a8d06-kube-api-access-4c748\") pod \"kuadrant-console-plugin-fbdd4f445-9v9sk\" (UID: \"58f8cb50-eaad-47f3-878c-2930d88a8d06\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9v9sk"
Sep 10 21:04:52.649136 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:52.649106    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/58f8cb50-eaad-47f3-878c-2930d88a8d06-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9v9sk\" (UID: \"58f8cb50-eaad-47f3-878c-2930d88a8d06\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9v9sk"
Sep 10 21:04:52.649246 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:52.649159    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/58f8cb50-eaad-47f3-878c-2930d88a8d06-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9v9sk\" (UID: \"58f8cb50-eaad-47f3-878c-2930d88a8d06\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9v9sk"
Sep 10 21:04:52.649246 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:52.649199    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c748\" (UniqueName: \"kubernetes.io/projected/58f8cb50-eaad-47f3-878c-2930d88a8d06-kube-api-access-4c748\") pod \"kuadrant-console-plugin-fbdd4f445-9v9sk\" (UID: \"58f8cb50-eaad-47f3-878c-2930d88a8d06\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9v9sk"
Sep 10 21:04:52.649422 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:04:52.649312    2572 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 10 21:04:52.649422 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:04:52.649396    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58f8cb50-eaad-47f3-878c-2930d88a8d06-plugin-serving-cert podName:58f8cb50-eaad-47f3-878c-2930d88a8d06 nodeName:}" failed. No retries permitted until 2026-09-10 21:04:53.149373451 +0000 UTC m=+478.361838361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/58f8cb50-eaad-47f3-878c-2930d88a8d06-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-9v9sk" (UID: "58f8cb50-eaad-47f3-878c-2930d88a8d06") : secret "plugin-serving-cert" not found
Sep 10 21:04:52.649799 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:52.649780    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/58f8cb50-eaad-47f3-878c-2930d88a8d06-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9v9sk\" (UID: \"58f8cb50-eaad-47f3-878c-2930d88a8d06\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9v9sk"
Sep 10 21:04:52.671816 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:52.671793    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c748\" (UniqueName: \"kubernetes.io/projected/58f8cb50-eaad-47f3-878c-2930d88a8d06-kube-api-access-4c748\") pod \"kuadrant-console-plugin-fbdd4f445-9v9sk\" (UID: \"58f8cb50-eaad-47f3-878c-2930d88a8d06\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9v9sk"
Sep 10 21:04:53.152504 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:53.152469    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/58f8cb50-eaad-47f3-878c-2930d88a8d06-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9v9sk\" (UID: \"58f8cb50-eaad-47f3-878c-2930d88a8d06\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9v9sk"
Sep 10 21:04:53.155052 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:53.155029    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/58f8cb50-eaad-47f3-878c-2930d88a8d06-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9v9sk\" (UID: \"58f8cb50-eaad-47f3-878c-2930d88a8d06\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9v9sk"
Sep 10 21:04:53.385678 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:53.385648    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9v9sk"
Sep 10 21:04:53.672210 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:53.672181    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9v9sk"]
Sep 10 21:04:53.676325 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:04:53.676299    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58f8cb50_eaad_47f3_878c_2930d88a8d06.slice/crio-0fd51b0b1c5ae3da8e3d3a687bfe8a4e84a9a5b8d043b69b6aeee229286c8539 WatchSource:0}: Error finding container 0fd51b0b1c5ae3da8e3d3a687bfe8a4e84a9a5b8d043b69b6aeee229286c8539: Status 404 returned error can't find the container with id 0fd51b0b1c5ae3da8e3d3a687bfe8a4e84a9a5b8d043b69b6aeee229286c8539
Sep 10 21:04:53.739442 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:53.739386    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9v9sk" event={"ID":"58f8cb50-eaad-47f3-878c-2930d88a8d06","Type":"ContainerStarted","Data":"0fd51b0b1c5ae3da8e3d3a687bfe8a4e84a9a5b8d043b69b6aeee229286c8539"}
Sep 10 21:04:58.765650 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:58.765610    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9v9sk" event={"ID":"58f8cb50-eaad-47f3-878c-2930d88a8d06","Type":"ContainerStarted","Data":"15d5dfcf8f491033f102f2fc1ed202a0141a481a0e0fd5f2d7f059e36bef0a3e"}
Sep 10 21:04:58.794938 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:04:58.794888    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9v9sk" podStartSLOduration=2.3307305879999998 podStartE2EDuration="6.794870215s" totalImagesPullingTime="4.464139627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:04:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:04:58.79375246 +0000 UTC m=+484.006217380" watchObservedRunningTime="2026-09-10 21:04:58.794870215 +0000 UTC m=+484.007335134"
Sep 10 21:05:50.344319 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:50.344223    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:05:50.346757 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:50.346732    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"
Sep 10 21:05:50.349355 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:50.349330    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 21:05:50.361920 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:50.361881    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:05:50.375348 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:50.375318    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:05:50.381018 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:50.380974    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntnmv\" (UniqueName: \"kubernetes.io/projected/4c3ebeca-4c69-4c1a-8f5e-6939411484a0-kube-api-access-ntnmv\") pod \"limitador-limitador-84994f7ffc-n4h5b\" (UID: \"4c3ebeca-4c69-4c1a-8f5e-6939411484a0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"
Sep 10 21:05:50.381117 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:50.381088    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4c3ebeca-4c69-4c1a-8f5e-6939411484a0-config-file\") pod \"limitador-limitador-84994f7ffc-n4h5b\" (UID: \"4c3ebeca-4c69-4c1a-8f5e-6939411484a0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"
Sep 10 21:05:50.481434 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:50.481404    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntnmv\" (UniqueName: \"kubernetes.io/projected/4c3ebeca-4c69-4c1a-8f5e-6939411484a0-kube-api-access-ntnmv\") pod \"limitador-limitador-84994f7ffc-n4h5b\" (UID: \"4c3ebeca-4c69-4c1a-8f5e-6939411484a0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"
Sep 10 21:05:50.481543 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:50.481460    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4c3ebeca-4c69-4c1a-8f5e-6939411484a0-config-file\") pod \"limitador-limitador-84994f7ffc-n4h5b\" (UID: \"4c3ebeca-4c69-4c1a-8f5e-6939411484a0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"
Sep 10 21:05:50.482073 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:50.482052    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4c3ebeca-4c69-4c1a-8f5e-6939411484a0-config-file\") pod \"limitador-limitador-84994f7ffc-n4h5b\" (UID: \"4c3ebeca-4c69-4c1a-8f5e-6939411484a0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"
Sep 10 21:05:50.492956 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:50.492925    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntnmv\" (UniqueName: \"kubernetes.io/projected/4c3ebeca-4c69-4c1a-8f5e-6939411484a0-kube-api-access-ntnmv\") pod \"limitador-limitador-84994f7ffc-n4h5b\" (UID: \"4c3ebeca-4c69-4c1a-8f5e-6939411484a0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"
Sep 10 21:05:50.657141 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:50.657083    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"
Sep 10 21:05:50.793571 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:50.793533    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jdw6r"]
Sep 10 21:05:50.795838 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:50.795820    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-jdw6r"
Sep 10 21:05:50.798396 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:50.798380    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cqhw2\""
Sep 10 21:05:50.804711 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:50.804685    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jdw6r"]
Sep 10 21:05:50.884717 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:50.884675    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtnz9\" (UniqueName: \"kubernetes.io/projected/842f26f9-418e-4106-807c-ba828d837f0a-kube-api-access-wtnz9\") pod \"authorino-7bcfd8f9cd-jdw6r\" (UID: \"842f26f9-418e-4106-807c-ba828d837f0a\") " pod="kuadrant-system/authorino-7bcfd8f9cd-jdw6r"
Sep 10 21:05:50.943569 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:50.943551    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:05:50.946244 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:05:50.946217    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c3ebeca_4c69_4c1a_8f5e_6939411484a0.slice/crio-aa610b5e7e07cf2dcd56eb32b8155654355d2731e149f6eeeed358528af242a2 WatchSource:0}: Error finding container aa610b5e7e07cf2dcd56eb32b8155654355d2731e149f6eeeed358528af242a2: Status 404 returned error can't find the container with id aa610b5e7e07cf2dcd56eb32b8155654355d2731e149f6eeeed358528af242a2
Sep 10 21:05:50.985674 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:50.985649    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtnz9\" (UniqueName: \"kubernetes.io/projected/842f26f9-418e-4106-807c-ba828d837f0a-kube-api-access-wtnz9\") pod \"authorino-7bcfd8f9cd-jdw6r\" (UID: \"842f26f9-418e-4106-807c-ba828d837f0a\") " pod="kuadrant-system/authorino-7bcfd8f9cd-jdw6r"
Sep 10 21:05:50.995164 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:50.995145    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtnz9\" (UniqueName: \"kubernetes.io/projected/842f26f9-418e-4106-807c-ba828d837f0a-kube-api-access-wtnz9\") pod \"authorino-7bcfd8f9cd-jdw6r\" (UID: \"842f26f9-418e-4106-807c-ba828d837f0a\") " pod="kuadrant-system/authorino-7bcfd8f9cd-jdw6r"
Sep 10 21:05:50.998842 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:50.998817    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-n4h5b" event={"ID":"4c3ebeca-4c69-4c1a-8f5e-6939411484a0","Type":"ContainerStarted","Data":"aa610b5e7e07cf2dcd56eb32b8155654355d2731e149f6eeeed358528af242a2"}
Sep 10 21:05:51.106308 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:51.106278    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-jdw6r"
Sep 10 21:05:51.376543 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:51.376517    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jdw6r"]
Sep 10 21:05:51.378606 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:05:51.378581    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod842f26f9_418e_4106_807c_ba828d837f0a.slice/crio-295b39288ab57e66f5c1a0e73a2fc08d4ea0b0b603436a049e742a74ccdb9fc9 WatchSource:0}: Error finding container 295b39288ab57e66f5c1a0e73a2fc08d4ea0b0b603436a049e742a74ccdb9fc9: Status 404 returned error can't find the container with id 295b39288ab57e66f5c1a0e73a2fc08d4ea0b0b603436a049e742a74ccdb9fc9
Sep 10 21:05:52.006625 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:52.006594    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-jdw6r" event={"ID":"842f26f9-418e-4106-807c-ba828d837f0a","Type":"ContainerStarted","Data":"295b39288ab57e66f5c1a0e73a2fc08d4ea0b0b603436a049e742a74ccdb9fc9"}
Sep 10 21:05:53.215046 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:53.215016    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jdw6r"]
Sep 10 21:05:55.021423 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:55.021380    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-jdw6r" event={"ID":"842f26f9-418e-4106-807c-ba828d837f0a","Type":"ContainerStarted","Data":"050834d20b01e26a63d67c08b45605a324e1bd0a74c3953c08ce370068e8448d"}
Sep 10 21:05:55.021931 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:55.021441    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-jdw6r" podUID="842f26f9-418e-4106-807c-ba828d837f0a" containerName="authorino" containerID="cri-o://050834d20b01e26a63d67c08b45605a324e1bd0a74c3953c08ce370068e8448d" gracePeriod=30
Sep 10 21:05:55.023528 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:55.023497    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-n4h5b" event={"ID":"4c3ebeca-4c69-4c1a-8f5e-6939411484a0","Type":"ContainerStarted","Data":"8d11470d52ef74c94893f9289963fa0f7a68fdd0c435daad5f08cb2d80d365d2"}
Sep 10 21:05:55.023649 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:55.023560    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"
Sep 10 21:05:55.042204 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:55.042157    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-jdw6r" podStartSLOduration=2.440619653 podStartE2EDuration="5.042120989s" totalImagesPullingTime="2.601501336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:05:55.041256542 +0000 UTC m=+540.253721461" watchObservedRunningTime="2026-09-10 21:05:55.042120989 +0000 UTC m=+540.254585909"
Sep 10 21:05:55.060670 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:55.060627    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-n4h5b" podStartSLOduration=2.031403663 podStartE2EDuration="5.060613282s" totalImagesPullingTime="3.029209619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:05:55.059335664 +0000 UTC m=+540.271800584" watchObservedRunningTime="2026-09-10 21:05:55.060613282 +0000 UTC m=+540.273078201"
Sep 10 21:05:55.258312 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:55.258289    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-jdw6r"
Sep 10 21:05:55.322054 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:55.322027    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtnz9\" (UniqueName: \"kubernetes.io/projected/842f26f9-418e-4106-807c-ba828d837f0a-kube-api-access-wtnz9\") pod \"842f26f9-418e-4106-807c-ba828d837f0a\" (UID: \"842f26f9-418e-4106-807c-ba828d837f0a\") "
Sep 10 21:05:55.324228 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:55.324199    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842f26f9-418e-4106-807c-ba828d837f0a-kube-api-access-wtnz9" (OuterVolumeSpecName: "kube-api-access-wtnz9") pod "842f26f9-418e-4106-807c-ba828d837f0a" (UID: "842f26f9-418e-4106-807c-ba828d837f0a"). InnerVolumeSpecName "kube-api-access-wtnz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:05:55.423395 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:55.423371    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtnz9\" (UniqueName: \"kubernetes.io/projected/842f26f9-418e-4106-807c-ba828d837f0a-kube-api-access-wtnz9\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:05:56.030219 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:56.030188    2572 generic.go:356] "Generic (PLEG): container finished" podID="842f26f9-418e-4106-807c-ba828d837f0a" containerID="050834d20b01e26a63d67c08b45605a324e1bd0a74c3953c08ce370068e8448d" exitCode=0
Sep 10 21:05:56.030612 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:56.030237    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-jdw6r"
Sep 10 21:05:56.030612 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:56.030273    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-jdw6r" event={"ID":"842f26f9-418e-4106-807c-ba828d837f0a","Type":"ContainerDied","Data":"050834d20b01e26a63d67c08b45605a324e1bd0a74c3953c08ce370068e8448d"}
Sep 10 21:05:56.030612 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:56.030308    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-jdw6r" event={"ID":"842f26f9-418e-4106-807c-ba828d837f0a","Type":"ContainerDied","Data":"295b39288ab57e66f5c1a0e73a2fc08d4ea0b0b603436a049e742a74ccdb9fc9"}
Sep 10 21:05:56.030612 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:56.030325    2572 scope.go:117] "RemoveContainer" containerID="050834d20b01e26a63d67c08b45605a324e1bd0a74c3953c08ce370068e8448d"
Sep 10 21:05:56.039435 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:56.039415    2572 scope.go:117] "RemoveContainer" containerID="050834d20b01e26a63d67c08b45605a324e1bd0a74c3953c08ce370068e8448d"
Sep 10 21:05:56.039789 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:05:56.039749    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"050834d20b01e26a63d67c08b45605a324e1bd0a74c3953c08ce370068e8448d\": container with ID starting with 050834d20b01e26a63d67c08b45605a324e1bd0a74c3953c08ce370068e8448d not found: ID does not exist" containerID="050834d20b01e26a63d67c08b45605a324e1bd0a74c3953c08ce370068e8448d"
Sep 10 21:05:56.039789 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:56.039786    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050834d20b01e26a63d67c08b45605a324e1bd0a74c3953c08ce370068e8448d"} err="failed to get container status \"050834d20b01e26a63d67c08b45605a324e1bd0a74c3953c08ce370068e8448d\": rpc error: code = NotFound desc = could not find container \"050834d20b01e26a63d67c08b45605a324e1bd0a74c3953c08ce370068e8448d\": container with ID starting with 050834d20b01e26a63d67c08b45605a324e1bd0a74c3953c08ce370068e8448d not found: ID does not exist"
Sep 10 21:05:56.049707 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:56.049681    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jdw6r"]
Sep 10 21:05:56.055141 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:56.055116    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-jdw6r"]
Sep 10 21:05:57.438221 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:05:57.438187    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="842f26f9-418e-4106-807c-ba828d837f0a" path="/var/lib/kubelet/pods/842f26f9-418e-4106-807c-ba828d837f0a/volumes"
Sep 10 21:06:06.031948 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:06.031917    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"
Sep 10 21:06:24.134431 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:24.134393    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-747bd7fb67-22frf"]
Sep 10 21:06:24.134830 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:24.134744    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="842f26f9-418e-4106-807c-ba828d837f0a" containerName="authorino"
Sep 10 21:06:24.134830 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:24.134755    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="842f26f9-418e-4106-807c-ba828d837f0a" containerName="authorino"
Sep 10 21:06:24.134830 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:24.134814    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="842f26f9-418e-4106-807c-ba828d837f0a" containerName="authorino"
Sep 10 21:06:24.136784 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:24.136764    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-747bd7fb67-22frf"
Sep 10 21:06:24.139711 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:24.139687    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cqhw2\""
Sep 10 21:06:24.147423 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:24.147393    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-747bd7fb67-22frf"]
Sep 10 21:06:24.176607 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:24.176578    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-747bd7fb67-22frf"]
Sep 10 21:06:24.176813 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:06:24.176793    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fgn2f], unattached volumes=[], failed to process volumes=[kube-api-access-fgn2f]: context canceled" pod="kuadrant-system/authorino-747bd7fb67-22frf" podUID="c16771c1-2e74-4058-8272-8c4b80bb2734"
Sep 10 21:06:24.210862 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:24.210828    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b76444df-hvh8v"]
Sep 10 21:06:24.214536 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:24.214517    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b76444df-hvh8v"
Sep 10 21:06:24.217972 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:24.217951    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 21:06:24.221427 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:24.221407    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b76444df-hvh8v"]
Sep 10 21:06:24.228478 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:24.228449    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgn2f\" (UniqueName: \"kubernetes.io/projected/c16771c1-2e74-4058-8272-8c4b80bb2734-kube-api-access-fgn2f\") pod \"authorino-747bd7fb67-22frf\" (UID: \"c16771c1-2e74-4058-8272-8c4b80bb2734\") " pod="kuadrant-system/authorino-747bd7fb67-22frf"
Sep 10 21:06:24.329692 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:24.329667    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgn2f\" (UniqueName: \"kubernetes.io/projected/c16771c1-2e74-4058-8272-8c4b80bb2734-kube-api-access-fgn2f\") pod \"authorino-747bd7fb67-22frf\" (UID: \"c16771c1-2e74-4058-8272-8c4b80bb2734\") " pod="kuadrant-system/authorino-747bd7fb67-22frf"
Sep 10 21:06:24.329777 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:24.329697    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c-tls-cert\") pod \"authorino-7b76444df-hvh8v\" (UID: \"a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c\") " pod="kuadrant-system/authorino-7b76444df-hvh8v"
Sep 10 21:06:24.329777 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:24.329726    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq28z\" (UniqueName: \"kubernetes.io/projected/a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c-kube-api-access-bq28z\") pod \"authorino-7b76444df-hvh8v\" (UID: \"a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c\") " pod="kuadrant-system/authorino-7b76444df-hvh8v"
Sep 10 21:06:24.338169 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:24.338147    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgn2f\" (UniqueName: \"kubernetes.io/projected/c16771c1-2e74-4058-8272-8c4b80bb2734-kube-api-access-fgn2f\") pod \"authorino-747bd7fb67-22frf\" (UID: \"c16771c1-2e74-4058-8272-8c4b80bb2734\") " pod="kuadrant-system/authorino-747bd7fb67-22frf"
Sep 10 21:06:24.430215 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:24.430164    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c-tls-cert\") pod \"authorino-7b76444df-hvh8v\" (UID: \"a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c\") " pod="kuadrant-system/authorino-7b76444df-hvh8v"
Sep 10 21:06:24.430215 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:24.430201    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq28z\" (UniqueName: \"kubernetes.io/projected/a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c-kube-api-access-bq28z\") pod \"authorino-7b76444df-hvh8v\" (UID: \"a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c\") " pod="kuadrant-system/authorino-7b76444df-hvh8v"
Sep 10 21:06:24.432597 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:24.432573    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c-tls-cert\") pod \"authorino-7b76444df-hvh8v\" (UID: \"a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c\") " pod="kuadrant-system/authorino-7b76444df-hvh8v"
Sep 10 21:06:24.441575 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:24.441551    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq28z\" (UniqueName: \"kubernetes.io/projected/a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c-kube-api-access-bq28z\") pod \"authorino-7b76444df-hvh8v\" (UID: \"a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c\") " pod="kuadrant-system/authorino-7b76444df-hvh8v"
Sep 10 21:06:24.526272 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:24.526250    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b76444df-hvh8v"
Sep 10 21:06:24.795022 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:24.795000    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b76444df-hvh8v"]
Sep 10 21:06:24.797671 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:06:24.797643    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2787dd2_a0bb_426b_b1d8_f4a5a11dae8c.slice/crio-acc4bb7857bc0bc560d159c14708e4b397529274d328d92cd38fc7d7fa9bd270 WatchSource:0}: Error finding container acc4bb7857bc0bc560d159c14708e4b397529274d328d92cd38fc7d7fa9bd270: Status 404 returned error can't find the container with id acc4bb7857bc0bc560d159c14708e4b397529274d328d92cd38fc7d7fa9bd270
Sep 10 21:06:25.167940 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:25.167915    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b76444df-hvh8v" event={"ID":"a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c","Type":"ContainerStarted","Data":"ebb057d9ad6eda0a00b2e6277587bff37f6e5b42eb95c96596ae2b0482a26ef1"}
Sep 10 21:06:25.168312 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:25.167962    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b76444df-hvh8v" event={"ID":"a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c","Type":"ContainerStarted","Data":"acc4bb7857bc0bc560d159c14708e4b397529274d328d92cd38fc7d7fa9bd270"}
Sep 10 21:06:25.168312 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:25.167973    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-747bd7fb67-22frf"
Sep 10 21:06:25.173012 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:25.172974    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-747bd7fb67-22frf"
Sep 10 21:06:25.185974 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:25.185926    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7b76444df-hvh8v" podStartSLOduration=0.874552099 podStartE2EDuration="1.185912992s" totalImagesPullingTime="311.360893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:06:25.184723362 +0000 UTC m=+570.397188282" watchObservedRunningTime="2026-09-10 21:06:25.185912992 +0000 UTC m=+570.398377911"
Sep 10 21:06:25.238424 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:25.238380    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgn2f\" (UniqueName: \"kubernetes.io/projected/c16771c1-2e74-4058-8272-8c4b80bb2734-kube-api-access-fgn2f\") pod \"c16771c1-2e74-4058-8272-8c4b80bb2734\" (UID: \"c16771c1-2e74-4058-8272-8c4b80bb2734\") "
Sep 10 21:06:25.241387 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:25.241324    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c16771c1-2e74-4058-8272-8c4b80bb2734-kube-api-access-fgn2f" (OuterVolumeSpecName: "kube-api-access-fgn2f") pod "c16771c1-2e74-4058-8272-8c4b80bb2734" (UID: "c16771c1-2e74-4058-8272-8c4b80bb2734"). InnerVolumeSpecName "kube-api-access-fgn2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:06:25.339429 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:25.339399    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgn2f\" (UniqueName: \"kubernetes.io/projected/c16771c1-2e74-4058-8272-8c4b80bb2734-kube-api-access-fgn2f\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:06:26.171903 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:26.171872    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-747bd7fb67-22frf"
Sep 10 21:06:26.205475 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:26.205448    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-747bd7fb67-22frf"]
Sep 10 21:06:26.210137 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:26.210114    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-747bd7fb67-22frf"]
Sep 10 21:06:27.437265 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:06:27.437232    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c16771c1-2e74-4058-8272-8c4b80bb2734" path="/var/lib/kubelet/pods/c16771c1-2e74-4058-8272-8c4b80bb2734/volumes"
Sep 10 21:07:09.277678 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:07:09.277646    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:07:25.992780 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:07:25.992699    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:07:43.531361 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:07:43.531325    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:08:14.067160 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:08:14.067127    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:08:25.230347 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:08:25.230314    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:08:38.929534 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:08:38.929496    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:08:59.846645 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:08:59.846570    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:09:05.011001 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.010942    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"]
Sep 10 21:09:05.014620 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.014594    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"
Sep 10 21:09:05.018128 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.018100    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 21:09:05.018255 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.018195    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 21:09:05.018328 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.018202    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-822jm\""
Sep 10 21:09:05.018328 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.018276    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 21:09:05.025026 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.024968    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"]
Sep 10 21:09:05.130044 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.130020    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad3cef32-8927-4b38-b735-3e48eefc2955-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn\" (UID: \"ad3cef32-8927-4b38-b735-3e48eefc2955\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"
Sep 10 21:09:05.130160 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.130053    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad3cef32-8927-4b38-b735-3e48eefc2955-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn\" (UID: \"ad3cef32-8927-4b38-b735-3e48eefc2955\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"
Sep 10 21:09:05.130160 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.130080    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad3cef32-8927-4b38-b735-3e48eefc2955-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn\" (UID: \"ad3cef32-8927-4b38-b735-3e48eefc2955\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"
Sep 10 21:09:05.130274 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.130215    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad3cef32-8927-4b38-b735-3e48eefc2955-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn\" (UID: \"ad3cef32-8927-4b38-b735-3e48eefc2955\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"
Sep 10 21:09:05.130274 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.130255    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbssk\" (UniqueName: \"kubernetes.io/projected/ad3cef32-8927-4b38-b735-3e48eefc2955-kube-api-access-cbssk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn\" (UID: \"ad3cef32-8927-4b38-b735-3e48eefc2955\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"
Sep 10 21:09:05.130366 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.130290    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad3cef32-8927-4b38-b735-3e48eefc2955-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn\" (UID: \"ad3cef32-8927-4b38-b735-3e48eefc2955\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"
Sep 10 21:09:05.231050 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.231024    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad3cef32-8927-4b38-b735-3e48eefc2955-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn\" (UID: \"ad3cef32-8927-4b38-b735-3e48eefc2955\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"
Sep 10 21:09:05.231138 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.231055    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbssk\" (UniqueName: \"kubernetes.io/projected/ad3cef32-8927-4b38-b735-3e48eefc2955-kube-api-access-cbssk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn\" (UID: \"ad3cef32-8927-4b38-b735-3e48eefc2955\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"
Sep 10 21:09:05.231138 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.231077    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad3cef32-8927-4b38-b735-3e48eefc2955-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn\" (UID: \"ad3cef32-8927-4b38-b735-3e48eefc2955\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"
Sep 10 21:09:05.231239 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.231198    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad3cef32-8927-4b38-b735-3e48eefc2955-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn\" (UID: \"ad3cef32-8927-4b38-b735-3e48eefc2955\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"
Sep 10 21:09:05.231299 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.231253    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad3cef32-8927-4b38-b735-3e48eefc2955-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn\" (UID: \"ad3cef32-8927-4b38-b735-3e48eefc2955\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"
Sep 10 21:09:05.231352 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.231307    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad3cef32-8927-4b38-b735-3e48eefc2955-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn\" (UID: \"ad3cef32-8927-4b38-b735-3e48eefc2955\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"
Sep 10 21:09:05.231551 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.231531    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad3cef32-8927-4b38-b735-3e48eefc2955-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn\" (UID: \"ad3cef32-8927-4b38-b735-3e48eefc2955\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"
Sep 10 21:09:05.231621 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.231591    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad3cef32-8927-4b38-b735-3e48eefc2955-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn\" (UID: \"ad3cef32-8927-4b38-b735-3e48eefc2955\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"
Sep 10 21:09:05.231673 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.231624    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad3cef32-8927-4b38-b735-3e48eefc2955-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn\" (UID: \"ad3cef32-8927-4b38-b735-3e48eefc2955\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"
Sep 10 21:09:05.233595 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.233567    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad3cef32-8927-4b38-b735-3e48eefc2955-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn\" (UID: \"ad3cef32-8927-4b38-b735-3e48eefc2955\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"
Sep 10 21:09:05.233832 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.233809    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad3cef32-8927-4b38-b735-3e48eefc2955-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn\" (UID: \"ad3cef32-8927-4b38-b735-3e48eefc2955\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"
Sep 10 21:09:05.240075 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.240053    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbssk\" (UniqueName: \"kubernetes.io/projected/ad3cef32-8927-4b38-b735-3e48eefc2955-kube-api-access-cbssk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn\" (UID: \"ad3cef32-8927-4b38-b735-3e48eefc2955\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"
Sep 10 21:09:05.324657 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.324637    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"
Sep 10 21:09:05.600924 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.600826    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"]
Sep 10 21:09:05.606846 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:09:05.606807    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad3cef32_8927_4b38_b735_3e48eefc2955.slice/crio-c8ec383f347fdb3d5a4a14d54216a4a78a37b58b832944051c20c97c77fc9156 WatchSource:0}: Error finding container c8ec383f347fdb3d5a4a14d54216a4a78a37b58b832944051c20c97c77fc9156: Status 404 returned error can't find the container with id c8ec383f347fdb3d5a4a14d54216a4a78a37b58b832944051c20c97c77fc9156
Sep 10 21:09:05.608309 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.608286    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:09:05.916754 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:05.916671    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn" event={"ID":"ad3cef32-8927-4b38-b735-3e48eefc2955","Type":"ContainerStarted","Data":"c8ec383f347fdb3d5a4a14d54216a4a78a37b58b832944051c20c97c77fc9156"}
Sep 10 21:09:06.922789 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:06.922711    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn" event={"ID":"ad3cef32-8927-4b38-b735-3e48eefc2955","Type":"ContainerStarted","Data":"7031aff238969cd0bea3de3be8fd4a20a60c9963762cd57c43d14912cc9697b8"}
Sep 10 21:09:06.956037 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:06.955969    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn" podStartSLOduration=1.930655528 podStartE2EDuration="2.955953405s" totalImagesPullingTime="1.025297877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:09:06.954397301 +0000 UTC m=+732.166862219" watchObservedRunningTime="2026-09-10 21:09:06.955953405 +0000 UTC m=+732.168418325"
Sep 10 21:09:08.024248 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:08.024202    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:09:15.325301 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:15.325262    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"
Sep 10 21:09:15.325301 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:15.325307    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"
Sep 10 21:09:15.337968 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:15.337938    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"
Sep 10 21:09:15.975839 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:09:15.975798    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn"
Sep 10 21:10:28.388805 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:28.388716    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5fd9f86dc6-2vsr9"]
Sep 10 21:10:28.391735 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:28.391702    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fd9f86dc6-2vsr9"
Sep 10 21:10:28.402100 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:28.402073    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fd9f86dc6-2vsr9"]
Sep 10 21:10:28.443538 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:28.443506    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pbmd\" (UniqueName: \"kubernetes.io/projected/5f9fd438-9816-4e56-8cec-eb342ed06382-kube-api-access-6pbmd\") pod \"authorino-5fd9f86dc6-2vsr9\" (UID: \"5f9fd438-9816-4e56-8cec-eb342ed06382\") " pod="kuadrant-system/authorino-5fd9f86dc6-2vsr9"
Sep 10 21:10:28.443653 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:28.443557    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5f9fd438-9816-4e56-8cec-eb342ed06382-tls-cert\") pod \"authorino-5fd9f86dc6-2vsr9\" (UID: \"5f9fd438-9816-4e56-8cec-eb342ed06382\") " pod="kuadrant-system/authorino-5fd9f86dc6-2vsr9"
Sep 10 21:10:28.544875 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:28.544848    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pbmd\" (UniqueName: \"kubernetes.io/projected/5f9fd438-9816-4e56-8cec-eb342ed06382-kube-api-access-6pbmd\") pod \"authorino-5fd9f86dc6-2vsr9\" (UID: \"5f9fd438-9816-4e56-8cec-eb342ed06382\") " pod="kuadrant-system/authorino-5fd9f86dc6-2vsr9"
Sep 10 21:10:28.545009 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:28.544891    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5f9fd438-9816-4e56-8cec-eb342ed06382-tls-cert\") pod \"authorino-5fd9f86dc6-2vsr9\" (UID: \"5f9fd438-9816-4e56-8cec-eb342ed06382\") " pod="kuadrant-system/authorino-5fd9f86dc6-2vsr9"
Sep 10 21:10:28.547424 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:28.547402    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5f9fd438-9816-4e56-8cec-eb342ed06382-tls-cert\") pod \"authorino-5fd9f86dc6-2vsr9\" (UID: \"5f9fd438-9816-4e56-8cec-eb342ed06382\") " pod="kuadrant-system/authorino-5fd9f86dc6-2vsr9"
Sep 10 21:10:28.553788 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:28.553748    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pbmd\" (UniqueName: \"kubernetes.io/projected/5f9fd438-9816-4e56-8cec-eb342ed06382-kube-api-access-6pbmd\") pod \"authorino-5fd9f86dc6-2vsr9\" (UID: \"5f9fd438-9816-4e56-8cec-eb342ed06382\") " pod="kuadrant-system/authorino-5fd9f86dc6-2vsr9"
Sep 10 21:10:28.702961 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:28.702902    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fd9f86dc6-2vsr9"
Sep 10 21:10:28.975041 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:28.975002    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fd9f86dc6-2vsr9"]
Sep 10 21:10:28.977392 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:10:28.977369    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f9fd438_9816_4e56_8cec_eb342ed06382.slice/crio-f40845070c68362f80153bc74462948741584f43aa2d8c5a3fd45f9349f53f3b WatchSource:0}: Error finding container f40845070c68362f80153bc74462948741584f43aa2d8c5a3fd45f9349f53f3b: Status 404 returned error can't find the container with id f40845070c68362f80153bc74462948741584f43aa2d8c5a3fd45f9349f53f3b
Sep 10 21:10:29.320608 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:29.320574    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fd9f86dc6-2vsr9" event={"ID":"5f9fd438-9816-4e56-8cec-eb342ed06382","Type":"ContainerStarted","Data":"f40845070c68362f80153bc74462948741584f43aa2d8c5a3fd45f9349f53f3b"}
Sep 10 21:10:30.327830 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:30.327795    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fd9f86dc6-2vsr9" event={"ID":"5f9fd438-9816-4e56-8cec-eb342ed06382","Type":"ContainerStarted","Data":"ac7762d51a4d2f70dee8c82a6341b6779509e5f612e233f47f5f779517548b6a"}
Sep 10 21:10:30.345280 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:30.345215    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5fd9f86dc6-2vsr9" podStartSLOduration=1.220554602 podStartE2EDuration="2.34519346s" totalImagesPullingTime="1.124638858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:10:30.343228114 +0000 UTC m=+815.555693037" watchObservedRunningTime="2026-09-10 21:10:30.34519346 +0000 UTC m=+815.557658379"
Sep 10 21:10:30.376238 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:30.376150    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b76444df-hvh8v"]
Sep 10 21:10:30.376391 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:30.376327    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7b76444df-hvh8v" podUID="a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c" containerName="authorino" containerID="cri-o://ebb057d9ad6eda0a00b2e6277587bff37f6e5b42eb95c96596ae2b0482a26ef1" gracePeriod=30
Sep 10 21:10:30.630536 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:30.630470    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b76444df-hvh8v"
Sep 10 21:10:30.662826 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:30.662798    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c-tls-cert\") pod \"a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c\" (UID: \"a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c\") "
Sep 10 21:10:30.663004 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:30.662868    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq28z\" (UniqueName: \"kubernetes.io/projected/a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c-kube-api-access-bq28z\") pod \"a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c\" (UID: \"a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c\") "
Sep 10 21:10:30.665123 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:30.665097    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c-kube-api-access-bq28z" (OuterVolumeSpecName: "kube-api-access-bq28z") pod "a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c" (UID: "a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c"). InnerVolumeSpecName "kube-api-access-bq28z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:10:30.673744 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:30.673718    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c" (UID: "a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:10:30.764368 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:30.764333    2572 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c-tls-cert\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:10:30.764368 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:30.764366    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq28z\" (UniqueName: \"kubernetes.io/projected/a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c-kube-api-access-bq28z\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:10:31.334493 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:31.334463    2572 generic.go:356] "Generic (PLEG): container finished" podID="a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c" containerID="ebb057d9ad6eda0a00b2e6277587bff37f6e5b42eb95c96596ae2b0482a26ef1" exitCode=0
Sep 10 21:10:31.334932 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:31.334517    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b76444df-hvh8v"
Sep 10 21:10:31.334932 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:31.334587    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b76444df-hvh8v" event={"ID":"a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c","Type":"ContainerDied","Data":"ebb057d9ad6eda0a00b2e6277587bff37f6e5b42eb95c96596ae2b0482a26ef1"}
Sep 10 21:10:31.334932 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:31.334632    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b76444df-hvh8v" event={"ID":"a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c","Type":"ContainerDied","Data":"acc4bb7857bc0bc560d159c14708e4b397529274d328d92cd38fc7d7fa9bd270"}
Sep 10 21:10:31.334932 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:31.334664    2572 scope.go:117] "RemoveContainer" containerID="ebb057d9ad6eda0a00b2e6277587bff37f6e5b42eb95c96596ae2b0482a26ef1"
Sep 10 21:10:31.344541 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:31.344525    2572 scope.go:117] "RemoveContainer" containerID="ebb057d9ad6eda0a00b2e6277587bff37f6e5b42eb95c96596ae2b0482a26ef1"
Sep 10 21:10:31.344781 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:10:31.344762    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebb057d9ad6eda0a00b2e6277587bff37f6e5b42eb95c96596ae2b0482a26ef1\": container with ID starting with ebb057d9ad6eda0a00b2e6277587bff37f6e5b42eb95c96596ae2b0482a26ef1 not found: ID does not exist" containerID="ebb057d9ad6eda0a00b2e6277587bff37f6e5b42eb95c96596ae2b0482a26ef1"
Sep 10 21:10:31.344826 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:31.344789    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebb057d9ad6eda0a00b2e6277587bff37f6e5b42eb95c96596ae2b0482a26ef1"} err="failed to get container status \"ebb057d9ad6eda0a00b2e6277587bff37f6e5b42eb95c96596ae2b0482a26ef1\": rpc error: code = NotFound desc = could not find container \"ebb057d9ad6eda0a00b2e6277587bff37f6e5b42eb95c96596ae2b0482a26ef1\": container with ID starting with ebb057d9ad6eda0a00b2e6277587bff37f6e5b42eb95c96596ae2b0482a26ef1 not found: ID does not exist"
Sep 10 21:10:31.355906 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:31.355886    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b76444df-hvh8v"]
Sep 10 21:10:31.363111 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:31.363055    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7b76444df-hvh8v"]
Sep 10 21:10:31.437937 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:31.437904    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c" path="/var/lib/kubelet/pods/a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c/volumes"
Sep 10 21:10:48.780493 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:48.780458    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"]
Sep 10 21:10:48.780971 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:48.780952    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c" containerName="authorino"
Sep 10 21:10:48.781069 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:48.780975    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c" containerName="authorino"
Sep 10 21:10:48.781110 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:48.781086    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2787dd2-a0bb-426b-b1d8-f4a5a11dae8c" containerName="authorino"
Sep 10 21:10:48.784960 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:48.784941    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:48.787324 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:48.787305    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 21:10:48.788197 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:48.788174    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-04db70-openshift-default-dockercfg-rsck9\""
Sep 10 21:10:48.799035 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:48.799013    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"]
Sep 10 21:10:48.910199 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:48.910168    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istio-envoy\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:48.910199 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:48.910205    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s84h\" (UniqueName: \"kubernetes.io/projected/e35a47b7-668d-4c95-b2b8-446d5e762a8c-kube-api-access-6s84h\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:48.910487 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:48.910225    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-workload-socket\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:48.910487 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:48.910344    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-credential-socket\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:48.910487 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:48.910384    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istio-token\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:48.910487 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:48.910411    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istio-podinfo\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:48.910487 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:48.910471    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istio-data\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:48.910717 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:48.910493    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istiod-ca-cert\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:48.910717 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:48.910529    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-workload-certs\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:49.011260 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:49.011229    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istio-data\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:49.011260 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:49.011259    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istiod-ca-cert\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:49.011476 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:49.011282    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-workload-certs\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:49.011476 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:49.011311    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istio-envoy\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:49.011476 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:49.011428    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s84h\" (UniqueName: \"kubernetes.io/projected/e35a47b7-668d-4c95-b2b8-446d5e762a8c-kube-api-access-6s84h\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:49.011476 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:49.011472    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-workload-socket\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:49.011680 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:49.011525    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-credential-socket\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:49.011680 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:49.011555    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istio-token\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:49.011680 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:49.011580    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istio-podinfo\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:49.011680 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:49.011625    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istio-data\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:49.011875 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:49.011697    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-workload-certs\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:49.011875 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:49.011835    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-workload-socket\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:49.012015 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:49.011873    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-credential-socket\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:49.012295 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:49.012273    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istiod-ca-cert\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:49.013702 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:49.013676    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istio-envoy\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:49.014037 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:49.014020    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istio-podinfo\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:49.020069 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:49.020045    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istio-token\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:49.020480 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:49.020456    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s84h\" (UniqueName: \"kubernetes.io/projected/e35a47b7-668d-4c95-b2b8-446d5e762a8c-kube-api-access-6s84h\") pod \"e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") " pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:49.096036 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:49.095957    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:49.373715 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:49.373689    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"]
Sep 10 21:10:49.376355 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:10:49.376327    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode35a47b7_668d_4c95_b2b8_446d5e762a8c.slice/crio-5ea9f16c5fefdd6e3e199277ce56948cdf0bce42a22be5deb6e75e0232c07f49 WatchSource:0}: Error finding container 5ea9f16c5fefdd6e3e199277ce56948cdf0bce42a22be5deb6e75e0232c07f49: Status 404 returned error can't find the container with id 5ea9f16c5fefdd6e3e199277ce56948cdf0bce42a22be5deb6e75e0232c07f49
Sep 10 21:10:49.428526 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:49.428496    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk" event={"ID":"e35a47b7-668d-4c95-b2b8-446d5e762a8c","Type":"ContainerStarted","Data":"5ea9f16c5fefdd6e3e199277ce56948cdf0bce42a22be5deb6e75e0232c07f49"}
Sep 10 21:10:50.396442 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:50.396409    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7918-74d908"]
Sep 10 21:10:50.400965 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:50.400949    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7918-74d908"
Sep 10 21:10:50.403244 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:50.403225    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zd8vg\""
Sep 10 21:10:50.406534 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:50.406515    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7918-74d908"]
Sep 10 21:10:50.526242 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:50.526211    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lhl4\" (UniqueName: \"kubernetes.io/projected/a92a3dbc-e05b-4346-b9a0-618e1c06de90-kube-api-access-4lhl4\") pod \"test-curl-7918-74d908\" (UID: \"a92a3dbc-e05b-4346-b9a0-618e1c06de90\") " pod="opendatahub/test-curl-7918-74d908"
Sep 10 21:10:50.627060 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:50.627023    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lhl4\" (UniqueName: \"kubernetes.io/projected/a92a3dbc-e05b-4346-b9a0-618e1c06de90-kube-api-access-4lhl4\") pod \"test-curl-7918-74d908\" (UID: \"a92a3dbc-e05b-4346-b9a0-618e1c06de90\") " pod="opendatahub/test-curl-7918-74d908"
Sep 10 21:10:50.637090 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:50.637063    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lhl4\" (UniqueName: \"kubernetes.io/projected/a92a3dbc-e05b-4346-b9a0-618e1c06de90-kube-api-access-4lhl4\") pod \"test-curl-7918-74d908\" (UID: \"a92a3dbc-e05b-4346-b9a0-618e1c06de90\") " pod="opendatahub/test-curl-7918-74d908"
Sep 10 21:10:50.711752 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:50.711686    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7918-74d908"
Sep 10 21:10:50.990792 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:50.990724    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7918-74d908"]
Sep 10 21:10:50.992063 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:10:50.992030    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda92a3dbc_e05b_4346_b9a0_618e1c06de90.slice/crio-a2c30d00912b85d7a5180e1762ed834bebd573f830d0a3fcc85ef2f9da0aa1bf WatchSource:0}: Error finding container a2c30d00912b85d7a5180e1762ed834bebd573f830d0a3fcc85ef2f9da0aa1bf: Status 404 returned error can't find the container with id a2c30d00912b85d7a5180e1762ed834bebd573f830d0a3fcc85ef2f9da0aa1bf
Sep 10 21:10:51.439521 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:51.439485    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7918-74d908" event={"ID":"a92a3dbc-e05b-4346-b9a0-618e1c06de90","Type":"ContainerStarted","Data":"a2c30d00912b85d7a5180e1762ed834bebd573f830d0a3fcc85ef2f9da0aa1bf"}
Sep 10 21:10:52.139911 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:52.139866    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:10:52.381862 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:52.381824    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:10:52.381949 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:52.381909    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:10:52.381949 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:52.381938    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:10:53.451518 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:53.451480    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk" event={"ID":"e35a47b7-668d-4c95-b2b8-446d5e762a8c","Type":"ContainerStarted","Data":"c42e42d0caa02bcdb3dd7789d1c054c14d0682df50abd4781b74dc535fd78b7e"}
Sep 10 21:10:53.474466 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:53.474410    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk" podStartSLOduration=2.47128075 podStartE2EDuration="5.474394347s" totalImagesPullingTime="3.003113597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:10:53.473239529 +0000 UTC m=+838.685704491" watchObservedRunningTime="2026-09-10 21:10:53.474394347 +0000 UTC m=+838.686859266"
Sep 10 21:10:54.097150 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:54.097125    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:54.101880 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:54.101851    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:54.457207 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:54.457142    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7918-74d908" event={"ID":"a92a3dbc-e05b-4346-b9a0-618e1c06de90","Type":"ContainerStarted","Data":"3f6a161f6d1df635fe219ba2b3dcc80560807c1a3840e66de7e2a73a3bf026c1"}
Sep 10 21:10:54.457519 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:54.457498    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:54.458399 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:54.458380    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:10:54.477884 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:54.477836    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7918-74d908" podStartSLOduration=1.711912541 podStartE2EDuration="4.477823568s" totalImagesPullingTime="2.765911027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:10:54.476270547 +0000 UTC m=+839.688735466" watchObservedRunningTime="2026-09-10 21:10:54.477823568 +0000 UTC m=+839.690288486"
Sep 10 21:10:55.102478 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:55.102445    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7918-74d908"]
Sep 10 21:10:55.106380 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:55.106352    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7918-74d908"]
Sep 10 21:10:55.227471 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:55.227414    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7918-efb80a"]
Sep 10 21:10:55.227751 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:55.227739    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a92a3dbc-e05b-4346-b9a0-618e1c06de90" containerName="test-curl-7918-74d908"
Sep 10 21:10:55.227791 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:55.227752    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a92a3dbc-e05b-4346-b9a0-618e1c06de90" containerName="test-curl-7918-74d908"
Sep 10 21:10:55.227827 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:55.227805    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a92a3dbc-e05b-4346-b9a0-618e1c06de90" containerName="test-curl-7918-74d908"
Sep 10 21:10:55.230761 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:55.230748    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7918-efb80a"
Sep 10 21:10:55.238896 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:55.238861    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7918-efb80a"]
Sep 10 21:10:55.370692 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:55.370666    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6ppc\" (UniqueName: \"kubernetes.io/projected/86c0b036-8278-4722-ae75-274f616c6a89-kube-api-access-d6ppc\") pod \"test-curl-7918-efb80a\" (UID: \"86c0b036-8278-4722-ae75-274f616c6a89\") " pod="opendatahub/test-curl-7918-efb80a"
Sep 10 21:10:55.461254 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:10:55.461229    2572 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7918-74d908" container="test-curl-7918-74d908"
Sep 10 21:10:55.471068 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:55.471043    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6ppc\" (UniqueName: \"kubernetes.io/projected/86c0b036-8278-4722-ae75-274f616c6a89-kube-api-access-d6ppc\") pod \"test-curl-7918-efb80a\" (UID: \"86c0b036-8278-4722-ae75-274f616c6a89\") " pod="opendatahub/test-curl-7918-efb80a"
Sep 10 21:10:55.487692 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:55.487675    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6ppc\" (UniqueName: \"kubernetes.io/projected/86c0b036-8278-4722-ae75-274f616c6a89-kube-api-access-d6ppc\") pod \"test-curl-7918-efb80a\" (UID: \"86c0b036-8278-4722-ae75-274f616c6a89\") " pod="opendatahub/test-curl-7918-efb80a"
Sep 10 21:10:55.539611 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:55.539593    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7918-efb80a"
Sep 10 21:10:55.823082 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:55.823058    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7918-efb80a"]
Sep 10 21:10:55.824888 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:10:55.824863    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86c0b036_8278_4722_ae75_274f616c6a89.slice/crio-0da1512ea6c01ab528119f927503198978e33d3e137f8e4186c0795da2786087 WatchSource:0}: Error finding container 0da1512ea6c01ab528119f927503198978e33d3e137f8e4186c0795da2786087: Status 404 returned error can't find the container with id 0da1512ea6c01ab528119f927503198978e33d3e137f8e4186c0795da2786087
Sep 10 21:10:56.468713 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:56.468669    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7918-efb80a" event={"ID":"86c0b036-8278-4722-ae75-274f616c6a89","Type":"ContainerStarted","Data":"3add59f59022962ebaa055c5f37d1f30b7f0b8f1d1ac3c0bd6ab167883cf80b9"}
Sep 10 21:10:56.469150 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:56.468724    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7918-efb80a" event={"ID":"86c0b036-8278-4722-ae75-274f616c6a89","Type":"ContainerStarted","Data":"0da1512ea6c01ab528119f927503198978e33d3e137f8e4186c0795da2786087"}
Sep 10 21:10:56.469150 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:56.469005    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7918-74d908" podUID="a92a3dbc-e05b-4346-b9a0-618e1c06de90" containerName="test-curl-7918-74d908" containerID="cri-o://3f6a161f6d1df635fe219ba2b3dcc80560807c1a3840e66de7e2a73a3bf026c1" gracePeriod=2
Sep 10 21:10:56.501543 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:56.501493    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7918-efb80a" podStartSLOduration=1.501454942 podStartE2EDuration="1.501454942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:10:56.499823907 +0000 UTC m=+841.712288825" watchObservedRunningTime="2026-09-10 21:10:56.501454942 +0000 UTC m=+841.713919955"
Sep 10 21:10:57.187227 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:57.187148    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7918-efb80a"]
Sep 10 21:10:57.192747 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:57.192713    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7918-efb80a"]
Sep 10 21:10:57.472756 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:10:57.472679    2572 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7918-efb80a" container="test-curl-7918-efb80a"
Sep 10 21:10:57.567573 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:57.567534    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw"]
Sep 10 21:10:57.567942 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:57.567916    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86c0b036-8278-4722-ae75-274f616c6a89" containerName="test-curl-7918-efb80a"
Sep 10 21:10:57.567942 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:57.567939    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="86c0b036-8278-4722-ae75-274f616c6a89" containerName="test-curl-7918-efb80a"
Sep 10 21:10:57.568153 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:57.568058    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="86c0b036-8278-4722-ae75-274f616c6a89" containerName="test-curl-7918-efb80a"
Sep 10 21:10:57.571458 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:57.571435    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw"
Sep 10 21:10:57.582427 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:57.582276    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-04db70\""
Sep 10 21:10:57.594060 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:57.594032    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw"]
Sep 10 21:10:57.686194 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:57.686159    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e80bd28-a8f4-4f72-a45f-9794ac6b3643-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw\" (UID: \"0e80bd28-a8f4-4f72-a45f-9794ac6b3643\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw"
Sep 10 21:10:57.686352 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:57.686248    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58jrv\" (UniqueName: \"kubernetes.io/projected/0e80bd28-a8f4-4f72-a45f-9794ac6b3643-kube-api-access-58jrv\") pod \"payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw\" (UID: \"0e80bd28-a8f4-4f72-a45f-9794ac6b3643\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw"
Sep 10 21:10:57.787301 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:57.787270    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58jrv\" (UniqueName: \"kubernetes.io/projected/0e80bd28-a8f4-4f72-a45f-9794ac6b3643-kube-api-access-58jrv\") pod \"payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw\" (UID: \"0e80bd28-a8f4-4f72-a45f-9794ac6b3643\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw"
Sep 10 21:10:57.787457 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:57.787373    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e80bd28-a8f4-4f72-a45f-9794ac6b3643-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw\" (UID: \"0e80bd28-a8f4-4f72-a45f-9794ac6b3643\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw"
Sep 10 21:10:57.788121 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:57.788102    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e80bd28-a8f4-4f72-a45f-9794ac6b3643-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw\" (UID: \"0e80bd28-a8f4-4f72-a45f-9794ac6b3643\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw"
Sep 10 21:10:57.802218 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:57.802193    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58jrv\" (UniqueName: \"kubernetes.io/projected/0e80bd28-a8f4-4f72-a45f-9794ac6b3643-kube-api-access-58jrv\") pod \"payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw\" (UID: \"0e80bd28-a8f4-4f72-a45f-9794ac6b3643\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw"
Sep 10 21:10:57.897027 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:57.896970    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw"
Sep 10 21:10:58.178068 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:58.177996    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw"]
Sep 10 21:10:58.180883 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:10:58.180858    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e80bd28_a8f4_4f72_a45f_9794ac6b3643.slice/crio-86de7ce3107c4bb472ce2e1a30c864b27b796131a204a7f8c923c1a9cbc49086 WatchSource:0}: Error finding container 86de7ce3107c4bb472ce2e1a30c864b27b796131a204a7f8c923c1a9cbc49086: Status 404 returned error can't find the container with id 86de7ce3107c4bb472ce2e1a30c864b27b796131a204a7f8c923c1a9cbc49086
Sep 10 21:10:58.479495 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:58.479465    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw" event={"ID":"0e80bd28-a8f4-4f72-a45f-9794ac6b3643","Type":"ContainerStarted","Data":"86de7ce3107c4bb472ce2e1a30c864b27b796131a204a7f8c923c1a9cbc49086"}
Sep 10 21:10:58.479495 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:58.479482    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7918-efb80a" podUID="86c0b036-8278-4722-ae75-274f616c6a89" containerName="test-curl-7918-efb80a" containerID="cri-o://3add59f59022962ebaa055c5f37d1f30b7f0b8f1d1ac3c0bd6ab167883cf80b9" gracePeriod=2
Sep 10 21:10:58.634864 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:58.634839    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7918-74d908"
Sep 10 21:10:58.695233 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:58.695211    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lhl4\" (UniqueName: \"kubernetes.io/projected/a92a3dbc-e05b-4346-b9a0-618e1c06de90-kube-api-access-4lhl4\") pod \"a92a3dbc-e05b-4346-b9a0-618e1c06de90\" (UID: \"a92a3dbc-e05b-4346-b9a0-618e1c06de90\") "
Sep 10 21:10:58.697389 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:58.697363    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a92a3dbc-e05b-4346-b9a0-618e1c06de90-kube-api-access-4lhl4" (OuterVolumeSpecName: "kube-api-access-4lhl4") pod "a92a3dbc-e05b-4346-b9a0-618e1c06de90" (UID: "a92a3dbc-e05b-4346-b9a0-618e1c06de90"). InnerVolumeSpecName "kube-api-access-4lhl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:10:58.796589 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:58.796567    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lhl4\" (UniqueName: \"kubernetes.io/projected/a92a3dbc-e05b-4346-b9a0-618e1c06de90-kube-api-access-4lhl4\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:10:59.438814 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:59.438770    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a92a3dbc-e05b-4346-b9a0-618e1c06de90" path="/var/lib/kubelet/pods/a92a3dbc-e05b-4346-b9a0-618e1c06de90/volumes"
Sep 10 21:10:59.486157 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:59.486127    2572 generic.go:356] "Generic (PLEG): container finished" podID="a92a3dbc-e05b-4346-b9a0-618e1c06de90" containerID="3f6a161f6d1df635fe219ba2b3dcc80560807c1a3840e66de7e2a73a3bf026c1" exitCode=137
Sep 10 21:10:59.486521 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:59.486255    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7918-74d908"
Sep 10 21:10:59.486521 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:59.486266    2572 scope.go:117] "RemoveContainer" containerID="3f6a161f6d1df635fe219ba2b3dcc80560807c1a3840e66de7e2a73a3bf026c1"
Sep 10 21:10:59.497213 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:59.497193    2572 scope.go:117] "RemoveContainer" containerID="3f6a161f6d1df635fe219ba2b3dcc80560807c1a3840e66de7e2a73a3bf026c1"
Sep 10 21:10:59.497689 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:10:59.497665    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f6a161f6d1df635fe219ba2b3dcc80560807c1a3840e66de7e2a73a3bf026c1\": container with ID starting with 3f6a161f6d1df635fe219ba2b3dcc80560807c1a3840e66de7e2a73a3bf026c1 not found: ID does not exist" containerID="3f6a161f6d1df635fe219ba2b3dcc80560807c1a3840e66de7e2a73a3bf026c1"
Sep 10 21:10:59.497762 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:10:59.497703    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f6a161f6d1df635fe219ba2b3dcc80560807c1a3840e66de7e2a73a3bf026c1"} err="failed to get container status \"3f6a161f6d1df635fe219ba2b3dcc80560807c1a3840e66de7e2a73a3bf026c1\": rpc error: code = NotFound desc = could not find container \"3f6a161f6d1df635fe219ba2b3dcc80560807c1a3840e66de7e2a73a3bf026c1\": container with ID starting with 3f6a161f6d1df635fe219ba2b3dcc80560807c1a3840e66de7e2a73a3bf026c1 not found: ID does not exist"
Sep 10 21:11:00.492457 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:00.492429    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw" event={"ID":"0e80bd28-a8f4-4f72-a45f-9794ac6b3643","Type":"ContainerStarted","Data":"751efde4d95e28efa4d1ac02602b57ec14dca08c6f799b6742bef56f1c3db65e"}
Sep 10 21:11:00.492802 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:00.492571    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw"
Sep 10 21:11:00.536856 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:00.536815    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw" podStartSLOduration=1.834456386 podStartE2EDuration="3.536799276s" totalImagesPullingTime="1.70234289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:10:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:00.535923959 +0000 UTC m=+845.748388879" watchObservedRunningTime="2026-09-10 21:11:00.536799276 +0000 UTC m=+845.749264195"
Sep 10 21:11:00.647162 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:00.647138    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7918-efb80a"
Sep 10 21:11:00.813238 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:00.813210    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6ppc\" (UniqueName: \"kubernetes.io/projected/86c0b036-8278-4722-ae75-274f616c6a89-kube-api-access-d6ppc\") pod \"86c0b036-8278-4722-ae75-274f616c6a89\" (UID: \"86c0b036-8278-4722-ae75-274f616c6a89\") "
Sep 10 21:11:00.815311 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:00.815285    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c0b036-8278-4722-ae75-274f616c6a89-kube-api-access-d6ppc" (OuterVolumeSpecName: "kube-api-access-d6ppc") pod "86c0b036-8278-4722-ae75-274f616c6a89" (UID: "86c0b036-8278-4722-ae75-274f616c6a89"). InnerVolumeSpecName "kube-api-access-d6ppc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:11:00.914151 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:00.914127    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6ppc\" (UniqueName: \"kubernetes.io/projected/86c0b036-8278-4722-ae75-274f616c6a89-kube-api-access-d6ppc\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:11:01.243485 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:01.243412    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:11:01.438412 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:01.438375    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86c0b036-8278-4722-ae75-274f616c6a89" path="/var/lib/kubelet/pods/86c0b036-8278-4722-ae75-274f616c6a89/volumes"
Sep 10 21:11:01.499863 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:01.499783    2572 generic.go:356] "Generic (PLEG): container finished" podID="86c0b036-8278-4722-ae75-274f616c6a89" containerID="3add59f59022962ebaa055c5f37d1f30b7f0b8f1d1ac3c0bd6ab167883cf80b9" exitCode=137
Sep 10 21:11:01.499863 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:01.499859    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7918-efb80a"
Sep 10 21:11:01.500350 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:01.499920    2572 scope.go:117] "RemoveContainer" containerID="3add59f59022962ebaa055c5f37d1f30b7f0b8f1d1ac3c0bd6ab167883cf80b9"
Sep 10 21:11:01.509118 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:01.509095    2572 scope.go:117] "RemoveContainer" containerID="3add59f59022962ebaa055c5f37d1f30b7f0b8f1d1ac3c0bd6ab167883cf80b9"
Sep 10 21:11:01.509433 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:11:01.509416    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3add59f59022962ebaa055c5f37d1f30b7f0b8f1d1ac3c0bd6ab167883cf80b9\": container with ID starting with 3add59f59022962ebaa055c5f37d1f30b7f0b8f1d1ac3c0bd6ab167883cf80b9 not found: ID does not exist" containerID="3add59f59022962ebaa055c5f37d1f30b7f0b8f1d1ac3c0bd6ab167883cf80b9"
Sep 10 21:11:01.509493 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:01.509445    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3add59f59022962ebaa055c5f37d1f30b7f0b8f1d1ac3c0bd6ab167883cf80b9"} err="failed to get container status \"3add59f59022962ebaa055c5f37d1f30b7f0b8f1d1ac3c0bd6ab167883cf80b9\": rpc error: code = NotFound desc = could not find container \"3add59f59022962ebaa055c5f37d1f30b7f0b8f1d1ac3c0bd6ab167883cf80b9\": container with ID starting with 3add59f59022962ebaa055c5f37d1f30b7f0b8f1d1ac3c0bd6ab167883cf80b9 not found: ID does not exist"
Sep 10 21:11:07.308552 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:07.308519    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb"]
Sep 10 21:11:07.313465 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:07.313443    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb"
Sep 10 21:11:07.321769 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:07.321749    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-c2de43\""
Sep 10 21:11:07.341717 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:07.341695    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb"]
Sep 10 21:11:07.475545 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:07.475517    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8068d378-073f-42f2-a59e-8e9aae3f51ba-plugins-config-volume\") pod \"payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb\" (UID: \"8068d378-073f-42f2-a59e-8e9aae3f51ba\") " pod="openshift-ingress/payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb"
Sep 10 21:11:07.475685 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:07.475584    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh648\" (UniqueName: \"kubernetes.io/projected/8068d378-073f-42f2-a59e-8e9aae3f51ba-kube-api-access-rh648\") pod \"payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb\" (UID: \"8068d378-073f-42f2-a59e-8e9aae3f51ba\") " pod="openshift-ingress/payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb"
Sep 10 21:11:07.507729 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:07.507699    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2"]
Sep 10 21:11:07.511124 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:07.511104    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2"
Sep 10 21:11:07.514541 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:11:07.514511    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2\" is forbidden: User \"system:node:ip-10-0-138-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-138-232.ec2.internal' and this object" podUID="473f99e9-50a0-48c9-9934-613b96cf5998" pod="openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2"
Sep 10 21:11:07.514851 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:11:07.514804    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"payload-processing-e2e-autores-c2de43-dockercfg-x9mtj\" is forbidden: User \"system:node:ip-10-0-138-232.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-138-232.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-c2de43-dockercfg-x9mtj\"" type="*v1.Secret"
Sep 10 21:11:07.545512 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:07.545489    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2"]
Sep 10 21:11:07.576219 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:07.576161    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8068d378-073f-42f2-a59e-8e9aae3f51ba-plugins-config-volume\") pod \"payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb\" (UID: \"8068d378-073f-42f2-a59e-8e9aae3f51ba\") " pod="openshift-ingress/payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb"
Sep 10 21:11:07.576310 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:07.576248    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh648\" (UniqueName: \"kubernetes.io/projected/8068d378-073f-42f2-a59e-8e9aae3f51ba-kube-api-access-rh648\") pod \"payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb\" (UID: \"8068d378-073f-42f2-a59e-8e9aae3f51ba\") " pod="openshift-ingress/payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb"
Sep 10 21:11:07.576845 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:07.576826    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8068d378-073f-42f2-a59e-8e9aae3f51ba-plugins-config-volume\") pod \"payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb\" (UID: \"8068d378-073f-42f2-a59e-8e9aae3f51ba\") " pod="openshift-ingress/payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb"
Sep 10 21:11:07.599093 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:07.599067    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh648\" (UniqueName: \"kubernetes.io/projected/8068d378-073f-42f2-a59e-8e9aae3f51ba-kube-api-access-rh648\") pod \"payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb\" (UID: \"8068d378-073f-42f2-a59e-8e9aae3f51ba\") " pod="openshift-ingress/payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb"
Sep 10 21:11:07.628116 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:07.628078    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb"
Sep 10 21:11:07.677519 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:07.677491    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sjrc\" (UniqueName: \"kubernetes.io/projected/473f99e9-50a0-48c9-9934-613b96cf5998-kube-api-access-8sjrc\") pod \"payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2\" (UID: \"473f99e9-50a0-48c9-9934-613b96cf5998\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2"
Sep 10 21:11:07.677662 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:07.677569    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473f99e9-50a0-48c9-9934-613b96cf5998-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2\" (UID: \"473f99e9-50a0-48c9-9934-613b96cf5998\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2"
Sep 10 21:11:07.778306 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:07.778275    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sjrc\" (UniqueName: \"kubernetes.io/projected/473f99e9-50a0-48c9-9934-613b96cf5998-kube-api-access-8sjrc\") pod \"payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2\" (UID: \"473f99e9-50a0-48c9-9934-613b96cf5998\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2"
Sep 10 21:11:07.778480 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:07.778338    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473f99e9-50a0-48c9-9934-613b96cf5998-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2\" (UID: \"473f99e9-50a0-48c9-9934-613b96cf5998\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2"
Sep 10 21:11:07.778938 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:07.778919    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473f99e9-50a0-48c9-9934-613b96cf5998-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2\" (UID: \"473f99e9-50a0-48c9-9934-613b96cf5998\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2"
Sep 10 21:11:07.792705 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:07.792676    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sjrc\" (UniqueName: \"kubernetes.io/projected/473f99e9-50a0-48c9-9934-613b96cf5998-kube-api-access-8sjrc\") pod \"payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2\" (UID: \"473f99e9-50a0-48c9-9934-613b96cf5998\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2"
Sep 10 21:11:07.911159 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:07.911126    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb"]
Sep 10 21:11:08.544133 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:08.544090    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb" event={"ID":"8068d378-073f-42f2-a59e-8e9aae3f51ba","Type":"ContainerStarted","Data":"0804750d486a444a6e0885e1d07708ba9e5475510aa95c77ed4aa60047dd039a"}
Sep 10 21:11:08.544569 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:08.544152    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb" event={"ID":"8068d378-073f-42f2-a59e-8e9aae3f51ba","Type":"ContainerStarted","Data":"84e5044c73bf1163b235fb19cb932b5225ea17a752b350a372d0cb1910762eb1"}
Sep 10 21:11:08.544569 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:08.544178    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb"
Sep 10 21:11:08.632055 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:08.632022    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-c2de43-dockercfg-x9mtj\""
Sep 10 21:11:08.641356 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:08.641329    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2"
Sep 10 21:11:08.919106 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:08.919014    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb" podStartSLOduration=1.918973842 podStartE2EDuration="1.918973842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:08.614423878 +0000 UTC m=+853.826888798" watchObservedRunningTime="2026-09-10 21:11:08.918973842 +0000 UTC m=+854.131438761"
Sep 10 21:11:08.921245 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:08.921109    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2"]
Sep 10 21:11:08.924079 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:11:08.924051    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod473f99e9_50a0_48c9_9934_613b96cf5998.slice/crio-22f1af82b51d531e1e66f69c192e0295f2c2668cb39512346df7c1270936afc9 WatchSource:0}: Error finding container 22f1af82b51d531e1e66f69c192e0295f2c2668cb39512346df7c1270936afc9: Status 404 returned error can't find the container with id 22f1af82b51d531e1e66f69c192e0295f2c2668cb39512346df7c1270936afc9
Sep 10 21:11:09.551135 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:09.551094    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2" event={"ID":"473f99e9-50a0-48c9-9934-613b96cf5998","Type":"ContainerStarted","Data":"c710a0768b0901ac418b9d0e030259a6ea93e410e5e12fe7660cb8302a99275e"}
Sep 10 21:11:09.551135 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:09.551144    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2" event={"ID":"473f99e9-50a0-48c9-9934-613b96cf5998","Type":"ContainerStarted","Data":"22f1af82b51d531e1e66f69c192e0295f2c2668cb39512346df7c1270936afc9"}
Sep 10 21:11:09.551668 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:09.551185    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2"
Sep 10 21:11:09.570781 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:09.570728    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2" podStartSLOduration=2.570713654 podStartE2EDuration="2.570713654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:09.569250518 +0000 UTC m=+854.781715437" watchObservedRunningTime="2026-09-10 21:11:09.570713654 +0000 UTC m=+854.783178573"
Sep 10 21:11:10.499159 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:10.499102    2572 patch_prober.go:28] interesting pod/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 21:11:10.499326 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:10.499176    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk" podUID="e35a47b7-668d-4c95-b2b8-446d5e762a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 21:11:11.502113 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:11.502058    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw"
Sep 10 21:11:11.562322 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:11.562291    2572 generic.go:356] "Generic (PLEG): container finished" podID="e35a47b7-668d-4c95-b2b8-446d5e762a8c" containerID="c42e42d0caa02bcdb3dd7789d1c054c14d0682df50abd4781b74dc535fd78b7e" exitCode=0
Sep 10 21:11:11.562439 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:11.562367    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk" event={"ID":"e35a47b7-668d-4c95-b2b8-446d5e762a8c","Type":"ContainerDied","Data":"c42e42d0caa02bcdb3dd7789d1c054c14d0682df50abd4781b74dc535fd78b7e"}
Sep 10 21:11:11.562735 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:11.562717    2572 scope.go:117] "RemoveContainer" containerID="c42e42d0caa02bcdb3dd7789d1c054c14d0682df50abd4781b74dc535fd78b7e"
Sep 10 21:11:11.563774 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:11.563721    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:11:11.563854 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:11.563811    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:11:11.563854 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:11.563844    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:11:12.571187 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:12.571152    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk" event={"ID":"e35a47b7-668d-4c95-b2b8-446d5e762a8c","Type":"ContainerStarted","Data":"30f3436cca127fe33e24548cb7222df89890e1e6d37fc052844336191eebce65"}
Sep 10 21:11:13.096287 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:13.096259    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:11:13.096522 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:13.096505    2572 patch_prober.go:28] interesting pod/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 10 21:11:13.096567 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:13.096543    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk" podUID="e35a47b7-668d-4c95-b2b8-446d5e762a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 10 21:11:13.795132 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:13.795098    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:11:14.097132 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:14.097049    2572 patch_prober.go:28] interesting pod/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 10 21:11:14.097132 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:14.097111    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk" podUID="e35a47b7-668d-4c95-b2b8-446d5e762a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 10 21:11:15.096667 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:15.096629    2572 patch_prober.go:28] interesting pod/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 10 21:11:15.097160 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:15.096689    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk" podUID="e35a47b7-668d-4c95-b2b8-446d5e762a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 10 21:11:16.097025 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:16.096973    2572 patch_prober.go:28] interesting pod/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 10 21:11:16.097399 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:16.097044    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk" podUID="e35a47b7-668d-4c95-b2b8-446d5e762a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 10 21:11:17.097096 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:17.097061    2572 patch_prober.go:28] interesting pod/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 10 21:11:17.097464 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:17.097117    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk" podUID="e35a47b7-668d-4c95-b2b8-446d5e762a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 10 21:11:18.097308 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:18.097269    2572 patch_prober.go:28] interesting pod/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 10 21:11:18.097681 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:18.097324    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk" podUID="e35a47b7-668d-4c95-b2b8-446d5e762a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 10 21:11:19.100143 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:19.100117    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:11:19.100485 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:19.100371    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:11:19.101135 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:19.101115    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:11:19.553949 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:19.553915    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb"
Sep 10 21:11:20.557944 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:20.557917    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2"
Sep 10 21:11:21.061470 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.061444    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"]
Sep 10 21:11:21.152570 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.152541    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"]
Sep 10 21:11:21.152716 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.152633    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.161147 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.161126    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7eaf1f-openshift-default-dockercfg-fvstm\""
Sep 10 21:11:21.202115 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.202091    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-workload-certs\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.202237 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.202122    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-workload-socket\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.202237 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.202144    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb5pt\" (UniqueName: \"kubernetes.io/projected/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-kube-api-access-fb5pt\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.202237 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.202194    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istio-podinfo\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.202237 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.202222    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istiod-ca-cert\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.202388 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.202252    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istio-data\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.202388 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.202274    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-credential-socket\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.202388 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.202300    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istio-envoy\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.202388 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.202319    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istio-token\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.302765 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.302740    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istio-data\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.302873 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.302771    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-credential-socket\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.302873 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.302795    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istio-envoy\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.302955 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.302900    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istio-token\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.303024 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.303001    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-workload-certs\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.303087 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.303026    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-workload-socket\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.303087 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.303045    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb5pt\" (UniqueName: \"kubernetes.io/projected/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-kube-api-access-fb5pt\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.303087 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.303065    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istio-podinfo\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.303087 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.303083    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istiod-ca-cert\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.303289 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.303151    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-credential-socket\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.303289 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.303180    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istio-data\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.303380 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.303335    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-workload-socket\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.303423 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.303390    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-workload-certs\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.303650 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.303633    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istiod-ca-cert\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.305110 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.305091    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istio-envoy\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.305567 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.305552    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istio-podinfo\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.316952 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.316900    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istio-token\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.317059 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.317001    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb5pt\" (UniqueName: \"kubernetes.io/projected/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-kube-api-access-fb5pt\") pod \"e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") " pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.464275 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.464253    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:21.748210 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.748184    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"]
Sep 10 21:11:21.750646 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:11:21.750618    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b28f1bf_6fbf_4a14_80fa_990f7ea4ff04.slice/crio-120b7c4a9580b0f91c6416bbab3a360514ead45343bef005d7d645af67466646 WatchSource:0}: Error finding container 120b7c4a9580b0f91c6416bbab3a360514ead45343bef005d7d645af67466646: Status 404 returned error can't find the container with id 120b7c4a9580b0f91c6416bbab3a360514ead45343bef005d7d645af67466646
Sep 10 21:11:21.752753 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.752719    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:11:21.752827 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.752783    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:11:21.752827 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:21.752810    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:11:22.623376 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:22.623332    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl" event={"ID":"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04","Type":"ContainerStarted","Data":"47343e050aa1cd3af2286ac1ed77e1d875c6970911d227ef2adc782298891d31"}
Sep 10 21:11:22.623537 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:22.623394    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl" event={"ID":"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04","Type":"ContainerStarted","Data":"120b7c4a9580b0f91c6416bbab3a360514ead45343bef005d7d645af67466646"}
Sep 10 21:11:22.649517 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:22.649468    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl" podStartSLOduration=1.6494515299999999 podStartE2EDuration="1.64945153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:22.647513039 +0000 UTC m=+867.859977971" watchObservedRunningTime="2026-09-10 21:11:22.64945153 +0000 UTC m=+867.861916448"
Sep 10 21:11:23.464857 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.464826    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:23.469975 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.469947    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:23.588727 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.588692    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"]
Sep 10 21:11:23.591114 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.591095    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:11:23.595825 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.595805    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-43415b1a-kserve-self-signed-certs\""
Sep 10 21:11:23.605561 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.605543    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"]
Sep 10 21:11:23.621068 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.621047    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-home\") pod \"e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz\" (UID: \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\") " pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:11:23.621166 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.621071    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-model-cache\") pod \"e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz\" (UID: \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\") " pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:11:23.621166 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.621095    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9btr8\" (UniqueName: \"kubernetes.io/projected/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-kube-api-access-9btr8\") pod \"e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz\" (UID: \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\") " pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:11:23.621166 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.621120    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-dshm\") pod \"e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz\" (UID: \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\") " pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:11:23.621278 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.621170    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-tls-certs\") pod \"e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz\" (UID: \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\") " pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:11:23.621278 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.621210    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-tmp-dir\") pod \"e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz\" (UID: \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\") " pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:11:23.628503 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.628483    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:23.629277 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.629261    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:23.722290 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.722214    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-home\") pod \"e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz\" (UID: \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\") " pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:11:23.722290 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.722245    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-model-cache\") pod \"e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz\" (UID: \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\") " pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:11:23.722290 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.722268    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9btr8\" (UniqueName: \"kubernetes.io/projected/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-kube-api-access-9btr8\") pod \"e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz\" (UID: \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\") " pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:11:23.722529 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.722310    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-dshm\") pod \"e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz\" (UID: \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\") " pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:11:23.722529 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.722350    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-tls-certs\") pod \"e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz\" (UID: \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\") " pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:11:23.722642 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.722608    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-model-cache\") pod \"e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz\" (UID: \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\") " pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:11:23.722713 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.722686    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-tmp-dir\") pod \"e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz\" (UID: \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\") " pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:11:23.722778 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.722685    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-home\") pod \"e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz\" (UID: \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\") " pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:11:23.722931 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.722912    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-tmp-dir\") pod \"e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz\" (UID: \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\") " pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:11:23.724635 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.724610    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-dshm\") pod \"e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz\" (UID: \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\") " pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:11:23.725117 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.725095    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-tls-certs\") pod \"e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz\" (UID: \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\") " pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:11:23.734821 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.734799    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9btr8\" (UniqueName: \"kubernetes.io/projected/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-kube-api-access-9btr8\") pod \"e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz\" (UID: \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\") " pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:11:23.900807 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:23.900783    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:11:24.177343 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:24.177320    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"]
Sep 10 21:11:24.179491 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:11:24.179460    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45b392a1_bb1b_4f97_a2f4_ccad9b6575ad.slice/crio-05065f4b9a0eaec0f0842f1b5f1fd0830069b1a6d0922e52cc29d3bc3d2fbc9c WatchSource:0}: Error finding container 05065f4b9a0eaec0f0842f1b5f1fd0830069b1a6d0922e52cc29d3bc3d2fbc9c: Status 404 returned error can't find the container with id 05065f4b9a0eaec0f0842f1b5f1fd0830069b1a6d0922e52cc29d3bc3d2fbc9c
Sep 10 21:11:24.635487 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:24.635452    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz" event={"ID":"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad","Type":"ContainerStarted","Data":"053732e9e0f79b5bf49d91de23bfffb6b893c324a0cd921d34a039a606426395"}
Sep 10 21:11:24.635850 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:24.635500    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz" event={"ID":"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad","Type":"ContainerStarted","Data":"05065f4b9a0eaec0f0842f1b5f1fd0830069b1a6d0922e52cc29d3bc3d2fbc9c"}
Sep 10 21:11:24.656609 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:24.656566    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz" podStartSLOduration=1.6565502300000001 podStartE2EDuration="1.65655023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:24.655255416 +0000 UTC m=+869.867720327" watchObservedRunningTime="2026-09-10 21:11:24.65655023 +0000 UTC m=+869.869015146"
Sep 10 21:11:33.901144 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:33.901098    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:11:33.901709 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:33.901166    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:11:33.925748 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:33.925698    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:11:33.971626 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:33.971590    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:11:34.700917 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:34.700890    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:11:45.744710 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:45.744677    2572 generic.go:356] "Generic (PLEG): container finished" podID="9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04" containerID="47343e050aa1cd3af2286ac1ed77e1d875c6970911d227ef2adc782298891d31" exitCode=0
Sep 10 21:11:45.744710 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:45.744723    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl" event={"ID":"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04","Type":"ContainerDied","Data":"47343e050aa1cd3af2286ac1ed77e1d875c6970911d227ef2adc782298891d31"}
Sep 10 21:11:45.745202 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:45.745115    2572 scope.go:117] "RemoveContainer" containerID="47343e050aa1cd3af2286ac1ed77e1d875c6970911d227ef2adc782298891d31"
Sep 10 21:11:45.746310 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:45.746265    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:11:45.746407 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:45.746366    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:11:45.746446 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:45.746418    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:11:46.751799 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:46.751766    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl" event={"ID":"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04","Type":"ContainerStarted","Data":"523b4d2e352411d602dac1297c69cff971d38868c334f6a1de6237183da1636d"}
Sep 10 21:11:47.464974 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:47.464884    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:47.465150 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:47.465121    2572 patch_prober.go:28] interesting pod/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 10 21:11:47.465195 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:47.465169    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl" podUID="9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 10 21:11:48.464860 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:48.464824    2572 patch_prober.go:28] interesting pod/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 10 21:11:48.465273 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:48.464885    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl" podUID="9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 10 21:11:49.465470 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:49.465437    2572 patch_prober.go:28] interesting pod/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 10 21:11:49.465853 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:49.465489    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl" podUID="9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 10 21:11:50.465574 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:50.465541    2572 patch_prober.go:28] interesting pod/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 10 21:11:50.465935 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:50.465609    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl" podUID="9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 10 21:11:50.575067 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:50.575029    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp"]
Sep 10 21:11:50.577421 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:50.577400    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp"
Sep 10 21:11:50.581534 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:50.581499    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d00f8e2d\""
Sep 10 21:11:50.581639 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:50.581576    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d00f8e2d-dockercfg-scnwk\""
Sep 10 21:11:50.635633 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:50.635592    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp"]
Sep 10 21:11:50.678722 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:50.678695    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687a09e6-cd02-465f-b059-df62de4d718c-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp\" (UID: \"687a09e6-cd02-465f-b059-df62de4d718c\") " pod="openshift-ingress/payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp"
Sep 10 21:11:50.678886 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:50.678744    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq78q\" (UniqueName: \"kubernetes.io/projected/687a09e6-cd02-465f-b059-df62de4d718c-kube-api-access-nq78q\") pod \"payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp\" (UID: \"687a09e6-cd02-465f-b059-df62de4d718c\") " pod="openshift-ingress/payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp"
Sep 10 21:11:50.779387 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:50.779360    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq78q\" (UniqueName: \"kubernetes.io/projected/687a09e6-cd02-465f-b059-df62de4d718c-kube-api-access-nq78q\") pod \"payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp\" (UID: \"687a09e6-cd02-465f-b059-df62de4d718c\") " pod="openshift-ingress/payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp"
Sep 10 21:11:50.779530 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:50.779452    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687a09e6-cd02-465f-b059-df62de4d718c-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp\" (UID: \"687a09e6-cd02-465f-b059-df62de4d718c\") " pod="openshift-ingress/payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp"
Sep 10 21:11:50.780085 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:50.780068    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687a09e6-cd02-465f-b059-df62de4d718c-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp\" (UID: \"687a09e6-cd02-465f-b059-df62de4d718c\") " pod="openshift-ingress/payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp"
Sep 10 21:11:50.794631 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:50.794605    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq78q\" (UniqueName: \"kubernetes.io/projected/687a09e6-cd02-465f-b059-df62de4d718c-kube-api-access-nq78q\") pod \"payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp\" (UID: \"687a09e6-cd02-465f-b059-df62de4d718c\") " pod="openshift-ingress/payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp"
Sep 10 21:11:50.888097 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:50.888071    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp"
Sep 10 21:11:51.189080 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:51.189055    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp"]
Sep 10 21:11:51.190463 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:11:51.190431    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod687a09e6_cd02_465f_b059_df62de4d718c.slice/crio-ccdb6ad65fca5e4fbbb527969a25b741355d58242e6b42953fa29d842989cdec WatchSource:0}: Error finding container ccdb6ad65fca5e4fbbb527969a25b741355d58242e6b42953fa29d842989cdec: Status 404 returned error can't find the container with id ccdb6ad65fca5e4fbbb527969a25b741355d58242e6b42953fa29d842989cdec
Sep 10 21:11:51.465648 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:51.465562    2572 patch_prober.go:28] interesting pod/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 10 21:11:51.465648 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:51.465616    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl" podUID="9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 10 21:11:51.778556 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:51.778526    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp" event={"ID":"687a09e6-cd02-465f-b059-df62de4d718c","Type":"ContainerStarted","Data":"d602cbe4984b11230109bae5473b8ab637521e5d93d0d63ac82e78b13d97529f"}
Sep 10 21:11:51.778731 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:51.778584    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp" event={"ID":"687a09e6-cd02-465f-b059-df62de4d718c","Type":"ContainerStarted","Data":"ccdb6ad65fca5e4fbbb527969a25b741355d58242e6b42953fa29d842989cdec"}
Sep 10 21:11:51.778731 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:51.778658    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp"
Sep 10 21:11:51.799677 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:51.799620    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp" podStartSLOduration=1.799605982 podStartE2EDuration="1.799605982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:51.797412966 +0000 UTC m=+897.009877886" watchObservedRunningTime="2026-09-10 21:11:51.799605982 +0000 UTC m=+897.012070900"
Sep 10 21:11:52.037252 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:52.037177    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:11:52.468861 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:52.468782    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:52.469297 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:52.469079    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:11:52.469808 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:11:52.469789    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:12:02.785795 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:02.785766    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp"
Sep 10 21:12:11.232548 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:11.232511    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:12:21.268364 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:21.268324    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp"]
Sep 10 21:12:21.268946 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:21.268584    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp" podUID="687a09e6-cd02-465f-b059-df62de4d718c" containerName="payload-processing" containerID="cri-o://d602cbe4984b11230109bae5473b8ab637521e5d93d0d63ac82e78b13d97529f" gracePeriod=30
Sep 10 21:12:21.519425 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:21.519367    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp"
Sep 10 21:12:21.606945 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:21.606914    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nq78q\" (UniqueName: \"kubernetes.io/projected/687a09e6-cd02-465f-b059-df62de4d718c-kube-api-access-nq78q\") pod \"687a09e6-cd02-465f-b059-df62de4d718c\" (UID: \"687a09e6-cd02-465f-b059-df62de4d718c\") "
Sep 10 21:12:21.607121 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:21.607092    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687a09e6-cd02-465f-b059-df62de4d718c-plugins-config-volume\") pod \"687a09e6-cd02-465f-b059-df62de4d718c\" (UID: \"687a09e6-cd02-465f-b059-df62de4d718c\") "
Sep 10 21:12:21.607476 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:21.607446    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/687a09e6-cd02-465f-b059-df62de4d718c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "687a09e6-cd02-465f-b059-df62de4d718c" (UID: "687a09e6-cd02-465f-b059-df62de4d718c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:12:21.609149 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:21.609125    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687a09e6-cd02-465f-b059-df62de4d718c-kube-api-access-nq78q" (OuterVolumeSpecName: "kube-api-access-nq78q") pod "687a09e6-cd02-465f-b059-df62de4d718c" (UID: "687a09e6-cd02-465f-b059-df62de4d718c"). InnerVolumeSpecName "kube-api-access-nq78q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:12:21.708805 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:21.708745    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687a09e6-cd02-465f-b059-df62de4d718c-plugins-config-volume\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:12:21.708805 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:21.708771    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq78q\" (UniqueName: \"kubernetes.io/projected/687a09e6-cd02-465f-b059-df62de4d718c-kube-api-access-nq78q\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:12:21.929949 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:21.929925    2572 generic.go:356] "Generic (PLEG): container finished" podID="687a09e6-cd02-465f-b059-df62de4d718c" containerID="d602cbe4984b11230109bae5473b8ab637521e5d93d0d63ac82e78b13d97529f" exitCode=0
Sep 10 21:12:21.930068 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:21.929972    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp"
Sep 10 21:12:21.930068 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:21.930024    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp" event={"ID":"687a09e6-cd02-465f-b059-df62de4d718c","Type":"ContainerDied","Data":"d602cbe4984b11230109bae5473b8ab637521e5d93d0d63ac82e78b13d97529f"}
Sep 10 21:12:21.930142 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:21.930080    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp" event={"ID":"687a09e6-cd02-465f-b059-df62de4d718c","Type":"ContainerDied","Data":"ccdb6ad65fca5e4fbbb527969a25b741355d58242e6b42953fa29d842989cdec"}
Sep 10 21:12:21.930142 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:21.930103    2572 scope.go:117] "RemoveContainer" containerID="d602cbe4984b11230109bae5473b8ab637521e5d93d0d63ac82e78b13d97529f"
Sep 10 21:12:21.941591 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:21.941577    2572 scope.go:117] "RemoveContainer" containerID="d602cbe4984b11230109bae5473b8ab637521e5d93d0d63ac82e78b13d97529f"
Sep 10 21:12:21.941916 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:12:21.941887    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d602cbe4984b11230109bae5473b8ab637521e5d93d0d63ac82e78b13d97529f\": container with ID starting with d602cbe4984b11230109bae5473b8ab637521e5d93d0d63ac82e78b13d97529f not found: ID does not exist" containerID="d602cbe4984b11230109bae5473b8ab637521e5d93d0d63ac82e78b13d97529f"
Sep 10 21:12:21.942009 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:21.941927    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d602cbe4984b11230109bae5473b8ab637521e5d93d0d63ac82e78b13d97529f"} err="failed to get container status \"d602cbe4984b11230109bae5473b8ab637521e5d93d0d63ac82e78b13d97529f\": rpc error: code = NotFound desc = could not find container \"d602cbe4984b11230109bae5473b8ab637521e5d93d0d63ac82e78b13d97529f\": container with ID starting with d602cbe4984b11230109bae5473b8ab637521e5d93d0d63ac82e78b13d97529f not found: ID does not exist"
Sep 10 21:12:21.954629 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:21.954604    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp"]
Sep 10 21:12:21.958135 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:21.958116    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d00f8e2d-6b85b6b45d-nflfp"]
Sep 10 21:12:22.285941 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.285910    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"]
Sep 10 21:12:22.286458 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.286437    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687a09e6-cd02-465f-b059-df62de4d718c" containerName="payload-processing"
Sep 10 21:12:22.286539 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.286459    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="687a09e6-cd02-465f-b059-df62de4d718c" containerName="payload-processing"
Sep 10 21:12:22.286590 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.286561    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="687a09e6-cd02-465f-b059-df62de4d718c" containerName="payload-processing"
Sep 10 21:12:22.290788 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.290771    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:12:22.293250 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.293231    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c2de43\"/\"autores-model-c2de43-kserve-self-signed-certs\""
Sep 10 21:12:22.293341 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.293276    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c2de43\"/\"openshift-service-ca.crt\""
Sep 10 21:12:22.294417 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.294397    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c2de43\"/\"default-dockercfg-dx2gc\""
Sep 10 21:12:22.294487 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.294458    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c2de43\"/\"kube-root-ca.crt\""
Sep 10 21:12:22.300226 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.300203    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"]
Sep 10 21:12:22.414105 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.414080    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7379f436-2286-4446-a281-2f37237b3838-model-cache\") pod \"autores-model-c2de43-kserve-56b4776879-xt6hs\" (UID: \"7379f436-2286-4446-a281-2f37237b3838\") " pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:12:22.414105 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.414106    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7379f436-2286-4446-a281-2f37237b3838-tmp-dir\") pod \"autores-model-c2de43-kserve-56b4776879-xt6hs\" (UID: \"7379f436-2286-4446-a281-2f37237b3838\") " pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:12:22.414378 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.414151    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7379f436-2286-4446-a281-2f37237b3838-home\") pod \"autores-model-c2de43-kserve-56b4776879-xt6hs\" (UID: \"7379f436-2286-4446-a281-2f37237b3838\") " pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:12:22.414378 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.414170    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7379f436-2286-4446-a281-2f37237b3838-dshm\") pod \"autores-model-c2de43-kserve-56b4776879-xt6hs\" (UID: \"7379f436-2286-4446-a281-2f37237b3838\") " pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:12:22.414378 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.414220    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xwf7\" (UniqueName: \"kubernetes.io/projected/7379f436-2286-4446-a281-2f37237b3838-kube-api-access-8xwf7\") pod \"autores-model-c2de43-kserve-56b4776879-xt6hs\" (UID: \"7379f436-2286-4446-a281-2f37237b3838\") " pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:12:22.414378 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.414270    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7379f436-2286-4446-a281-2f37237b3838-tls-certs\") pod \"autores-model-c2de43-kserve-56b4776879-xt6hs\" (UID: \"7379f436-2286-4446-a281-2f37237b3838\") " pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:12:22.514704 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.514671    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7379f436-2286-4446-a281-2f37237b3838-home\") pod \"autores-model-c2de43-kserve-56b4776879-xt6hs\" (UID: \"7379f436-2286-4446-a281-2f37237b3838\") " pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:12:22.514704 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.514703    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7379f436-2286-4446-a281-2f37237b3838-dshm\") pod \"autores-model-c2de43-kserve-56b4776879-xt6hs\" (UID: \"7379f436-2286-4446-a281-2f37237b3838\") " pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:12:22.514916 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.514744    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xwf7\" (UniqueName: \"kubernetes.io/projected/7379f436-2286-4446-a281-2f37237b3838-kube-api-access-8xwf7\") pod \"autores-model-c2de43-kserve-56b4776879-xt6hs\" (UID: \"7379f436-2286-4446-a281-2f37237b3838\") " pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:12:22.514916 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.514782    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7379f436-2286-4446-a281-2f37237b3838-tls-certs\") pod \"autores-model-c2de43-kserve-56b4776879-xt6hs\" (UID: \"7379f436-2286-4446-a281-2f37237b3838\") " pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:12:22.514916 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.514835    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7379f436-2286-4446-a281-2f37237b3838-model-cache\") pod \"autores-model-c2de43-kserve-56b4776879-xt6hs\" (UID: \"7379f436-2286-4446-a281-2f37237b3838\") " pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:12:22.514916 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.514855    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7379f436-2286-4446-a281-2f37237b3838-tmp-dir\") pod \"autores-model-c2de43-kserve-56b4776879-xt6hs\" (UID: \"7379f436-2286-4446-a281-2f37237b3838\") " pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:12:22.515246 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.515224    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7379f436-2286-4446-a281-2f37237b3838-home\") pod \"autores-model-c2de43-kserve-56b4776879-xt6hs\" (UID: \"7379f436-2286-4446-a281-2f37237b3838\") " pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:12:22.515246 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.515239    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7379f436-2286-4446-a281-2f37237b3838-tmp-dir\") pod \"autores-model-c2de43-kserve-56b4776879-xt6hs\" (UID: \"7379f436-2286-4446-a281-2f37237b3838\") " pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:12:22.515348 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.515264    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7379f436-2286-4446-a281-2f37237b3838-model-cache\") pod \"autores-model-c2de43-kserve-56b4776879-xt6hs\" (UID: \"7379f436-2286-4446-a281-2f37237b3838\") " pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:12:22.517125 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.517097    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7379f436-2286-4446-a281-2f37237b3838-dshm\") pod \"autores-model-c2de43-kserve-56b4776879-xt6hs\" (UID: \"7379f436-2286-4446-a281-2f37237b3838\") " pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:12:22.517427 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.517404    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7379f436-2286-4446-a281-2f37237b3838-tls-certs\") pod \"autores-model-c2de43-kserve-56b4776879-xt6hs\" (UID: \"7379f436-2286-4446-a281-2f37237b3838\") " pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:12:22.523012 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.522970    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xwf7\" (UniqueName: \"kubernetes.io/projected/7379f436-2286-4446-a281-2f37237b3838-kube-api-access-8xwf7\") pod \"autores-model-c2de43-kserve-56b4776879-xt6hs\" (UID: \"7379f436-2286-4446-a281-2f37237b3838\") " pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:12:22.600482 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.600424    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:12:22.872021 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.871930    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"]
Sep 10 21:12:22.874308 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:12:22.874281    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7379f436_2286_4446_a281_2f37237b3838.slice/crio-0d371a9f86afa58ad737d1b8df6e718e8a07cfc22cc0bb8561dbdd152211955b WatchSource:0}: Error finding container 0d371a9f86afa58ad737d1b8df6e718e8a07cfc22cc0bb8561dbdd152211955b: Status 404 returned error can't find the container with id 0d371a9f86afa58ad737d1b8df6e718e8a07cfc22cc0bb8561dbdd152211955b
Sep 10 21:12:22.938358 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:22.938330    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs" event={"ID":"7379f436-2286-4446-a281-2f37237b3838","Type":"ContainerStarted","Data":"0d371a9f86afa58ad737d1b8df6e718e8a07cfc22cc0bb8561dbdd152211955b"}
Sep 10 21:12:23.438267 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:23.438236    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="687a09e6-cd02-465f-b059-df62de4d718c" path="/var/lib/kubelet/pods/687a09e6-cd02-465f-b059-df62de4d718c/volumes"
Sep 10 21:12:23.946436 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:23.946399    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs" event={"ID":"7379f436-2286-4446-a281-2f37237b3838","Type":"ContainerStarted","Data":"19351ed1646950d842b33aee72876ce372f0195649a1046c312c64ec0654ba81"}
Sep 10 21:12:23.965680 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:23.965625    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs" podStartSLOduration=1.9656086780000002 podStartE2EDuration="1.965608678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:12:23.964139125 +0000 UTC m=+929.176604046" watchObservedRunningTime="2026-09-10 21:12:23.965608678 +0000 UTC m=+929.178073618"
Sep 10 21:12:26.865943 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:26.865910    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v"]
Sep 10 21:12:26.869450 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:26.869434    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v"
Sep 10 21:12:26.872366 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:26.872342    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-801c3274\""
Sep 10 21:12:26.872513 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:26.872386    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-801c3274-dockercfg-6wk6x\""
Sep 10 21:12:26.880033 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:26.880011    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v"]
Sep 10 21:12:26.955131 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:26.955099    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d541aa49-d228-42be-ab69-4ad1719bbd37-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v\" (UID: \"d541aa49-d228-42be-ab69-4ad1719bbd37\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v"
Sep 10 21:12:26.955131 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:26.955133    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntxn4\" (UniqueName: \"kubernetes.io/projected/d541aa49-d228-42be-ab69-4ad1719bbd37-kube-api-access-ntxn4\") pod \"payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v\" (UID: \"d541aa49-d228-42be-ab69-4ad1719bbd37\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v"
Sep 10 21:12:27.056517 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:27.056494    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d541aa49-d228-42be-ab69-4ad1719bbd37-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v\" (UID: \"d541aa49-d228-42be-ab69-4ad1719bbd37\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v"
Sep 10 21:12:27.056656 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:27.056523    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntxn4\" (UniqueName: \"kubernetes.io/projected/d541aa49-d228-42be-ab69-4ad1719bbd37-kube-api-access-ntxn4\") pod \"payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v\" (UID: \"d541aa49-d228-42be-ab69-4ad1719bbd37\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v"
Sep 10 21:12:27.057221 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:27.057204    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d541aa49-d228-42be-ab69-4ad1719bbd37-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v\" (UID: \"d541aa49-d228-42be-ab69-4ad1719bbd37\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v"
Sep 10 21:12:27.065392 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:27.065371    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntxn4\" (UniqueName: \"kubernetes.io/projected/d541aa49-d228-42be-ab69-4ad1719bbd37-kube-api-access-ntxn4\") pod \"payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v\" (UID: \"d541aa49-d228-42be-ab69-4ad1719bbd37\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v"
Sep 10 21:12:27.180237 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:27.180190    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v"
Sep 10 21:12:27.454371 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:27.454345    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v"]
Sep 10 21:12:27.456109 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:12:27.456078    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd541aa49_d228_42be_ab69_4ad1719bbd37.slice/crio-de4699f0d94f4ad5045c58edc036081fcb89e6b1d2b3a8ae33e8c14860e56da7 WatchSource:0}: Error finding container de4699f0d94f4ad5045c58edc036081fcb89e6b1d2b3a8ae33e8c14860e56da7: Status 404 returned error can't find the container with id de4699f0d94f4ad5045c58edc036081fcb89e6b1d2b3a8ae33e8c14860e56da7
Sep 10 21:12:27.966849 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:27.966814    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v" event={"ID":"d541aa49-d228-42be-ab69-4ad1719bbd37","Type":"ContainerStarted","Data":"cc4f5cf4d5071d44b970ae38899fc14e0002305419e510157d6e18b1d24f4fff"}
Sep 10 21:12:27.967202 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:27.966870    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v" event={"ID":"d541aa49-d228-42be-ab69-4ad1719bbd37","Type":"ContainerStarted","Data":"de4699f0d94f4ad5045c58edc036081fcb89e6b1d2b3a8ae33e8c14860e56da7"}
Sep 10 21:12:27.967202 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:27.966891    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v"
Sep 10 21:12:27.986387 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:27.986342    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v" podStartSLOduration=1.986327296 podStartE2EDuration="1.986327296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:12:27.983335653 +0000 UTC m=+933.195800572" watchObservedRunningTime="2026-09-10 21:12:27.986327296 +0000 UTC m=+933.198792208"
Sep 10 21:12:32.334853 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:32.334818    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:12:32.601258 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:32.601171    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:12:32.601258 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:32.601221    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:12:32.614593 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:32.614561    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:12:33.004149 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:33.004062    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:12:38.973104 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:38.973069    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v"
Sep 10 21:12:53.928802 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:53.928767    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:12:56.564004 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:56.563941    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v"]
Sep 10 21:12:56.564901 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:56.564211    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v" podUID="d541aa49-d228-42be-ab69-4ad1719bbd37" containerName="payload-pre-processing" containerID="cri-o://cc4f5cf4d5071d44b970ae38899fc14e0002305419e510157d6e18b1d24f4fff" gracePeriod=30
Sep 10 21:12:56.811856 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:56.811830    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v"
Sep 10 21:12:56.994551 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:56.994467    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntxn4\" (UniqueName: \"kubernetes.io/projected/d541aa49-d228-42be-ab69-4ad1719bbd37-kube-api-access-ntxn4\") pod \"d541aa49-d228-42be-ab69-4ad1719bbd37\" (UID: \"d541aa49-d228-42be-ab69-4ad1719bbd37\") "
Sep 10 21:12:56.994722 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:56.994561    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d541aa49-d228-42be-ab69-4ad1719bbd37-plugins-config-volume\") pod \"d541aa49-d228-42be-ab69-4ad1719bbd37\" (UID: \"d541aa49-d228-42be-ab69-4ad1719bbd37\") "
Sep 10 21:12:56.994915 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:56.994888    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d541aa49-d228-42be-ab69-4ad1719bbd37-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d541aa49-d228-42be-ab69-4ad1719bbd37" (UID: "d541aa49-d228-42be-ab69-4ad1719bbd37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:12:56.996715 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:56.996694    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d541aa49-d228-42be-ab69-4ad1719bbd37-kube-api-access-ntxn4" (OuterVolumeSpecName: "kube-api-access-ntxn4") pod "d541aa49-d228-42be-ab69-4ad1719bbd37" (UID: "d541aa49-d228-42be-ab69-4ad1719bbd37"). InnerVolumeSpecName "kube-api-access-ntxn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:12:57.096108 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:57.096084    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntxn4\" (UniqueName: \"kubernetes.io/projected/d541aa49-d228-42be-ab69-4ad1719bbd37-kube-api-access-ntxn4\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:12:57.096108 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:57.096107    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d541aa49-d228-42be-ab69-4ad1719bbd37-plugins-config-volume\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:12:57.115936 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:57.115911    2572 generic.go:356] "Generic (PLEG): container finished" podID="d541aa49-d228-42be-ab69-4ad1719bbd37" containerID="cc4f5cf4d5071d44b970ae38899fc14e0002305419e510157d6e18b1d24f4fff" exitCode=0
Sep 10 21:12:57.116054 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:57.115959    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v"
Sep 10 21:12:57.116054 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:57.115963    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v" event={"ID":"d541aa49-d228-42be-ab69-4ad1719bbd37","Type":"ContainerDied","Data":"cc4f5cf4d5071d44b970ae38899fc14e0002305419e510157d6e18b1d24f4fff"}
Sep 10 21:12:57.116054 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:57.116047    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v" event={"ID":"d541aa49-d228-42be-ab69-4ad1719bbd37","Type":"ContainerDied","Data":"de4699f0d94f4ad5045c58edc036081fcb89e6b1d2b3a8ae33e8c14860e56da7"}
Sep 10 21:12:57.116175 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:57.116070    2572 scope.go:117] "RemoveContainer" containerID="cc4f5cf4d5071d44b970ae38899fc14e0002305419e510157d6e18b1d24f4fff"
Sep 10 21:12:57.124897 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:57.124882    2572 scope.go:117] "RemoveContainer" containerID="cc4f5cf4d5071d44b970ae38899fc14e0002305419e510157d6e18b1d24f4fff"
Sep 10 21:12:57.125171 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:12:57.125153    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc4f5cf4d5071d44b970ae38899fc14e0002305419e510157d6e18b1d24f4fff\": container with ID starting with cc4f5cf4d5071d44b970ae38899fc14e0002305419e510157d6e18b1d24f4fff not found: ID does not exist" containerID="cc4f5cf4d5071d44b970ae38899fc14e0002305419e510157d6e18b1d24f4fff"
Sep 10 21:12:57.125228 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:57.125180    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc4f5cf4d5071d44b970ae38899fc14e0002305419e510157d6e18b1d24f4fff"} err="failed to get container status \"cc4f5cf4d5071d44b970ae38899fc14e0002305419e510157d6e18b1d24f4fff\": rpc error: code = NotFound desc = could not find container \"cc4f5cf4d5071d44b970ae38899fc14e0002305419e510157d6e18b1d24f4fff\": container with ID starting with cc4f5cf4d5071d44b970ae38899fc14e0002305419e510157d6e18b1d24f4fff not found: ID does not exist"
Sep 10 21:12:57.137552 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:57.137529    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v"]
Sep 10 21:12:57.143703 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:57.143681    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-801c3274-6949946866-jnj6v"]
Sep 10 21:12:57.438685 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:12:57.438658    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d541aa49-d228-42be-ab69-4ad1719bbd37" path="/var/lib/kubelet/pods/d541aa49-d228-42be-ab69-4ad1719bbd37/volumes"
Sep 10 21:13:02.911710 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:02.911675    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm"]
Sep 10 21:13:02.912149 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:02.912044    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d541aa49-d228-42be-ab69-4ad1719bbd37" containerName="payload-pre-processing"
Sep 10 21:13:02.912149 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:02.912055    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d541aa49-d228-42be-ab69-4ad1719bbd37" containerName="payload-pre-processing"
Sep 10 21:13:02.912149 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:02.912126    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d541aa49-d228-42be-ab69-4ad1719bbd37" containerName="payload-pre-processing"
Sep 10 21:13:02.916631 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:02.916613    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm"
Sep 10 21:13:02.919572 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:02.919554    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-cbc740e6-dockercfg-g9997\""
Sep 10 21:13:02.920070 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:02.920053    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-cbc740e6\""
Sep 10 21:13:02.928153 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:02.928130    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm"]
Sep 10 21:13:02.932793 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:02.932768    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk"]
Sep 10 21:13:02.935939 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:02.935919    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk"
Sep 10 21:13:02.948273 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:02.948250    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk"]
Sep 10 21:13:03.043819 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:03.043786    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll5kd\" (UniqueName: \"kubernetes.io/projected/7f5bf3e5-884e-45c7-962b-7b9c61da7c65-kube-api-access-ll5kd\") pod \"payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm\" (UID: \"7f5bf3e5-884e-45c7-962b-7b9c61da7c65\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm"
Sep 10 21:13:03.043819 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:03.043821    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f5bf3e5-884e-45c7-962b-7b9c61da7c65-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm\" (UID: \"7f5bf3e5-884e-45c7-962b-7b9c61da7c65\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm"
Sep 10 21:13:03.044014 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:03.043837    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a30a98-83e5-4648-8aa5-52144bd8b090-plugins-config-volume\") pod \"payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk\" (UID: \"19a30a98-83e5-4648-8aa5-52144bd8b090\") " pod="openshift-ingress/payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk"
Sep 10 21:13:03.044014 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:03.043859    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whjqs\" (UniqueName: \"kubernetes.io/projected/19a30a98-83e5-4648-8aa5-52144bd8b090-kube-api-access-whjqs\") pod \"payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk\" (UID: \"19a30a98-83e5-4648-8aa5-52144bd8b090\") " pod="openshift-ingress/payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk"
Sep 10 21:13:03.144540 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:03.144515    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll5kd\" (UniqueName: \"kubernetes.io/projected/7f5bf3e5-884e-45c7-962b-7b9c61da7c65-kube-api-access-ll5kd\") pod \"payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm\" (UID: \"7f5bf3e5-884e-45c7-962b-7b9c61da7c65\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm"
Sep 10 21:13:03.144702 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:03.144546    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f5bf3e5-884e-45c7-962b-7b9c61da7c65-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm\" (UID: \"7f5bf3e5-884e-45c7-962b-7b9c61da7c65\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm"
Sep 10 21:13:03.144702 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:03.144565    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a30a98-83e5-4648-8aa5-52144bd8b090-plugins-config-volume\") pod \"payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk\" (UID: \"19a30a98-83e5-4648-8aa5-52144bd8b090\") " pod="openshift-ingress/payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk"
Sep 10 21:13:03.144702 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:03.144619    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whjqs\" (UniqueName: \"kubernetes.io/projected/19a30a98-83e5-4648-8aa5-52144bd8b090-kube-api-access-whjqs\") pod \"payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk\" (UID: \"19a30a98-83e5-4648-8aa5-52144bd8b090\") " pod="openshift-ingress/payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk"
Sep 10 21:13:03.145314 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:03.145296    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a30a98-83e5-4648-8aa5-52144bd8b090-plugins-config-volume\") pod \"payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk\" (UID: \"19a30a98-83e5-4648-8aa5-52144bd8b090\") " pod="openshift-ingress/payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk"
Sep 10 21:13:03.145380 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:03.145296    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f5bf3e5-884e-45c7-962b-7b9c61da7c65-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm\" (UID: \"7f5bf3e5-884e-45c7-962b-7b9c61da7c65\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm"
Sep 10 21:13:03.153251 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:03.153224    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whjqs\" (UniqueName: \"kubernetes.io/projected/19a30a98-83e5-4648-8aa5-52144bd8b090-kube-api-access-whjqs\") pod \"payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk\" (UID: \"19a30a98-83e5-4648-8aa5-52144bd8b090\") " pod="openshift-ingress/payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk"
Sep 10 21:13:03.153357 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:03.153327    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll5kd\" (UniqueName: \"kubernetes.io/projected/7f5bf3e5-884e-45c7-962b-7b9c61da7c65-kube-api-access-ll5kd\") pod \"payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm\" (UID: \"7f5bf3e5-884e-45c7-962b-7b9c61da7c65\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm"
Sep 10 21:13:03.226370 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:03.226311    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm"
Sep 10 21:13:03.244964 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:03.244941    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk"
Sep 10 21:13:03.501709 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:03.501685    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm"]
Sep 10 21:13:03.503865 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:13:03.503839    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5bf3e5_884e_45c7_962b_7b9c61da7c65.slice/crio-0d460d9b9ee90a82d5308d1223e6ade9e4c91d665a261969794de4cfaee9ba04 WatchSource:0}: Error finding container 0d460d9b9ee90a82d5308d1223e6ade9e4c91d665a261969794de4cfaee9ba04: Status 404 returned error can't find the container with id 0d460d9b9ee90a82d5308d1223e6ade9e4c91d665a261969794de4cfaee9ba04
Sep 10 21:13:03.530387 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:03.530367    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk"]
Sep 10 21:13:03.535974 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:13:03.535951    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a30a98_83e5_4648_8aa5_52144bd8b090.slice/crio-ac9633a924d02bc25387564cf083bab6a2b04838408874fc08944287bdb37800 WatchSource:0}: Error finding container ac9633a924d02bc25387564cf083bab6a2b04838408874fc08944287bdb37800: Status 404 returned error can't find the container with id ac9633a924d02bc25387564cf083bab6a2b04838408874fc08944287bdb37800
Sep 10 21:13:04.156561 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:04.156527    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk" event={"ID":"19a30a98-83e5-4648-8aa5-52144bd8b090","Type":"ContainerStarted","Data":"2a3a19923fd1305ed8d5c2dec41ae5ac10d1978d4a928ec86b2ffe5917f8158e"}
Sep 10 21:13:04.157018 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:04.156593    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk"
Sep 10 21:13:04.157018 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:04.156619    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk" event={"ID":"19a30a98-83e5-4648-8aa5-52144bd8b090","Type":"ContainerStarted","Data":"ac9633a924d02bc25387564cf083bab6a2b04838408874fc08944287bdb37800"}
Sep 10 21:13:04.158450 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:04.158420    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm" event={"ID":"7f5bf3e5-884e-45c7-962b-7b9c61da7c65","Type":"ContainerStarted","Data":"15c9b6b6f31b9a46cb22db1b5f88a4a85d3907ed2d420581e7a7fe456cc64a1e"}
Sep 10 21:13:04.158591 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:04.158469    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm" event={"ID":"7f5bf3e5-884e-45c7-962b-7b9c61da7c65","Type":"ContainerStarted","Data":"0d460d9b9ee90a82d5308d1223e6ade9e4c91d665a261969794de4cfaee9ba04"}
Sep 10 21:13:04.158591 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:04.158560    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm"
Sep 10 21:13:04.174783 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:04.174731    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk" podStartSLOduration=2.174713684 podStartE2EDuration="2.174713684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:13:04.173957439 +0000 UTC m=+969.386422340" watchObservedRunningTime="2026-09-10 21:13:04.174713684 +0000 UTC m=+969.387178603"
Sep 10 21:13:04.193882 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:04.193834    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm" podStartSLOduration=2.193820786 podStartE2EDuration="2.193820786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:13:04.192566598 +0000 UTC m=+969.405031516" watchObservedRunningTime="2026-09-10 21:13:04.193820786 +0000 UTC m=+969.406285705"
Sep 10 21:13:07.769724 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:07.769691    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb"]
Sep 10 21:13:07.770200 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:07.769946    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb" podUID="8068d378-073f-42f2-a59e-8e9aae3f51ba" containerName="payload-processing" containerID="cri-o://0804750d486a444a6e0885e1d07708ba9e5475510aa95c77ed4aa60047dd039a" gracePeriod=30
Sep 10 21:13:07.778264 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:07.778136    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2"]
Sep 10 21:13:07.778389 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:07.778356    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2" podUID="473f99e9-50a0-48c9-9934-613b96cf5998" containerName="payload-pre-processing" containerID="cri-o://c710a0768b0901ac418b9d0e030259a6ea93e410e5e12fe7660cb8302a99275e" gracePeriod=30
Sep 10 21:13:08.050828 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.050799    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb"
Sep 10 21:13:08.053691 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.053674    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2"
Sep 10 21:13:08.081289 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.081237    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rh648\" (UniqueName: \"kubernetes.io/projected/8068d378-073f-42f2-a59e-8e9aae3f51ba-kube-api-access-rh648\") pod \"8068d378-073f-42f2-a59e-8e9aae3f51ba\" (UID: \"8068d378-073f-42f2-a59e-8e9aae3f51ba\") "
Sep 10 21:13:08.081452 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.081351    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8068d378-073f-42f2-a59e-8e9aae3f51ba-plugins-config-volume\") pod \"8068d378-073f-42f2-a59e-8e9aae3f51ba\" (UID: \"8068d378-073f-42f2-a59e-8e9aae3f51ba\") "
Sep 10 21:13:08.081452 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.081385    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sjrc\" (UniqueName: \"kubernetes.io/projected/473f99e9-50a0-48c9-9934-613b96cf5998-kube-api-access-8sjrc\") pod \"473f99e9-50a0-48c9-9934-613b96cf5998\" (UID: \"473f99e9-50a0-48c9-9934-613b96cf5998\") "
Sep 10 21:13:08.081452 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.081440    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473f99e9-50a0-48c9-9934-613b96cf5998-plugins-config-volume\") pod \"473f99e9-50a0-48c9-9934-613b96cf5998\" (UID: \"473f99e9-50a0-48c9-9934-613b96cf5998\") "
Sep 10 21:13:08.082206 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.082153    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/473f99e9-50a0-48c9-9934-613b96cf5998-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "473f99e9-50a0-48c9-9934-613b96cf5998" (UID: "473f99e9-50a0-48c9-9934-613b96cf5998"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:13:08.082475 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.082255    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8068d378-073f-42f2-a59e-8e9aae3f51ba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8068d378-073f-42f2-a59e-8e9aae3f51ba" (UID: "8068d378-073f-42f2-a59e-8e9aae3f51ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:13:08.084231 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.084204    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/473f99e9-50a0-48c9-9934-613b96cf5998-kube-api-access-8sjrc" (OuterVolumeSpecName: "kube-api-access-8sjrc") pod "473f99e9-50a0-48c9-9934-613b96cf5998" (UID: "473f99e9-50a0-48c9-9934-613b96cf5998"). InnerVolumeSpecName "kube-api-access-8sjrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:13:08.084351 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.084240    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8068d378-073f-42f2-a59e-8e9aae3f51ba-kube-api-access-rh648" (OuterVolumeSpecName: "kube-api-access-rh648") pod "8068d378-073f-42f2-a59e-8e9aae3f51ba" (UID: "8068d378-073f-42f2-a59e-8e9aae3f51ba"). InnerVolumeSpecName "kube-api-access-rh648". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:13:08.182205 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.182179    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8068d378-073f-42f2-a59e-8e9aae3f51ba-plugins-config-volume\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:13:08.182205 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.182200    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sjrc\" (UniqueName: \"kubernetes.io/projected/473f99e9-50a0-48c9-9934-613b96cf5998-kube-api-access-8sjrc\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:13:08.182205 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.182209    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473f99e9-50a0-48c9-9934-613b96cf5998-plugins-config-volume\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:13:08.182429 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.182218    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh648\" (UniqueName: \"kubernetes.io/projected/8068d378-073f-42f2-a59e-8e9aae3f51ba-kube-api-access-rh648\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:13:08.184235 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.184207    2572 generic.go:356] "Generic (PLEG): container finished" podID="8068d378-073f-42f2-a59e-8e9aae3f51ba" containerID="0804750d486a444a6e0885e1d07708ba9e5475510aa95c77ed4aa60047dd039a" exitCode=0
Sep 10 21:13:08.184355 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.184246    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb"
Sep 10 21:13:08.184355 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.184291    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb" event={"ID":"8068d378-073f-42f2-a59e-8e9aae3f51ba","Type":"ContainerDied","Data":"0804750d486a444a6e0885e1d07708ba9e5475510aa95c77ed4aa60047dd039a"}
Sep 10 21:13:08.184355 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.184348    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb" event={"ID":"8068d378-073f-42f2-a59e-8e9aae3f51ba","Type":"ContainerDied","Data":"84e5044c73bf1163b235fb19cb932b5225ea17a752b350a372d0cb1910762eb1"}
Sep 10 21:13:08.184495 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.184371    2572 scope.go:117] "RemoveContainer" containerID="0804750d486a444a6e0885e1d07708ba9e5475510aa95c77ed4aa60047dd039a"
Sep 10 21:13:08.185649 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.185627    2572 generic.go:356] "Generic (PLEG): container finished" podID="473f99e9-50a0-48c9-9934-613b96cf5998" containerID="c710a0768b0901ac418b9d0e030259a6ea93e410e5e12fe7660cb8302a99275e" exitCode=0
Sep 10 21:13:08.185749 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.185660    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2" event={"ID":"473f99e9-50a0-48c9-9934-613b96cf5998","Type":"ContainerDied","Data":"c710a0768b0901ac418b9d0e030259a6ea93e410e5e12fe7660cb8302a99275e"}
Sep 10 21:13:08.185749 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.185680    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2"
Sep 10 21:13:08.185749 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.185709    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2" event={"ID":"473f99e9-50a0-48c9-9934-613b96cf5998","Type":"ContainerDied","Data":"22f1af82b51d531e1e66f69c192e0295f2c2668cb39512346df7c1270936afc9"}
Sep 10 21:13:08.193208 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.193191    2572 scope.go:117] "RemoveContainer" containerID="0804750d486a444a6e0885e1d07708ba9e5475510aa95c77ed4aa60047dd039a"
Sep 10 21:13:08.193532 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:13:08.193511    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0804750d486a444a6e0885e1d07708ba9e5475510aa95c77ed4aa60047dd039a\": container with ID starting with 0804750d486a444a6e0885e1d07708ba9e5475510aa95c77ed4aa60047dd039a not found: ID does not exist" containerID="0804750d486a444a6e0885e1d07708ba9e5475510aa95c77ed4aa60047dd039a"
Sep 10 21:13:08.193612 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.193541    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0804750d486a444a6e0885e1d07708ba9e5475510aa95c77ed4aa60047dd039a"} err="failed to get container status \"0804750d486a444a6e0885e1d07708ba9e5475510aa95c77ed4aa60047dd039a\": rpc error: code = NotFound desc = could not find container \"0804750d486a444a6e0885e1d07708ba9e5475510aa95c77ed4aa60047dd039a\": container with ID starting with 0804750d486a444a6e0885e1d07708ba9e5475510aa95c77ed4aa60047dd039a not found: ID does not exist"
Sep 10 21:13:08.193612 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.193556    2572 scope.go:117] "RemoveContainer" containerID="c710a0768b0901ac418b9d0e030259a6ea93e410e5e12fe7660cb8302a99275e"
Sep 10 21:13:08.201576 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.201558    2572 scope.go:117] "RemoveContainer" containerID="c710a0768b0901ac418b9d0e030259a6ea93e410e5e12fe7660cb8302a99275e"
Sep 10 21:13:08.201840 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:13:08.201824    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c710a0768b0901ac418b9d0e030259a6ea93e410e5e12fe7660cb8302a99275e\": container with ID starting with c710a0768b0901ac418b9d0e030259a6ea93e410e5e12fe7660cb8302a99275e not found: ID does not exist" containerID="c710a0768b0901ac418b9d0e030259a6ea93e410e5e12fe7660cb8302a99275e"
Sep 10 21:13:08.201905 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.201846    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c710a0768b0901ac418b9d0e030259a6ea93e410e5e12fe7660cb8302a99275e"} err="failed to get container status \"c710a0768b0901ac418b9d0e030259a6ea93e410e5e12fe7660cb8302a99275e\": rpc error: code = NotFound desc = could not find container \"c710a0768b0901ac418b9d0e030259a6ea93e410e5e12fe7660cb8302a99275e\": container with ID starting with c710a0768b0901ac418b9d0e030259a6ea93e410e5e12fe7660cb8302a99275e not found: ID does not exist"
Sep 10 21:13:08.217629 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.217605    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2"]
Sep 10 21:13:08.221432 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.221410    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c2de43-65fb855bc8-m2sr2"]
Sep 10 21:13:08.230967 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.230944    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb"]
Sep 10 21:13:08.236283 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:08.236262    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c2de43-dd55f969f-h6nvb"]
Sep 10 21:13:09.437724 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:09.437687    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="473f99e9-50a0-48c9-9934-613b96cf5998" path="/var/lib/kubelet/pods/473f99e9-50a0-48c9-9934-613b96cf5998/volumes"
Sep 10 21:13:09.438127 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:09.438069    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8068d378-073f-42f2-a59e-8e9aae3f51ba" path="/var/lib/kubelet/pods/8068d378-073f-42f2-a59e-8e9aae3f51ba/volumes"
Sep 10 21:13:11.628114 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:11.628083    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:13:13.312522 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:13.312485    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"]
Sep 10 21:13:15.168207 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:15.168164    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk"
Sep 10 21:13:15.169081 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:15.169054    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm"
Sep 10 21:13:26.200160 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.200087    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"]
Sep 10 21:13:26.200544 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.200527    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="473f99e9-50a0-48c9-9934-613b96cf5998" containerName="payload-pre-processing"
Sep 10 21:13:26.200588 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.200545    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="473f99e9-50a0-48c9-9934-613b96cf5998" containerName="payload-pre-processing"
Sep 10 21:13:26.200588 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.200561    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8068d378-073f-42f2-a59e-8e9aae3f51ba" containerName="payload-processing"
Sep 10 21:13:26.200588 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.200566    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8068d378-073f-42f2-a59e-8e9aae3f51ba" containerName="payload-processing"
Sep 10 21:13:26.200706 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.200632    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8068d378-073f-42f2-a59e-8e9aae3f51ba" containerName="payload-processing"
Sep 10 21:13:26.200706 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.200644    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="473f99e9-50a0-48c9-9934-613b96cf5998" containerName="payload-pre-processing"
Sep 10 21:13:26.203660 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.203642    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:13:26.206631 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.206610    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-04db70\"/\"default-dockercfg-x44qg\""
Sep 10 21:13:26.207052 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.207034    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-04db70\"/\"openshift-service-ca.crt\""
Sep 10 21:13:26.207456 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.207431    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-04db70\"/\"kube-root-ca.crt\""
Sep 10 21:13:26.207579 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.207489    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-04db70\"/\"test-model-04db70-kserve-self-signed-certs\""
Sep 10 21:13:26.215481 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.215460    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"]
Sep 10 21:13:26.316376 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.316355    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf556761-5303-4c4b-a91c-972065ca11b7-home\") pod \"test-model-04db70-kserve-86f7fbc76c-5kj45\" (UID: \"bf556761-5303-4c4b-a91c-972065ca11b7\") " pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:13:26.316489 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.316382    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf556761-5303-4c4b-a91c-972065ca11b7-tmp-dir\") pod \"test-model-04db70-kserve-86f7fbc76c-5kj45\" (UID: \"bf556761-5303-4c4b-a91c-972065ca11b7\") " pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:13:26.316489 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.316408    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r72x5\" (UniqueName: \"kubernetes.io/projected/bf556761-5303-4c4b-a91c-972065ca11b7-kube-api-access-r72x5\") pod \"test-model-04db70-kserve-86f7fbc76c-5kj45\" (UID: \"bf556761-5303-4c4b-a91c-972065ca11b7\") " pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:13:26.316590 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.316496    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf556761-5303-4c4b-a91c-972065ca11b7-dshm\") pod \"test-model-04db70-kserve-86f7fbc76c-5kj45\" (UID: \"bf556761-5303-4c4b-a91c-972065ca11b7\") " pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:13:26.316590 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.316531    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf556761-5303-4c4b-a91c-972065ca11b7-model-cache\") pod \"test-model-04db70-kserve-86f7fbc76c-5kj45\" (UID: \"bf556761-5303-4c4b-a91c-972065ca11b7\") " pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:13:26.316590 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.316558    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf556761-5303-4c4b-a91c-972065ca11b7-tls-certs\") pod \"test-model-04db70-kserve-86f7fbc76c-5kj45\" (UID: \"bf556761-5303-4c4b-a91c-972065ca11b7\") " pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:13:26.417251 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.417218    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf556761-5303-4c4b-a91c-972065ca11b7-tls-certs\") pod \"test-model-04db70-kserve-86f7fbc76c-5kj45\" (UID: \"bf556761-5303-4c4b-a91c-972065ca11b7\") " pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:13:26.417404 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.417386    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf556761-5303-4c4b-a91c-972065ca11b7-home\") pod \"test-model-04db70-kserve-86f7fbc76c-5kj45\" (UID: \"bf556761-5303-4c4b-a91c-972065ca11b7\") " pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:13:26.417520 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.417417    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf556761-5303-4c4b-a91c-972065ca11b7-tmp-dir\") pod \"test-model-04db70-kserve-86f7fbc76c-5kj45\" (UID: \"bf556761-5303-4c4b-a91c-972065ca11b7\") " pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:13:26.417520 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.417452    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r72x5\" (UniqueName: \"kubernetes.io/projected/bf556761-5303-4c4b-a91c-972065ca11b7-kube-api-access-r72x5\") pod \"test-model-04db70-kserve-86f7fbc76c-5kj45\" (UID: \"bf556761-5303-4c4b-a91c-972065ca11b7\") " pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:13:26.417520 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.417518    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf556761-5303-4c4b-a91c-972065ca11b7-dshm\") pod \"test-model-04db70-kserve-86f7fbc76c-5kj45\" (UID: \"bf556761-5303-4c4b-a91c-972065ca11b7\") " pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:13:26.417952 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.417555    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf556761-5303-4c4b-a91c-972065ca11b7-model-cache\") pod \"test-model-04db70-kserve-86f7fbc76c-5kj45\" (UID: \"bf556761-5303-4c4b-a91c-972065ca11b7\") " pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:13:26.419381 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.418957    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf556761-5303-4c4b-a91c-972065ca11b7-home\") pod \"test-model-04db70-kserve-86f7fbc76c-5kj45\" (UID: \"bf556761-5303-4c4b-a91c-972065ca11b7\") " pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:13:26.419381 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.419271    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf556761-5303-4c4b-a91c-972065ca11b7-tmp-dir\") pod \"test-model-04db70-kserve-86f7fbc76c-5kj45\" (UID: \"bf556761-5303-4c4b-a91c-972065ca11b7\") " pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:13:26.419381 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.419308    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf556761-5303-4c4b-a91c-972065ca11b7-model-cache\") pod \"test-model-04db70-kserve-86f7fbc76c-5kj45\" (UID: \"bf556761-5303-4c4b-a91c-972065ca11b7\") " pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:13:26.420432 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.420407    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf556761-5303-4c4b-a91c-972065ca11b7-tls-certs\") pod \"test-model-04db70-kserve-86f7fbc76c-5kj45\" (UID: \"bf556761-5303-4c4b-a91c-972065ca11b7\") " pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:13:26.421463 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.421444    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf556761-5303-4c4b-a91c-972065ca11b7-dshm\") pod \"test-model-04db70-kserve-86f7fbc76c-5kj45\" (UID: \"bf556761-5303-4c4b-a91c-972065ca11b7\") " pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:13:26.430397 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.430376    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r72x5\" (UniqueName: \"kubernetes.io/projected/bf556761-5303-4c4b-a91c-972065ca11b7-kube-api-access-r72x5\") pod \"test-model-04db70-kserve-86f7fbc76c-5kj45\" (UID: \"bf556761-5303-4c4b-a91c-972065ca11b7\") " pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:13:26.513641 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.513580    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:13:26.790390 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:26.790354    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"]
Sep 10 21:13:26.795801 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:13:26.795776    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf556761_5303_4c4b_a91c_972065ca11b7.slice/crio-b56db4ed592a6723bee9e71b6ec803d85f9dbdfd0ededf5e82cc55beb7f6ac70 WatchSource:0}: Error finding container b56db4ed592a6723bee9e71b6ec803d85f9dbdfd0ededf5e82cc55beb7f6ac70: Status 404 returned error can't find the container with id b56db4ed592a6723bee9e71b6ec803d85f9dbdfd0ededf5e82cc55beb7f6ac70
Sep 10 21:13:27.288800 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:27.288764    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45" event={"ID":"bf556761-5303-4c4b-a91c-972065ca11b7","Type":"ContainerStarted","Data":"99cfcaa15ace4a3a3144d432fe0359af094262f5b0546171bdf077e2401a1e8a"}
Sep 10 21:13:27.288800 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:27.288806    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45" event={"ID":"bf556761-5303-4c4b-a91c-972065ca11b7","Type":"ContainerStarted","Data":"b56db4ed592a6723bee9e71b6ec803d85f9dbdfd0ededf5e82cc55beb7f6ac70"}
Sep 10 21:13:27.307178 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:27.307132    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45" podStartSLOduration=1.307116676 podStartE2EDuration="1.307116676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:13:27.306189486 +0000 UTC m=+992.518654405" watchObservedRunningTime="2026-09-10 21:13:27.307116676 +0000 UTC m=+992.519581595"
Sep 10 21:13:28.332167 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:28.332103    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs" podUID="7379f436-2286-4446-a281-2f37237b3838" containerName="main" containerID="cri-o://19351ed1646950d842b33aee72876ce372f0195649a1046c312c64ec0654ba81" gracePeriod=45
Sep 10 21:13:28.480576 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:28.480543    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"]
Sep 10 21:13:28.610433 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:28.610411    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:13:28.638396 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:28.638370    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7379f436-2286-4446-a281-2f37237b3838-dshm\") pod \"7379f436-2286-4446-a281-2f37237b3838\" (UID: \"7379f436-2286-4446-a281-2f37237b3838\") "
Sep 10 21:13:28.638525 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:28.638406    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7379f436-2286-4446-a281-2f37237b3838-model-cache\") pod \"7379f436-2286-4446-a281-2f37237b3838\" (UID: \"7379f436-2286-4446-a281-2f37237b3838\") "
Sep 10 21:13:28.638525 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:28.638429    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7379f436-2286-4446-a281-2f37237b3838-tls-certs\") pod \"7379f436-2286-4446-a281-2f37237b3838\" (UID: \"7379f436-2286-4446-a281-2f37237b3838\") "
Sep 10 21:13:28.638525 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:28.638477    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7379f436-2286-4446-a281-2f37237b3838-tmp-dir\") pod \"7379f436-2286-4446-a281-2f37237b3838\" (UID: \"7379f436-2286-4446-a281-2f37237b3838\") "
Sep 10 21:13:28.638525 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:28.638502    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7379f436-2286-4446-a281-2f37237b3838-home\") pod \"7379f436-2286-4446-a281-2f37237b3838\" (UID: \"7379f436-2286-4446-a281-2f37237b3838\") "
Sep 10 21:13:28.638703 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:28.638597    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xwf7\" (UniqueName: \"kubernetes.io/projected/7379f436-2286-4446-a281-2f37237b3838-kube-api-access-8xwf7\") pod \"7379f436-2286-4446-a281-2f37237b3838\" (UID: \"7379f436-2286-4446-a281-2f37237b3838\") "
Sep 10 21:13:28.638703 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:28.638670    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7379f436-2286-4446-a281-2f37237b3838-model-cache" (OuterVolumeSpecName: "model-cache") pod "7379f436-2286-4446-a281-2f37237b3838" (UID: "7379f436-2286-4446-a281-2f37237b3838"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:28.639098 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:28.638916    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7379f436-2286-4446-a281-2f37237b3838-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7379f436-2286-4446-a281-2f37237b3838" (UID: "7379f436-2286-4446-a281-2f37237b3838"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:28.639098 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:28.638942    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7379f436-2286-4446-a281-2f37237b3838-model-cache\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:13:28.639268 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:28.639040    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7379f436-2286-4446-a281-2f37237b3838-home" (OuterVolumeSpecName: "home") pod "7379f436-2286-4446-a281-2f37237b3838" (UID: "7379f436-2286-4446-a281-2f37237b3838"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:28.640807 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:28.640783    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7379f436-2286-4446-a281-2f37237b3838-dshm" (OuterVolumeSpecName: "dshm") pod "7379f436-2286-4446-a281-2f37237b3838" (UID: "7379f436-2286-4446-a281-2f37237b3838"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:28.641327 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:28.641304    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7379f436-2286-4446-a281-2f37237b3838-kube-api-access-8xwf7" (OuterVolumeSpecName: "kube-api-access-8xwf7") pod "7379f436-2286-4446-a281-2f37237b3838" (UID: "7379f436-2286-4446-a281-2f37237b3838"). InnerVolumeSpecName "kube-api-access-8xwf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:13:28.641483 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:28.641382    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7379f436-2286-4446-a281-2f37237b3838-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7379f436-2286-4446-a281-2f37237b3838" (UID: "7379f436-2286-4446-a281-2f37237b3838"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:13:28.739645 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:28.739621    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7379f436-2286-4446-a281-2f37237b3838-dshm\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:13:28.739645 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:28.739645    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7379f436-2286-4446-a281-2f37237b3838-tls-certs\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:13:28.739797 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:28.739654    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7379f436-2286-4446-a281-2f37237b3838-tmp-dir\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:13:28.739797 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:28.739663    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7379f436-2286-4446-a281-2f37237b3838-home\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:13:28.739797 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:28.739671    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xwf7\" (UniqueName: \"kubernetes.io/projected/7379f436-2286-4446-a281-2f37237b3838-kube-api-access-8xwf7\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:13:29.301072 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:29.301042    2572 generic.go:356] "Generic (PLEG): container finished" podID="7379f436-2286-4446-a281-2f37237b3838" containerID="19351ed1646950d842b33aee72876ce372f0195649a1046c312c64ec0654ba81" exitCode=0
Sep 10 21:13:29.301233 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:29.301084    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"
Sep 10 21:13:29.301233 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:29.301101    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs" event={"ID":"7379f436-2286-4446-a281-2f37237b3838","Type":"ContainerDied","Data":"19351ed1646950d842b33aee72876ce372f0195649a1046c312c64ec0654ba81"}
Sep 10 21:13:29.301233 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:29.301145    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs" event={"ID":"7379f436-2286-4446-a281-2f37237b3838","Type":"ContainerDied","Data":"0d371a9f86afa58ad737d1b8df6e718e8a07cfc22cc0bb8561dbdd152211955b"}
Sep 10 21:13:29.301233 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:29.301169    2572 scope.go:117] "RemoveContainer" containerID="19351ed1646950d842b33aee72876ce372f0195649a1046c312c64ec0654ba81"
Sep 10 21:13:29.310925 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:29.310906    2572 scope.go:117] "RemoveContainer" containerID="19351ed1646950d842b33aee72876ce372f0195649a1046c312c64ec0654ba81"
Sep 10 21:13:29.311253 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:13:29.311231    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19351ed1646950d842b33aee72876ce372f0195649a1046c312c64ec0654ba81\": container with ID starting with 19351ed1646950d842b33aee72876ce372f0195649a1046c312c64ec0654ba81 not found: ID does not exist" containerID="19351ed1646950d842b33aee72876ce372f0195649a1046c312c64ec0654ba81"
Sep 10 21:13:29.311335 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:29.311261    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19351ed1646950d842b33aee72876ce372f0195649a1046c312c64ec0654ba81"} err="failed to get container status \"19351ed1646950d842b33aee72876ce372f0195649a1046c312c64ec0654ba81\": rpc error: code = NotFound desc = could not find container \"19351ed1646950d842b33aee72876ce372f0195649a1046c312c64ec0654ba81\": container with ID starting with 19351ed1646950d842b33aee72876ce372f0195649a1046c312c64ec0654ba81 not found: ID does not exist"
Sep 10 21:13:29.326455 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:29.326430    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"]
Sep 10 21:13:29.330328 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:29.330303    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-c2de43/autores-model-c2de43-kserve-56b4776879-xt6hs"]
Sep 10 21:13:29.439279 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:29.439254    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7379f436-2286-4446-a281-2f37237b3838" path="/var/lib/kubelet/pods/7379f436-2286-4446-a281-2f37237b3838/volumes"
Sep 10 21:13:32.140827 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:32.140784    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:13:33.092021 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:33.091964    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk"]
Sep 10 21:13:33.092256 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:33.092226    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk" podUID="19a30a98-83e5-4648-8aa5-52144bd8b090" containerName="payload-processing" containerID="cri-o://2a3a19923fd1305ed8d5c2dec41ae5ac10d1978d4a928ec86b2ffe5917f8158e" gracePeriod=30
Sep 10 21:13:33.113141 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:33.113115    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm"]
Sep 10 21:13:33.113333 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:33.113313    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm" podUID="7f5bf3e5-884e-45c7-962b-7b9c61da7c65" containerName="payload-pre-processing" containerID="cri-o://15c9b6b6f31b9a46cb22db1b5f88a4a85d3907ed2d420581e7a7fe456cc64a1e" gracePeriod=30
Sep 10 21:13:33.328629 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:33.328586    2572 generic.go:356] "Generic (PLEG): container finished" podID="19a30a98-83e5-4648-8aa5-52144bd8b090" containerID="2a3a19923fd1305ed8d5c2dec41ae5ac10d1978d4a928ec86b2ffe5917f8158e" exitCode=0
Sep 10 21:13:33.329017 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:33.328661    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk" event={"ID":"19a30a98-83e5-4648-8aa5-52144bd8b090","Type":"ContainerDied","Data":"2a3a19923fd1305ed8d5c2dec41ae5ac10d1978d4a928ec86b2ffe5917f8158e"}
Sep 10 21:13:33.330628 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:33.330592    2572 generic.go:356] "Generic (PLEG): container finished" podID="7f5bf3e5-884e-45c7-962b-7b9c61da7c65" containerID="15c9b6b6f31b9a46cb22db1b5f88a4a85d3907ed2d420581e7a7fe456cc64a1e" exitCode=0
Sep 10 21:13:33.330743 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:33.330637    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm" event={"ID":"7f5bf3e5-884e-45c7-962b-7b9c61da7c65","Type":"ContainerDied","Data":"15c9b6b6f31b9a46cb22db1b5f88a4a85d3907ed2d420581e7a7fe456cc64a1e"}
Sep 10 21:13:33.383920 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:33.383899    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk"
Sep 10 21:13:33.386665 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:33.386649    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm"
Sep 10 21:13:33.480391 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:33.480366    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ll5kd\" (UniqueName: \"kubernetes.io/projected/7f5bf3e5-884e-45c7-962b-7b9c61da7c65-kube-api-access-ll5kd\") pod \"7f5bf3e5-884e-45c7-962b-7b9c61da7c65\" (UID: \"7f5bf3e5-884e-45c7-962b-7b9c61da7c65\") "
Sep 10 21:13:33.480555 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:33.480438    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f5bf3e5-884e-45c7-962b-7b9c61da7c65-plugins-config-volume\") pod \"7f5bf3e5-884e-45c7-962b-7b9c61da7c65\" (UID: \"7f5bf3e5-884e-45c7-962b-7b9c61da7c65\") "
Sep 10 21:13:33.480555 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:33.480462    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whjqs\" (UniqueName: \"kubernetes.io/projected/19a30a98-83e5-4648-8aa5-52144bd8b090-kube-api-access-whjqs\") pod \"19a30a98-83e5-4648-8aa5-52144bd8b090\" (UID: \"19a30a98-83e5-4648-8aa5-52144bd8b090\") "
Sep 10 21:13:33.480555 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:33.480485    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a30a98-83e5-4648-8aa5-52144bd8b090-plugins-config-volume\") pod \"19a30a98-83e5-4648-8aa5-52144bd8b090\" (UID: \"19a30a98-83e5-4648-8aa5-52144bd8b090\") "
Sep 10 21:13:33.480822 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:33.480794    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f5bf3e5-884e-45c7-962b-7b9c61da7c65-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7f5bf3e5-884e-45c7-962b-7b9c61da7c65" (UID: "7f5bf3e5-884e-45c7-962b-7b9c61da7c65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:13:33.480918 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:33.480884    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a30a98-83e5-4648-8aa5-52144bd8b090-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "19a30a98-83e5-4648-8aa5-52144bd8b090" (UID: "19a30a98-83e5-4648-8aa5-52144bd8b090"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:13:33.482653 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:33.482630    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5bf3e5-884e-45c7-962b-7b9c61da7c65-kube-api-access-ll5kd" (OuterVolumeSpecName: "kube-api-access-ll5kd") pod "7f5bf3e5-884e-45c7-962b-7b9c61da7c65" (UID: "7f5bf3e5-884e-45c7-962b-7b9c61da7c65"). InnerVolumeSpecName "kube-api-access-ll5kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:13:33.482653 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:33.482637    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a30a98-83e5-4648-8aa5-52144bd8b090-kube-api-access-whjqs" (OuterVolumeSpecName: "kube-api-access-whjqs") pod "19a30a98-83e5-4648-8aa5-52144bd8b090" (UID: "19a30a98-83e5-4648-8aa5-52144bd8b090"). InnerVolumeSpecName "kube-api-access-whjqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:13:33.581236 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:33.581202    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f5bf3e5-884e-45c7-962b-7b9c61da7c65-plugins-config-volume\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:13:33.581236 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:33.581229    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whjqs\" (UniqueName: \"kubernetes.io/projected/19a30a98-83e5-4648-8aa5-52144bd8b090-kube-api-access-whjqs\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:13:33.581236 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:33.581238    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a30a98-83e5-4648-8aa5-52144bd8b090-plugins-config-volume\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:13:33.581444 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:33.581246    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll5kd\" (UniqueName: \"kubernetes.io/projected/7f5bf3e5-884e-45c7-962b-7b9c61da7c65-kube-api-access-ll5kd\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:13:34.336481 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:34.336454    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk"
Sep 10 21:13:34.336922 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:34.336483    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk" event={"ID":"19a30a98-83e5-4648-8aa5-52144bd8b090","Type":"ContainerDied","Data":"ac9633a924d02bc25387564cf083bab6a2b04838408874fc08944287bdb37800"}
Sep 10 21:13:34.336922 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:34.336548    2572 scope.go:117] "RemoveContainer" containerID="2a3a19923fd1305ed8d5c2dec41ae5ac10d1978d4a928ec86b2ffe5917f8158e"
Sep 10 21:13:34.337996 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:34.337926    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm" event={"ID":"7f5bf3e5-884e-45c7-962b-7b9c61da7c65","Type":"ContainerDied","Data":"0d460d9b9ee90a82d5308d1223e6ade9e4c91d665a261969794de4cfaee9ba04"}
Sep 10 21:13:34.337996 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:34.337951    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm"
Sep 10 21:13:34.350304 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:34.350251    2572 scope.go:117] "RemoveContainer" containerID="15c9b6b6f31b9a46cb22db1b5f88a4a85d3907ed2d420581e7a7fe456cc64a1e"
Sep 10 21:13:34.366349 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:34.366327    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk"]
Sep 10 21:13:34.371069 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:34.371049    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cbc740e6-6b7578bcfc-5qshk"]
Sep 10 21:13:34.385797 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:34.385768    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm"]
Sep 10 21:13:34.390279 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:34.390261    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cbc740e6-7f5454b658-zxzzm"]
Sep 10 21:13:35.438282 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:35.438250    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19a30a98-83e5-4648-8aa5-52144bd8b090" path="/var/lib/kubelet/pods/19a30a98-83e5-4648-8aa5-52144bd8b090/volumes"
Sep 10 21:13:35.438631 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:35.438584    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f5bf3e5-884e-45c7-962b-7b9c61da7c65" path="/var/lib/kubelet/pods/7f5bf3e5-884e-45c7-962b-7b9c61da7c65/volumes"
Sep 10 21:13:36.514074 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:36.514023    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:13:36.514467 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:36.514093    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:13:36.526491 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:36.526469    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:13:37.366629 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:37.366602    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:13:40.104119 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.104089    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"]
Sep 10 21:13:40.104536 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.104439    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19a30a98-83e5-4648-8aa5-52144bd8b090" containerName="payload-processing"
Sep 10 21:13:40.104536 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.104450    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="19a30a98-83e5-4648-8aa5-52144bd8b090" containerName="payload-processing"
Sep 10 21:13:40.104536 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.104459    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f5bf3e5-884e-45c7-962b-7b9c61da7c65" containerName="payload-pre-processing"
Sep 10 21:13:40.104536 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.104464    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f5bf3e5-884e-45c7-962b-7b9c61da7c65" containerName="payload-pre-processing"
Sep 10 21:13:40.104536 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.104479    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7379f436-2286-4446-a281-2f37237b3838" containerName="main"
Sep 10 21:13:40.104536 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.104484    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7379f436-2286-4446-a281-2f37237b3838" containerName="main"
Sep 10 21:13:40.104748 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.104542    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f5bf3e5-884e-45c7-962b-7b9c61da7c65" containerName="payload-pre-processing"
Sep 10 21:13:40.104748 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.104552    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7379f436-2286-4446-a281-2f37237b3838" containerName="main"
Sep 10 21:13:40.104748 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.104558    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="19a30a98-83e5-4648-8aa5-52144bd8b090" containerName="payload-processing"
Sep 10 21:13:40.108882 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.108865    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:13:40.112119 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.112095    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7eaf1f\"/\"test-model-7eaf1f-kserve-self-signed-certs\""
Sep 10 21:13:40.112119 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.112095    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7eaf1f\"/\"openshift-service-ca.crt\""
Sep 10 21:13:40.112363 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.112154    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7eaf1f\"/\"default-dockercfg-8htpv\""
Sep 10 21:13:40.112363 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.112155    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7eaf1f\"/\"kube-root-ca.crt\""
Sep 10 21:13:40.121320 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.121300    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"]
Sep 10 21:13:40.229423 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.229387    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5faa84-5402-40e6-b011-1f4cfe719278-tls-certs\") pod \"test-model-7eaf1f-kserve-86576fdb77-jxwvq\" (UID: \"3b5faa84-5402-40e6-b011-1f4cfe719278\") " pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:13:40.229566 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.229446    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b5faa84-5402-40e6-b011-1f4cfe719278-dshm\") pod \"test-model-7eaf1f-kserve-86576fdb77-jxwvq\" (UID: \"3b5faa84-5402-40e6-b011-1f4cfe719278\") " pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:13:40.229566 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.229473    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b5faa84-5402-40e6-b011-1f4cfe719278-home\") pod \"test-model-7eaf1f-kserve-86576fdb77-jxwvq\" (UID: \"3b5faa84-5402-40e6-b011-1f4cfe719278\") " pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:13:40.229681 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.229582    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b5faa84-5402-40e6-b011-1f4cfe719278-model-cache\") pod \"test-model-7eaf1f-kserve-86576fdb77-jxwvq\" (UID: \"3b5faa84-5402-40e6-b011-1f4cfe719278\") " pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:13:40.229681 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.229611    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns24t\" (UniqueName: \"kubernetes.io/projected/3b5faa84-5402-40e6-b011-1f4cfe719278-kube-api-access-ns24t\") pod \"test-model-7eaf1f-kserve-86576fdb77-jxwvq\" (UID: \"3b5faa84-5402-40e6-b011-1f4cfe719278\") " pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:13:40.229681 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.229641    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b5faa84-5402-40e6-b011-1f4cfe719278-tmp-dir\") pod \"test-model-7eaf1f-kserve-86576fdb77-jxwvq\" (UID: \"3b5faa84-5402-40e6-b011-1f4cfe719278\") " pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:13:40.330669 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.330631    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b5faa84-5402-40e6-b011-1f4cfe719278-dshm\") pod \"test-model-7eaf1f-kserve-86576fdb77-jxwvq\" (UID: \"3b5faa84-5402-40e6-b011-1f4cfe719278\") " pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:13:40.330669 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.330672    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b5faa84-5402-40e6-b011-1f4cfe719278-home\") pod \"test-model-7eaf1f-kserve-86576fdb77-jxwvq\" (UID: \"3b5faa84-5402-40e6-b011-1f4cfe719278\") " pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:13:40.330843 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.330765    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b5faa84-5402-40e6-b011-1f4cfe719278-model-cache\") pod \"test-model-7eaf1f-kserve-86576fdb77-jxwvq\" (UID: \"3b5faa84-5402-40e6-b011-1f4cfe719278\") " pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:13:40.330843 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.330794    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns24t\" (UniqueName: \"kubernetes.io/projected/3b5faa84-5402-40e6-b011-1f4cfe719278-kube-api-access-ns24t\") pod \"test-model-7eaf1f-kserve-86576fdb77-jxwvq\" (UID: \"3b5faa84-5402-40e6-b011-1f4cfe719278\") " pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:13:40.330843 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.330823    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b5faa84-5402-40e6-b011-1f4cfe719278-tmp-dir\") pod \"test-model-7eaf1f-kserve-86576fdb77-jxwvq\" (UID: \"3b5faa84-5402-40e6-b011-1f4cfe719278\") " pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:13:40.331020 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.330858    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5faa84-5402-40e6-b011-1f4cfe719278-tls-certs\") pod \"test-model-7eaf1f-kserve-86576fdb77-jxwvq\" (UID: \"3b5faa84-5402-40e6-b011-1f4cfe719278\") " pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:13:40.331209 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.331187    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b5faa84-5402-40e6-b011-1f4cfe719278-home\") pod \"test-model-7eaf1f-kserve-86576fdb77-jxwvq\" (UID: \"3b5faa84-5402-40e6-b011-1f4cfe719278\") " pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:13:40.331280 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.331210    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b5faa84-5402-40e6-b011-1f4cfe719278-tmp-dir\") pod \"test-model-7eaf1f-kserve-86576fdb77-jxwvq\" (UID: \"3b5faa84-5402-40e6-b011-1f4cfe719278\") " pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:13:40.331280 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.331249    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b5faa84-5402-40e6-b011-1f4cfe719278-model-cache\") pod \"test-model-7eaf1f-kserve-86576fdb77-jxwvq\" (UID: \"3b5faa84-5402-40e6-b011-1f4cfe719278\") " pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:13:40.333106 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.333084    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b5faa84-5402-40e6-b011-1f4cfe719278-dshm\") pod \"test-model-7eaf1f-kserve-86576fdb77-jxwvq\" (UID: \"3b5faa84-5402-40e6-b011-1f4cfe719278\") " pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:13:40.333368 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.333352    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5faa84-5402-40e6-b011-1f4cfe719278-tls-certs\") pod \"test-model-7eaf1f-kserve-86576fdb77-jxwvq\" (UID: \"3b5faa84-5402-40e6-b011-1f4cfe719278\") " pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:13:40.340841 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.340821    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns24t\" (UniqueName: \"kubernetes.io/projected/3b5faa84-5402-40e6-b011-1f4cfe719278-kube-api-access-ns24t\") pod \"test-model-7eaf1f-kserve-86576fdb77-jxwvq\" (UID: \"3b5faa84-5402-40e6-b011-1f4cfe719278\") " pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:13:40.417766 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.417715    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:13:40.690999 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:40.690952    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"]
Sep 10 21:13:40.693091 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:13:40.693056    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b5faa84_5402_40e6_b011_1f4cfe719278.slice/crio-b3a1d7c23501dd43de6b3714966bc405de4c98131640e746b393c9287c69265f WatchSource:0}: Error finding container b3a1d7c23501dd43de6b3714966bc405de4c98131640e746b393c9287c69265f: Status 404 returned error can't find the container with id b3a1d7c23501dd43de6b3714966bc405de4c98131640e746b393c9287c69265f
Sep 10 21:13:41.376467 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:41.376426    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq" event={"ID":"3b5faa84-5402-40e6-b011-1f4cfe719278","Type":"ContainerStarted","Data":"fda6380378f36f14db86b11eb609c0ce5241db545d5a32338b7f1a82028ced6b"}
Sep 10 21:13:41.376891 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:41.376471    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq" event={"ID":"3b5faa84-5402-40e6-b011-1f4cfe719278","Type":"ContainerStarted","Data":"b3a1d7c23501dd43de6b3714966bc405de4c98131640e746b393c9287c69265f"}
Sep 10 21:13:41.403709 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:41.403666    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq" podStartSLOduration=1.40365098 podStartE2EDuration="1.40365098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:13:41.40142029 +0000 UTC m=+1006.613885209" watchObservedRunningTime="2026-09-10 21:13:41.40365098 +0000 UTC m=+1006.616115899"
Sep 10 21:13:43.501832 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:43.501800    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz" podUID="45b392a1-bb1b-4f97-a2f4-ccad9b6575ad" containerName="main" containerID="cri-o://053732e9e0f79b5bf49d91de23bfffb6b893c324a0cd921d34a039a606426395" gracePeriod=45
Sep 10 21:13:43.752104 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:43.752046    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:13:43.863076 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:43.863051    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-dshm\") pod \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\" (UID: \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\") "
Sep 10 21:13:43.863203 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:43.863080    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9btr8\" (UniqueName: \"kubernetes.io/projected/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-kube-api-access-9btr8\") pod \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\" (UID: \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\") "
Sep 10 21:13:43.863203 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:43.863100    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-tls-certs\") pod \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\" (UID: \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\") "
Sep 10 21:13:43.863203 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:43.863130    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-model-cache\") pod \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\" (UID: \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\") "
Sep 10 21:13:43.863203 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:43.863202    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-tmp-dir\") pod \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\" (UID: \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\") "
Sep 10 21:13:43.863418 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:43.863248    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-home\") pod \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\" (UID: \"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad\") "
Sep 10 21:13:43.863821 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:43.863790    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "45b392a1-bb1b-4f97-a2f4-ccad9b6575ad" (UID: "45b392a1-bb1b-4f97-a2f4-ccad9b6575ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:43.863971 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:43.863805    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-home" (OuterVolumeSpecName: "home") pod "45b392a1-bb1b-4f97-a2f4-ccad9b6575ad" (UID: "45b392a1-bb1b-4f97-a2f4-ccad9b6575ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:43.863971 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:43.863826    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "45b392a1-bb1b-4f97-a2f4-ccad9b6575ad" (UID: "45b392a1-bb1b-4f97-a2f4-ccad9b6575ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:43.866158 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:43.866085    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-model-cache\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:13:43.866158 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:43.866113    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-tmp-dir\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:13:43.866158 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:43.866135    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-home\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:13:43.868032 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:43.866530    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-kube-api-access-9btr8" (OuterVolumeSpecName: "kube-api-access-9btr8") pod "45b392a1-bb1b-4f97-a2f4-ccad9b6575ad" (UID: "45b392a1-bb1b-4f97-a2f4-ccad9b6575ad"). InnerVolumeSpecName "kube-api-access-9btr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:13:43.868032 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:43.866713    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45b392a1-bb1b-4f97-a2f4-ccad9b6575ad" (UID: "45b392a1-bb1b-4f97-a2f4-ccad9b6575ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:13:43.871872 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:43.871845    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-dshm" (OuterVolumeSpecName: "dshm") pod "45b392a1-bb1b-4f97-a2f4-ccad9b6575ad" (UID: "45b392a1-bb1b-4f97-a2f4-ccad9b6575ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:43.966931 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:43.966910    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-dshm\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:13:43.966931 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:43.966931    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9btr8\" (UniqueName: \"kubernetes.io/projected/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-kube-api-access-9btr8\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:13:43.967081 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:43.966941    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad-tls-certs\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:13:44.394529 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:44.394497    2572 generic.go:356] "Generic (PLEG): container finished" podID="45b392a1-bb1b-4f97-a2f4-ccad9b6575ad" containerID="053732e9e0f79b5bf49d91de23bfffb6b893c324a0cd921d34a039a606426395" exitCode=0
Sep 10 21:13:44.394657 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:44.394565    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"
Sep 10 21:13:44.394657 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:44.394587    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz" event={"ID":"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad","Type":"ContainerDied","Data":"053732e9e0f79b5bf49d91de23bfffb6b893c324a0cd921d34a039a606426395"}
Sep 10 21:13:44.394657 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:44.394639    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz" event={"ID":"45b392a1-bb1b-4f97-a2f4-ccad9b6575ad","Type":"ContainerDied","Data":"05065f4b9a0eaec0f0842f1b5f1fd0830069b1a6d0922e52cc29d3bc3d2fbc9c"}
Sep 10 21:13:44.394657 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:44.394658    2572 scope.go:117] "RemoveContainer" containerID="053732e9e0f79b5bf49d91de23bfffb6b893c324a0cd921d34a039a606426395"
Sep 10 21:13:44.404761 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:44.404744    2572 scope.go:117] "RemoveContainer" containerID="053732e9e0f79b5bf49d91de23bfffb6b893c324a0cd921d34a039a606426395"
Sep 10 21:13:44.405086 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:13:44.405058    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"053732e9e0f79b5bf49d91de23bfffb6b893c324a0cd921d34a039a606426395\": container with ID starting with 053732e9e0f79b5bf49d91de23bfffb6b893c324a0cd921d34a039a606426395 not found: ID does not exist" containerID="053732e9e0f79b5bf49d91de23bfffb6b893c324a0cd921d34a039a606426395"
Sep 10 21:13:44.405200 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:44.405092    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"053732e9e0f79b5bf49d91de23bfffb6b893c324a0cd921d34a039a606426395"} err="failed to get container status \"053732e9e0f79b5bf49d91de23bfffb6b893c324a0cd921d34a039a606426395\": rpc error: code = NotFound desc = could not find container \"053732e9e0f79b5bf49d91de23bfffb6b893c324a0cd921d34a039a606426395\": container with ID starting with 053732e9e0f79b5bf49d91de23bfffb6b893c324a0cd921d34a039a606426395 not found: ID does not exist"
Sep 10 21:13:44.416715 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:44.416688    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"]
Sep 10 21:13:44.420773 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:44.420751    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-43415b1a-kserve-79c64bcdd8-n4pdz"]
Sep 10 21:13:45.438097 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:45.438069    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45b392a1-bb1b-4f97-a2f4-ccad9b6575ad" path="/var/lib/kubelet/pods/45b392a1-bb1b-4f97-a2f4-ccad9b6575ad/volumes"
Sep 10 21:13:47.098676 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:47.098645    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:13:50.418097 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:50.418044    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:13:50.418097 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:50.418114    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:13:50.431386 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:50.431357    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:13:50.442778 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:13:50.442747    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:14:05.755276 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:05.755244    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:14:23.985954 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:23.985916    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"]
Sep 10 21:14:23.986362 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:23.986316    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45b392a1-bb1b-4f97-a2f4-ccad9b6575ad" containerName="main"
Sep 10 21:14:23.986362 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:23.986330    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="45b392a1-bb1b-4f97-a2f4-ccad9b6575ad" containerName="main"
Sep 10 21:14:23.986444 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:23.986417    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="45b392a1-bb1b-4f97-a2f4-ccad9b6575ad" containerName="main"
Sep 10 21:14:23.989374 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:23.989354    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:23.991714 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:23.991685    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-65af3d\"/\"openshift-service-ca.crt\""
Sep 10 21:14:23.992564 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:23.992539    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-65af3d\"/\"kube-root-ca.crt\""
Sep 10 21:14:23.992716 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:23.992596    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-65af3d\"/\"explicit-model-65af3d-kserve-self-signed-certs\""
Sep 10 21:14:23.992716 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:23.992597    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-65af3d\"/\"default-dockercfg-4qvtl\""
Sep 10 21:14:24.000138 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:24.000112    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"]
Sep 10 21:14:24.078168 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:24.078145    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16700be5-7584-4b07-9fe5-b1755a532e47-dshm\") pod \"explicit-model-65af3d-kserve-8b849f667-lsq5p\" (UID: \"16700be5-7584-4b07-9fe5-b1755a532e47\") " pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:24.078291 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:24.078178    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16700be5-7584-4b07-9fe5-b1755a532e47-model-cache\") pod \"explicit-model-65af3d-kserve-8b849f667-lsq5p\" (UID: \"16700be5-7584-4b07-9fe5-b1755a532e47\") " pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:24.078291 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:24.078223    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssz4r\" (UniqueName: \"kubernetes.io/projected/16700be5-7584-4b07-9fe5-b1755a532e47-kube-api-access-ssz4r\") pod \"explicit-model-65af3d-kserve-8b849f667-lsq5p\" (UID: \"16700be5-7584-4b07-9fe5-b1755a532e47\") " pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:24.078383 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:24.078289    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16700be5-7584-4b07-9fe5-b1755a532e47-tls-certs\") pod \"explicit-model-65af3d-kserve-8b849f667-lsq5p\" (UID: \"16700be5-7584-4b07-9fe5-b1755a532e47\") " pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:24.078383 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:24.078333    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16700be5-7584-4b07-9fe5-b1755a532e47-home\") pod \"explicit-model-65af3d-kserve-8b849f667-lsq5p\" (UID: \"16700be5-7584-4b07-9fe5-b1755a532e47\") " pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:24.078383 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:24.078350    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16700be5-7584-4b07-9fe5-b1755a532e47-tmp-dir\") pod \"explicit-model-65af3d-kserve-8b849f667-lsq5p\" (UID: \"16700be5-7584-4b07-9fe5-b1755a532e47\") " pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:24.178967 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:24.178935    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16700be5-7584-4b07-9fe5-b1755a532e47-tls-certs\") pod \"explicit-model-65af3d-kserve-8b849f667-lsq5p\" (UID: \"16700be5-7584-4b07-9fe5-b1755a532e47\") " pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:24.179132 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:24.179004    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16700be5-7584-4b07-9fe5-b1755a532e47-home\") pod \"explicit-model-65af3d-kserve-8b849f667-lsq5p\" (UID: \"16700be5-7584-4b07-9fe5-b1755a532e47\") " pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:24.179132 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:24.179020    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16700be5-7584-4b07-9fe5-b1755a532e47-tmp-dir\") pod \"explicit-model-65af3d-kserve-8b849f667-lsq5p\" (UID: \"16700be5-7584-4b07-9fe5-b1755a532e47\") " pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:24.179132 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:24.179072    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16700be5-7584-4b07-9fe5-b1755a532e47-dshm\") pod \"explicit-model-65af3d-kserve-8b849f667-lsq5p\" (UID: \"16700be5-7584-4b07-9fe5-b1755a532e47\") " pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:24.179132 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:24.179088    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16700be5-7584-4b07-9fe5-b1755a532e47-model-cache\") pod \"explicit-model-65af3d-kserve-8b849f667-lsq5p\" (UID: \"16700be5-7584-4b07-9fe5-b1755a532e47\") " pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:24.179132 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:24.179121    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssz4r\" (UniqueName: \"kubernetes.io/projected/16700be5-7584-4b07-9fe5-b1755a532e47-kube-api-access-ssz4r\") pod \"explicit-model-65af3d-kserve-8b849f667-lsq5p\" (UID: \"16700be5-7584-4b07-9fe5-b1755a532e47\") " pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:24.179570 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:24.179537    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16700be5-7584-4b07-9fe5-b1755a532e47-model-cache\") pod \"explicit-model-65af3d-kserve-8b849f667-lsq5p\" (UID: \"16700be5-7584-4b07-9fe5-b1755a532e47\") " pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:24.179570 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:24.179559    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16700be5-7584-4b07-9fe5-b1755a532e47-tmp-dir\") pod \"explicit-model-65af3d-kserve-8b849f667-lsq5p\" (UID: \"16700be5-7584-4b07-9fe5-b1755a532e47\") " pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:24.179708 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:24.179615    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16700be5-7584-4b07-9fe5-b1755a532e47-home\") pod \"explicit-model-65af3d-kserve-8b849f667-lsq5p\" (UID: \"16700be5-7584-4b07-9fe5-b1755a532e47\") " pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:24.181521 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:24.181491    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16700be5-7584-4b07-9fe5-b1755a532e47-dshm\") pod \"explicit-model-65af3d-kserve-8b849f667-lsq5p\" (UID: \"16700be5-7584-4b07-9fe5-b1755a532e47\") " pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:24.181679 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:24.181659    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16700be5-7584-4b07-9fe5-b1755a532e47-tls-certs\") pod \"explicit-model-65af3d-kserve-8b849f667-lsq5p\" (UID: \"16700be5-7584-4b07-9fe5-b1755a532e47\") " pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:24.187448 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:24.187420    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssz4r\" (UniqueName: \"kubernetes.io/projected/16700be5-7584-4b07-9fe5-b1755a532e47-kube-api-access-ssz4r\") pod \"explicit-model-65af3d-kserve-8b849f667-lsq5p\" (UID: \"16700be5-7584-4b07-9fe5-b1755a532e47\") " pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:24.301677 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:24.301658    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:24.426540 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:24.426509    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:14:24.574349 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:24.574327    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"]
Sep 10 21:14:24.576661 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:14:24.576638    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16700be5_7584_4b07_9fe5_b1755a532e47.slice/crio-2a3cfbed7005fc61a20fd267daef069fcd633f2911f85982d37b109cd90cc16b WatchSource:0}: Error finding container 2a3cfbed7005fc61a20fd267daef069fcd633f2911f85982d37b109cd90cc16b: Status 404 returned error can't find the container with id 2a3cfbed7005fc61a20fd267daef069fcd633f2911f85982d37b109cd90cc16b
Sep 10 21:14:24.599437 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:24.599412    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p" event={"ID":"16700be5-7584-4b07-9fe5-b1755a532e47","Type":"ContainerStarted","Data":"2a3cfbed7005fc61a20fd267daef069fcd633f2911f85982d37b109cd90cc16b"}
Sep 10 21:14:25.605127 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:25.605095    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p" event={"ID":"16700be5-7584-4b07-9fe5-b1755a532e47","Type":"ContainerStarted","Data":"91d776ece268cdf98182a7cd948211ab987e89923d5e3a5b09ef5c8fc14dd118"}
Sep 10 21:14:25.625751 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:25.625704    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p" podStartSLOduration=2.625689708 podStartE2EDuration="2.625689708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:14:25.624867737 +0000 UTC m=+1050.837332656" watchObservedRunningTime="2026-09-10 21:14:25.625689708 +0000 UTC m=+1050.838154627"
Sep 10 21:14:34.302308 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:34.302270    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:34.302692 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:34.302331    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:34.314714 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:34.314690    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:34.659131 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:34.659104    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:35.993650 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:35.993620    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"]
Sep 10 21:14:40.960517 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:40.960483    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:14:45.857598 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:45.857566    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"]
Sep 10 21:14:46.830828 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:46.830797    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"]
Sep 10 21:14:51.677100 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:51.677000    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p" podUID="16700be5-7584-4b07-9fe5-b1755a532e47" containerName="main" containerID="cri-o://91d776ece268cdf98182a7cd948211ab987e89923d5e3a5b09ef5c8fc14dd118" gracePeriod=45
Sep 10 21:14:51.927488 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:51.927438    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:52.022463 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.022440    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssz4r\" (UniqueName: \"kubernetes.io/projected/16700be5-7584-4b07-9fe5-b1755a532e47-kube-api-access-ssz4r\") pod \"16700be5-7584-4b07-9fe5-b1755a532e47\" (UID: \"16700be5-7584-4b07-9fe5-b1755a532e47\") "
Sep 10 21:14:52.022609 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.022473    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16700be5-7584-4b07-9fe5-b1755a532e47-tmp-dir\") pod \"16700be5-7584-4b07-9fe5-b1755a532e47\" (UID: \"16700be5-7584-4b07-9fe5-b1755a532e47\") "
Sep 10 21:14:52.022609 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.022498    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16700be5-7584-4b07-9fe5-b1755a532e47-dshm\") pod \"16700be5-7584-4b07-9fe5-b1755a532e47\" (UID: \"16700be5-7584-4b07-9fe5-b1755a532e47\") "
Sep 10 21:14:52.022609 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.022516    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16700be5-7584-4b07-9fe5-b1755a532e47-model-cache\") pod \"16700be5-7584-4b07-9fe5-b1755a532e47\" (UID: \"16700be5-7584-4b07-9fe5-b1755a532e47\") "
Sep 10 21:14:52.022609 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.022540    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16700be5-7584-4b07-9fe5-b1755a532e47-home\") pod \"16700be5-7584-4b07-9fe5-b1755a532e47\" (UID: \"16700be5-7584-4b07-9fe5-b1755a532e47\") "
Sep 10 21:14:52.022609 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.022556    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16700be5-7584-4b07-9fe5-b1755a532e47-tls-certs\") pod \"16700be5-7584-4b07-9fe5-b1755a532e47\" (UID: \"16700be5-7584-4b07-9fe5-b1755a532e47\") "
Sep 10 21:14:52.022854 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.022757    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16700be5-7584-4b07-9fe5-b1755a532e47-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "16700be5-7584-4b07-9fe5-b1755a532e47" (UID: "16700be5-7584-4b07-9fe5-b1755a532e47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:52.022854 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.022773    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16700be5-7584-4b07-9fe5-b1755a532e47-model-cache" (OuterVolumeSpecName: "model-cache") pod "16700be5-7584-4b07-9fe5-b1755a532e47" (UID: "16700be5-7584-4b07-9fe5-b1755a532e47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:52.022854 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.022789    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16700be5-7584-4b07-9fe5-b1755a532e47-home" (OuterVolumeSpecName: "home") pod "16700be5-7584-4b07-9fe5-b1755a532e47" (UID: "16700be5-7584-4b07-9fe5-b1755a532e47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:52.024899 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.024873    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16700be5-7584-4b07-9fe5-b1755a532e47-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "16700be5-7584-4b07-9fe5-b1755a532e47" (UID: "16700be5-7584-4b07-9fe5-b1755a532e47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:14:52.025025 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.024922    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16700be5-7584-4b07-9fe5-b1755a532e47-dshm" (OuterVolumeSpecName: "dshm") pod "16700be5-7584-4b07-9fe5-b1755a532e47" (UID: "16700be5-7584-4b07-9fe5-b1755a532e47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:52.025025 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.024925    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16700be5-7584-4b07-9fe5-b1755a532e47-kube-api-access-ssz4r" (OuterVolumeSpecName: "kube-api-access-ssz4r") pod "16700be5-7584-4b07-9fe5-b1755a532e47" (UID: "16700be5-7584-4b07-9fe5-b1755a532e47"). InnerVolumeSpecName "kube-api-access-ssz4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:14:52.123576 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.123557    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16700be5-7584-4b07-9fe5-b1755a532e47-tls-certs\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:14:52.123576 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.123575    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssz4r\" (UniqueName: \"kubernetes.io/projected/16700be5-7584-4b07-9fe5-b1755a532e47-kube-api-access-ssz4r\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:14:52.123704 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.123584    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16700be5-7584-4b07-9fe5-b1755a532e47-tmp-dir\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:14:52.123704 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.123593    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16700be5-7584-4b07-9fe5-b1755a532e47-dshm\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:14:52.123704 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.123601    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16700be5-7584-4b07-9fe5-b1755a532e47-model-cache\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:14:52.123704 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.123608    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16700be5-7584-4b07-9fe5-b1755a532e47-home\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:14:52.744377 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.744341    2572 generic.go:356] "Generic (PLEG): container finished" podID="16700be5-7584-4b07-9fe5-b1755a532e47" containerID="91d776ece268cdf98182a7cd948211ab987e89923d5e3a5b09ef5c8fc14dd118" exitCode=0
Sep 10 21:14:52.744826 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.744393    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"
Sep 10 21:14:52.744826 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.744423    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p" event={"ID":"16700be5-7584-4b07-9fe5-b1755a532e47","Type":"ContainerDied","Data":"91d776ece268cdf98182a7cd948211ab987e89923d5e3a5b09ef5c8fc14dd118"}
Sep 10 21:14:52.744826 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.744482    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p" event={"ID":"16700be5-7584-4b07-9fe5-b1755a532e47","Type":"ContainerDied","Data":"2a3cfbed7005fc61a20fd267daef069fcd633f2911f85982d37b109cd90cc16b"}
Sep 10 21:14:52.744826 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.744504    2572 scope.go:117] "RemoveContainer" containerID="91d776ece268cdf98182a7cd948211ab987e89923d5e3a5b09ef5c8fc14dd118"
Sep 10 21:14:52.754500 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.754479    2572 scope.go:117] "RemoveContainer" containerID="91d776ece268cdf98182a7cd948211ab987e89923d5e3a5b09ef5c8fc14dd118"
Sep 10 21:14:52.754805 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:14:52.754783    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d776ece268cdf98182a7cd948211ab987e89923d5e3a5b09ef5c8fc14dd118\": container with ID starting with 91d776ece268cdf98182a7cd948211ab987e89923d5e3a5b09ef5c8fc14dd118 not found: ID does not exist" containerID="91d776ece268cdf98182a7cd948211ab987e89923d5e3a5b09ef5c8fc14dd118"
Sep 10 21:14:52.754882 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.754818    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d776ece268cdf98182a7cd948211ab987e89923d5e3a5b09ef5c8fc14dd118"} err="failed to get container status \"91d776ece268cdf98182a7cd948211ab987e89923d5e3a5b09ef5c8fc14dd118\": rpc error: code = NotFound desc = could not find container \"91d776ece268cdf98182a7cd948211ab987e89923d5e3a5b09ef5c8fc14dd118\": container with ID starting with 91d776ece268cdf98182a7cd948211ab987e89923d5e3a5b09ef5c8fc14dd118 not found: ID does not exist"
Sep 10 21:14:52.765534 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.765486    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"]
Sep 10 21:14:52.767962 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:52.767939    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-65af3d/explicit-model-65af3d-kserve-8b849f667-lsq5p"]
Sep 10 21:14:53.439183 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:53.439153    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16700be5-7584-4b07-9fe5-b1755a532e47" path="/var/lib/kubelet/pods/16700be5-7584-4b07-9fe5-b1755a532e47/volumes"
Sep 10 21:14:55.751088 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:55.751061    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:14:57.830943 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:57.830908    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw"]
Sep 10 21:14:57.831356 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:57.831162    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw" podUID="0e80bd28-a8f4-4f72-a45f-9794ac6b3643" containerName="payload-processing" containerID="cri-o://751efde4d95e28efa4d1ac02602b57ec14dca08c6f799b6742bef56f1c3db65e" gracePeriod=30
Sep 10 21:14:58.083274 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:58.083207    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw"
Sep 10 21:14:58.170708 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:58.170683    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58jrv\" (UniqueName: \"kubernetes.io/projected/0e80bd28-a8f4-4f72-a45f-9794ac6b3643-kube-api-access-58jrv\") pod \"0e80bd28-a8f4-4f72-a45f-9794ac6b3643\" (UID: \"0e80bd28-a8f4-4f72-a45f-9794ac6b3643\") "
Sep 10 21:14:58.170871 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:58.170728    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e80bd28-a8f4-4f72-a45f-9794ac6b3643-plugins-config-volume\") pod \"0e80bd28-a8f4-4f72-a45f-9794ac6b3643\" (UID: \"0e80bd28-a8f4-4f72-a45f-9794ac6b3643\") "
Sep 10 21:14:58.171097 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:58.171072    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e80bd28-a8f4-4f72-a45f-9794ac6b3643-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0e80bd28-a8f4-4f72-a45f-9794ac6b3643" (UID: "0e80bd28-a8f4-4f72-a45f-9794ac6b3643"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:14:58.172856 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:58.172833    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e80bd28-a8f4-4f72-a45f-9794ac6b3643-kube-api-access-58jrv" (OuterVolumeSpecName: "kube-api-access-58jrv") pod "0e80bd28-a8f4-4f72-a45f-9794ac6b3643" (UID: "0e80bd28-a8f4-4f72-a45f-9794ac6b3643"). InnerVolumeSpecName "kube-api-access-58jrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:14:58.271551 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:58.271526    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58jrv\" (UniqueName: \"kubernetes.io/projected/0e80bd28-a8f4-4f72-a45f-9794ac6b3643-kube-api-access-58jrv\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:14:58.271551 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:58.271548    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e80bd28-a8f4-4f72-a45f-9794ac6b3643-plugins-config-volume\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:14:58.774122 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:58.774092    2572 generic.go:356] "Generic (PLEG): container finished" podID="0e80bd28-a8f4-4f72-a45f-9794ac6b3643" containerID="751efde4d95e28efa4d1ac02602b57ec14dca08c6f799b6742bef56f1c3db65e" exitCode=0
Sep 10 21:14:58.774285 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:58.774150    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw"
Sep 10 21:14:58.774285 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:58.774188    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw" event={"ID":"0e80bd28-a8f4-4f72-a45f-9794ac6b3643","Type":"ContainerDied","Data":"751efde4d95e28efa4d1ac02602b57ec14dca08c6f799b6742bef56f1c3db65e"}
Sep 10 21:14:58.774285 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:58.774247    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw" event={"ID":"0e80bd28-a8f4-4f72-a45f-9794ac6b3643","Type":"ContainerDied","Data":"86de7ce3107c4bb472ce2e1a30c864b27b796131a204a7f8c923c1a9cbc49086"}
Sep 10 21:14:58.774285 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:58.774279    2572 scope.go:117] "RemoveContainer" containerID="751efde4d95e28efa4d1ac02602b57ec14dca08c6f799b6742bef56f1c3db65e"
Sep 10 21:14:58.783542 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:58.783524    2572 scope.go:117] "RemoveContainer" containerID="751efde4d95e28efa4d1ac02602b57ec14dca08c6f799b6742bef56f1c3db65e"
Sep 10 21:14:58.783828 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:14:58.783808    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"751efde4d95e28efa4d1ac02602b57ec14dca08c6f799b6742bef56f1c3db65e\": container with ID starting with 751efde4d95e28efa4d1ac02602b57ec14dca08c6f799b6742bef56f1c3db65e not found: ID does not exist" containerID="751efde4d95e28efa4d1ac02602b57ec14dca08c6f799b6742bef56f1c3db65e"
Sep 10 21:14:58.783909 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:58.783835    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"751efde4d95e28efa4d1ac02602b57ec14dca08c6f799b6742bef56f1c3db65e"} err="failed to get container status \"751efde4d95e28efa4d1ac02602b57ec14dca08c6f799b6742bef56f1c3db65e\": rpc error: code = NotFound desc = could not find container \"751efde4d95e28efa4d1ac02602b57ec14dca08c6f799b6742bef56f1c3db65e\": container with ID starting with 751efde4d95e28efa4d1ac02602b57ec14dca08c6f799b6742bef56f1c3db65e not found: ID does not exist"
Sep 10 21:14:58.802956 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:58.802935    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw"]
Sep 10 21:14:58.808661 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:58.808641    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04db70-bf66f56dc-z7pkw"]
Sep 10 21:14:59.440345 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:14:59.440310    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e80bd28-a8f4-4f72-a45f-9794ac6b3643" path="/var/lib/kubelet/pods/0e80bd28-a8f4-4f72-a45f-9794ac6b3643/volumes"
Sep 10 21:15:00.147662 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:00.147630    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817915-tvvlp"]
Sep 10 21:15:00.148043 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:00.148027    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e80bd28-a8f4-4f72-a45f-9794ac6b3643" containerName="payload-processing"
Sep 10 21:15:00.148043 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:00.148043    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e80bd28-a8f4-4f72-a45f-9794ac6b3643" containerName="payload-processing"
Sep 10 21:15:00.148139 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:00.148052    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16700be5-7584-4b07-9fe5-b1755a532e47" containerName="main"
Sep 10 21:15:00.148139 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:00.148059    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="16700be5-7584-4b07-9fe5-b1755a532e47" containerName="main"
Sep 10 21:15:00.148139 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:00.148122    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e80bd28-a8f4-4f72-a45f-9794ac6b3643" containerName="payload-processing"
Sep 10 21:15:00.148139 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:00.148131    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="16700be5-7584-4b07-9fe5-b1755a532e47" containerName="main"
Sep 10 21:15:00.152669 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:00.152649    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-tvvlp"
Sep 10 21:15:00.155057 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:00.155035    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5c7x\""
Sep 10 21:15:00.164022 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:00.163969    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817915-tvvlp"]
Sep 10 21:15:00.165175 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:00.165150    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-tvvlp"
Sep 10 21:15:00.461204 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:00.461181    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817915-tvvlp"]
Sep 10 21:15:00.463112 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:15:00.463087    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ddd7181_205c_465e_a978_3fa74b7b2c78.slice/crio-c4eff2d2f34d456470fab0d8af200d056fb50861c0fd1a800c997c419bf78936 WatchSource:0}: Error finding container c4eff2d2f34d456470fab0d8af200d056fb50861c0fd1a800c997c419bf78936: Status 404 returned error can't find the container with id c4eff2d2f34d456470fab0d8af200d056fb50861c0fd1a800c997c419bf78936
Sep 10 21:15:00.464874 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:00.464856    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:15:00.787025 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:00.786972    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-tvvlp" event={"ID":"1ddd7181-205c-465e-a978-3fa74b7b2c78","Type":"ContainerStarted","Data":"c4eff2d2f34d456470fab0d8af200d056fb50861c0fd1a800c997c419bf78936"}
Sep 10 21:15:00.876868 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:00.876834    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45" podUID="bf556761-5303-4c4b-a91c-972065ca11b7" containerName="main" containerID="cri-o://99cfcaa15ace4a3a3144d432fe0359af094262f5b0546171bdf077e2401a1e8a" gracePeriod=45
Sep 10 21:15:01.119655 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.119633    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:15:01.296010 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.295938    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf556761-5303-4c4b-a91c-972065ca11b7-dshm\") pod \"bf556761-5303-4c4b-a91c-972065ca11b7\" (UID: \"bf556761-5303-4c4b-a91c-972065ca11b7\") "
Sep 10 21:15:01.296136 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.296013    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf556761-5303-4c4b-a91c-972065ca11b7-home\") pod \"bf556761-5303-4c4b-a91c-972065ca11b7\" (UID: \"bf556761-5303-4c4b-a91c-972065ca11b7\") "
Sep 10 21:15:01.296136 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.296034    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf556761-5303-4c4b-a91c-972065ca11b7-tmp-dir\") pod \"bf556761-5303-4c4b-a91c-972065ca11b7\" (UID: \"bf556761-5303-4c4b-a91c-972065ca11b7\") "
Sep 10 21:15:01.296136 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.296049    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf556761-5303-4c4b-a91c-972065ca11b7-model-cache\") pod \"bf556761-5303-4c4b-a91c-972065ca11b7\" (UID: \"bf556761-5303-4c4b-a91c-972065ca11b7\") "
Sep 10 21:15:01.296136 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.296084    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r72x5\" (UniqueName: \"kubernetes.io/projected/bf556761-5303-4c4b-a91c-972065ca11b7-kube-api-access-r72x5\") pod \"bf556761-5303-4c4b-a91c-972065ca11b7\" (UID: \"bf556761-5303-4c4b-a91c-972065ca11b7\") "
Sep 10 21:15:01.296136 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.296103    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf556761-5303-4c4b-a91c-972065ca11b7-tls-certs\") pod \"bf556761-5303-4c4b-a91c-972065ca11b7\" (UID: \"bf556761-5303-4c4b-a91c-972065ca11b7\") "
Sep 10 21:15:01.296432 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.296306    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf556761-5303-4c4b-a91c-972065ca11b7-home" (OuterVolumeSpecName: "home") pod "bf556761-5303-4c4b-a91c-972065ca11b7" (UID: "bf556761-5303-4c4b-a91c-972065ca11b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:01.296432 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.296318    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf556761-5303-4c4b-a91c-972065ca11b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf556761-5303-4c4b-a91c-972065ca11b7" (UID: "bf556761-5303-4c4b-a91c-972065ca11b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:01.296432 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.296334    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf556761-5303-4c4b-a91c-972065ca11b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf556761-5303-4c4b-a91c-972065ca11b7" (UID: "bf556761-5303-4c4b-a91c-972065ca11b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:01.296432 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.296410    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf556761-5303-4c4b-a91c-972065ca11b7-home\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:01.296432 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.296426    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf556761-5303-4c4b-a91c-972065ca11b7-tmp-dir\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:01.296432 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.296435    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf556761-5303-4c4b-a91c-972065ca11b7-model-cache\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:01.298294 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.298263    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf556761-5303-4c4b-a91c-972065ca11b7-dshm" (OuterVolumeSpecName: "dshm") pod "bf556761-5303-4c4b-a91c-972065ca11b7" (UID: "bf556761-5303-4c4b-a91c-972065ca11b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:01.298408 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.298384    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf556761-5303-4c4b-a91c-972065ca11b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf556761-5303-4c4b-a91c-972065ca11b7" (UID: "bf556761-5303-4c4b-a91c-972065ca11b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:15:01.298494 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.298464    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf556761-5303-4c4b-a91c-972065ca11b7-kube-api-access-r72x5" (OuterVolumeSpecName: "kube-api-access-r72x5") pod "bf556761-5303-4c4b-a91c-972065ca11b7" (UID: "bf556761-5303-4c4b-a91c-972065ca11b7"). InnerVolumeSpecName "kube-api-access-r72x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:15:01.397641 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.397619    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf556761-5303-4c4b-a91c-972065ca11b7-dshm\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:01.397641 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.397642    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r72x5\" (UniqueName: \"kubernetes.io/projected/bf556761-5303-4c4b-a91c-972065ca11b7-kube-api-access-r72x5\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:01.397795 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.397653    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf556761-5303-4c4b-a91c-972065ca11b7-tls-certs\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:01.794668 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.794624    2572 generic.go:356] "Generic (PLEG): container finished" podID="bf556761-5303-4c4b-a91c-972065ca11b7" containerID="99cfcaa15ace4a3a3144d432fe0359af094262f5b0546171bdf077e2401a1e8a" exitCode=0
Sep 10 21:15:01.795173 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.794678    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"
Sep 10 21:15:01.795173 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.794701    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45" event={"ID":"bf556761-5303-4c4b-a91c-972065ca11b7","Type":"ContainerDied","Data":"99cfcaa15ace4a3a3144d432fe0359af094262f5b0546171bdf077e2401a1e8a"}
Sep 10 21:15:01.795173 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.794742    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45" event={"ID":"bf556761-5303-4c4b-a91c-972065ca11b7","Type":"ContainerDied","Data":"b56db4ed592a6723bee9e71b6ec803d85f9dbdfd0ededf5e82cc55beb7f6ac70"}
Sep 10 21:15:01.795173 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.794764    2572 scope.go:117] "RemoveContainer" containerID="99cfcaa15ace4a3a3144d432fe0359af094262f5b0546171bdf077e2401a1e8a"
Sep 10 21:15:01.804769 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.804752    2572 scope.go:117] "RemoveContainer" containerID="99cfcaa15ace4a3a3144d432fe0359af094262f5b0546171bdf077e2401a1e8a"
Sep 10 21:15:01.805176 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:15:01.805149    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99cfcaa15ace4a3a3144d432fe0359af094262f5b0546171bdf077e2401a1e8a\": container with ID starting with 99cfcaa15ace4a3a3144d432fe0359af094262f5b0546171bdf077e2401a1e8a not found: ID does not exist" containerID="99cfcaa15ace4a3a3144d432fe0359af094262f5b0546171bdf077e2401a1e8a"
Sep 10 21:15:01.805261 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.805186    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99cfcaa15ace4a3a3144d432fe0359af094262f5b0546171bdf077e2401a1e8a"} err="failed to get container status \"99cfcaa15ace4a3a3144d432fe0359af094262f5b0546171bdf077e2401a1e8a\": rpc error: code = NotFound desc = could not find container \"99cfcaa15ace4a3a3144d432fe0359af094262f5b0546171bdf077e2401a1e8a\": container with ID starting with 99cfcaa15ace4a3a3144d432fe0359af094262f5b0546171bdf077e2401a1e8a not found: ID does not exist"
Sep 10 21:15:01.812661 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.812633    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"]
Sep 10 21:15:01.814433 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.814413    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-04db70/test-model-04db70-kserve-86f7fbc76c-5kj45"]
Sep 10 21:15:01.849959 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:01.849926    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq" podUID="3b5faa84-5402-40e6-b011-1f4cfe719278" containerName="main" containerID="cri-o://fda6380378f36f14db86b11eb609c0ce5241db545d5a32338b7f1a82028ced6b" gracePeriod=45
Sep 10 21:15:02.078525 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.078447    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"]
Sep 10 21:15:02.078880 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.078862    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf556761-5303-4c4b-a91c-972065ca11b7" containerName="main"
Sep 10 21:15:02.078951 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.078884    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf556761-5303-4c4b-a91c-972065ca11b7" containerName="main"
Sep 10 21:15:02.079032 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.078995    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf556761-5303-4c4b-a91c-972065ca11b7" containerName="main"
Sep 10 21:15:02.085223 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.085199    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"
Sep 10 21:15:02.087838 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.087811    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-6324ec\"/\"noten-model-6324ec-kserve-self-signed-certs\""
Sep 10 21:15:02.088004 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.087877    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-6324ec\"/\"openshift-service-ca.crt\""
Sep 10 21:15:02.088097 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.088008    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-6324ec\"/\"default-dockercfg-6g26d\""
Sep 10 21:15:02.088676 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.088657    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-6324ec\"/\"kube-root-ca.crt\""
Sep 10 21:15:02.092799 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.092772    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"]
Sep 10 21:15:02.204641 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.204614    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msd76\" (UniqueName: \"kubernetes.io/projected/b4994cae-5984-4f3f-bce1-a6142ed7f466-kube-api-access-msd76\") pod \"noten-model-6324ec-kserve-789bc5b979-cf2z8\" (UID: \"b4994cae-5984-4f3f-bce1-a6142ed7f466\") " pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"
Sep 10 21:15:02.204810 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.204661    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4994cae-5984-4f3f-bce1-a6142ed7f466-tls-certs\") pod \"noten-model-6324ec-kserve-789bc5b979-cf2z8\" (UID: \"b4994cae-5984-4f3f-bce1-a6142ed7f466\") " pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"
Sep 10 21:15:02.204810 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.204755    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4994cae-5984-4f3f-bce1-a6142ed7f466-dshm\") pod \"noten-model-6324ec-kserve-789bc5b979-cf2z8\" (UID: \"b4994cae-5984-4f3f-bce1-a6142ed7f466\") " pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"
Sep 10 21:15:02.204810 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.204786    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4994cae-5984-4f3f-bce1-a6142ed7f466-model-cache\") pod \"noten-model-6324ec-kserve-789bc5b979-cf2z8\" (UID: \"b4994cae-5984-4f3f-bce1-a6142ed7f466\") " pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"
Sep 10 21:15:02.204946 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.204834    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4994cae-5984-4f3f-bce1-a6142ed7f466-tmp-dir\") pod \"noten-model-6324ec-kserve-789bc5b979-cf2z8\" (UID: \"b4994cae-5984-4f3f-bce1-a6142ed7f466\") " pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"
Sep 10 21:15:02.204946 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.204878    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4994cae-5984-4f3f-bce1-a6142ed7f466-home\") pod \"noten-model-6324ec-kserve-789bc5b979-cf2z8\" (UID: \"b4994cae-5984-4f3f-bce1-a6142ed7f466\") " pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"
Sep 10 21:15:02.306029 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.305993    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4994cae-5984-4f3f-bce1-a6142ed7f466-dshm\") pod \"noten-model-6324ec-kserve-789bc5b979-cf2z8\" (UID: \"b4994cae-5984-4f3f-bce1-a6142ed7f466\") " pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"
Sep 10 21:15:02.306194 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.306038    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4994cae-5984-4f3f-bce1-a6142ed7f466-model-cache\") pod \"noten-model-6324ec-kserve-789bc5b979-cf2z8\" (UID: \"b4994cae-5984-4f3f-bce1-a6142ed7f466\") " pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"
Sep 10 21:15:02.306194 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.306076    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4994cae-5984-4f3f-bce1-a6142ed7f466-tmp-dir\") pod \"noten-model-6324ec-kserve-789bc5b979-cf2z8\" (UID: \"b4994cae-5984-4f3f-bce1-a6142ed7f466\") " pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"
Sep 10 21:15:02.306194 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.306125    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4994cae-5984-4f3f-bce1-a6142ed7f466-home\") pod \"noten-model-6324ec-kserve-789bc5b979-cf2z8\" (UID: \"b4994cae-5984-4f3f-bce1-a6142ed7f466\") " pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"
Sep 10 21:15:02.306194 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.306185    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msd76\" (UniqueName: \"kubernetes.io/projected/b4994cae-5984-4f3f-bce1-a6142ed7f466-kube-api-access-msd76\") pod \"noten-model-6324ec-kserve-789bc5b979-cf2z8\" (UID: \"b4994cae-5984-4f3f-bce1-a6142ed7f466\") " pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"
Sep 10 21:15:02.306435 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.306223    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4994cae-5984-4f3f-bce1-a6142ed7f466-tls-certs\") pod \"noten-model-6324ec-kserve-789bc5b979-cf2z8\" (UID: \"b4994cae-5984-4f3f-bce1-a6142ed7f466\") " pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"
Sep 10 21:15:02.306501 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.306472    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4994cae-5984-4f3f-bce1-a6142ed7f466-model-cache\") pod \"noten-model-6324ec-kserve-789bc5b979-cf2z8\" (UID: \"b4994cae-5984-4f3f-bce1-a6142ed7f466\") " pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"
Sep 10 21:15:02.306556 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.306524    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4994cae-5984-4f3f-bce1-a6142ed7f466-tmp-dir\") pod \"noten-model-6324ec-kserve-789bc5b979-cf2z8\" (UID: \"b4994cae-5984-4f3f-bce1-a6142ed7f466\") " pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"
Sep 10 21:15:02.306641 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.306613    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4994cae-5984-4f3f-bce1-a6142ed7f466-home\") pod \"noten-model-6324ec-kserve-789bc5b979-cf2z8\" (UID: \"b4994cae-5984-4f3f-bce1-a6142ed7f466\") " pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"
Sep 10 21:15:02.309027 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.309002    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4994cae-5984-4f3f-bce1-a6142ed7f466-dshm\") pod \"noten-model-6324ec-kserve-789bc5b979-cf2z8\" (UID: \"b4994cae-5984-4f3f-bce1-a6142ed7f466\") " pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"
Sep 10 21:15:02.309299 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.309279    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4994cae-5984-4f3f-bce1-a6142ed7f466-tls-certs\") pod \"noten-model-6324ec-kserve-789bc5b979-cf2z8\" (UID: \"b4994cae-5984-4f3f-bce1-a6142ed7f466\") " pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"
Sep 10 21:15:02.315170 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.315152    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msd76\" (UniqueName: \"kubernetes.io/projected/b4994cae-5984-4f3f-bce1-a6142ed7f466-kube-api-access-msd76\") pod \"noten-model-6324ec-kserve-789bc5b979-cf2z8\" (UID: \"b4994cae-5984-4f3f-bce1-a6142ed7f466\") " pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"
Sep 10 21:15:02.398865 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.398774    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"
Sep 10 21:15:02.792853 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.792825    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:15:02.801744 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.801715    2572 generic.go:356] "Generic (PLEG): container finished" podID="3b5faa84-5402-40e6-b011-1f4cfe719278" containerID="fda6380378f36f14db86b11eb609c0ce5241db545d5a32338b7f1a82028ced6b" exitCode=0
Sep 10 21:15:02.802052 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.801745    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"
Sep 10 21:15:02.802052 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.801754    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq" event={"ID":"3b5faa84-5402-40e6-b011-1f4cfe719278","Type":"ContainerDied","Data":"fda6380378f36f14db86b11eb609c0ce5241db545d5a32338b7f1a82028ced6b"}
Sep 10 21:15:02.802052 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.801783    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq" event={"ID":"3b5faa84-5402-40e6-b011-1f4cfe719278","Type":"ContainerDied","Data":"b3a1d7c23501dd43de6b3714966bc405de4c98131640e746b393c9287c69265f"}
Sep 10 21:15:02.802052 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.801799    2572 scope.go:117] "RemoveContainer" containerID="fda6380378f36f14db86b11eb609c0ce5241db545d5a32338b7f1a82028ced6b"
Sep 10 21:15:02.809524 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.809511    2572 scope.go:117] "RemoveContainer" containerID="fda6380378f36f14db86b11eb609c0ce5241db545d5a32338b7f1a82028ced6b"
Sep 10 21:15:02.809787 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:15:02.809771    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fda6380378f36f14db86b11eb609c0ce5241db545d5a32338b7f1a82028ced6b\": container with ID starting with fda6380378f36f14db86b11eb609c0ce5241db545d5a32338b7f1a82028ced6b not found: ID does not exist" containerID="fda6380378f36f14db86b11eb609c0ce5241db545d5a32338b7f1a82028ced6b"
Sep 10 21:15:02.809828 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.809794    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fda6380378f36f14db86b11eb609c0ce5241db545d5a32338b7f1a82028ced6b"} err="failed to get container status \"fda6380378f36f14db86b11eb609c0ce5241db545d5a32338b7f1a82028ced6b\": rpc error: code = NotFound desc = could not find container \"fda6380378f36f14db86b11eb609c0ce5241db545d5a32338b7f1a82028ced6b\": container with ID starting with fda6380378f36f14db86b11eb609c0ce5241db545d5a32338b7f1a82028ced6b not found: ID does not exist"
Sep 10 21:15:02.908892 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.908870    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"]
Sep 10 21:15:02.911767 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:15:02.911744    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4994cae_5984_4f3f_bce1_a6142ed7f466.slice/crio-dbe1aec26b61a90026c0b1b543ecf83575bb4c9e9174a9bfb818fe9beaf6fd33 WatchSource:0}: Error finding container dbe1aec26b61a90026c0b1b543ecf83575bb4c9e9174a9bfb818fe9beaf6fd33: Status 404 returned error can't find the container with id dbe1aec26b61a90026c0b1b543ecf83575bb4c9e9174a9bfb818fe9beaf6fd33
Sep 10 21:15:02.912718 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.912698    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b5faa84-5402-40e6-b011-1f4cfe719278-tmp-dir\") pod \"3b5faa84-5402-40e6-b011-1f4cfe719278\" (UID: \"3b5faa84-5402-40e6-b011-1f4cfe719278\") "
Sep 10 21:15:02.912806 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.912744    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ns24t\" (UniqueName: \"kubernetes.io/projected/3b5faa84-5402-40e6-b011-1f4cfe719278-kube-api-access-ns24t\") pod \"3b5faa84-5402-40e6-b011-1f4cfe719278\" (UID: \"3b5faa84-5402-40e6-b011-1f4cfe719278\") "
Sep 10 21:15:02.912806 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.912772    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5faa84-5402-40e6-b011-1f4cfe719278-tls-certs\") pod \"3b5faa84-5402-40e6-b011-1f4cfe719278\" (UID: \"3b5faa84-5402-40e6-b011-1f4cfe719278\") "
Sep 10 21:15:02.912806 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.912802    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b5faa84-5402-40e6-b011-1f4cfe719278-dshm\") pod \"3b5faa84-5402-40e6-b011-1f4cfe719278\" (UID: \"3b5faa84-5402-40e6-b011-1f4cfe719278\") "
Sep 10 21:15:02.912968 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.912861    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b5faa84-5402-40e6-b011-1f4cfe719278-model-cache\") pod \"3b5faa84-5402-40e6-b011-1f4cfe719278\" (UID: \"3b5faa84-5402-40e6-b011-1f4cfe719278\") "
Sep 10 21:15:02.912968 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.912893    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b5faa84-5402-40e6-b011-1f4cfe719278-home\") pod \"3b5faa84-5402-40e6-b011-1f4cfe719278\" (UID: \"3b5faa84-5402-40e6-b011-1f4cfe719278\") "
Sep 10 21:15:02.913104 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.913013    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5faa84-5402-40e6-b011-1f4cfe719278-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3b5faa84-5402-40e6-b011-1f4cfe719278" (UID: "3b5faa84-5402-40e6-b011-1f4cfe719278"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:02.913215 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.913193    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5faa84-5402-40e6-b011-1f4cfe719278-model-cache" (OuterVolumeSpecName: "model-cache") pod "3b5faa84-5402-40e6-b011-1f4cfe719278" (UID: "3b5faa84-5402-40e6-b011-1f4cfe719278"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:02.913296 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.913209    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5faa84-5402-40e6-b011-1f4cfe719278-home" (OuterVolumeSpecName: "home") pod "3b5faa84-5402-40e6-b011-1f4cfe719278" (UID: "3b5faa84-5402-40e6-b011-1f4cfe719278"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:02.913702 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.913674    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b5faa84-5402-40e6-b011-1f4cfe719278-tmp-dir\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:02.913702 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.913702    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b5faa84-5402-40e6-b011-1f4cfe719278-model-cache\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:02.913880 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.913715    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b5faa84-5402-40e6-b011-1f4cfe719278-home\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:02.915418 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.915395    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5faa84-5402-40e6-b011-1f4cfe719278-kube-api-access-ns24t" (OuterVolumeSpecName: "kube-api-access-ns24t") pod "3b5faa84-5402-40e6-b011-1f4cfe719278" (UID: "3b5faa84-5402-40e6-b011-1f4cfe719278"). InnerVolumeSpecName "kube-api-access-ns24t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:15:02.915498 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.915442    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b5faa84-5402-40e6-b011-1f4cfe719278-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b5faa84-5402-40e6-b011-1f4cfe719278" (UID: "3b5faa84-5402-40e6-b011-1f4cfe719278"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:15:02.915560 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:02.915545    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5faa84-5402-40e6-b011-1f4cfe719278-dshm" (OuterVolumeSpecName: "dshm") pod "3b5faa84-5402-40e6-b011-1f4cfe719278" (UID: "3b5faa84-5402-40e6-b011-1f4cfe719278"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:03.014616 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:03.014594    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b5faa84-5402-40e6-b011-1f4cfe719278-dshm\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:03.014616 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:03.014616    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns24t\" (UniqueName: \"kubernetes.io/projected/3b5faa84-5402-40e6-b011-1f4cfe719278-kube-api-access-ns24t\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:03.014740 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:03.014625    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b5faa84-5402-40e6-b011-1f4cfe719278-tls-certs\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:03.122520 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:03.122487    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"]
Sep 10 21:15:03.124190 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:03.124169    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7eaf1f/test-model-7eaf1f-kserve-86576fdb77-jxwvq"]
Sep 10 21:15:03.438704 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:03.438670    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b5faa84-5402-40e6-b011-1f4cfe719278" path="/var/lib/kubelet/pods/3b5faa84-5402-40e6-b011-1f4cfe719278/volumes"
Sep 10 21:15:03.439121 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:03.439102    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf556761-5303-4c4b-a91c-972065ca11b7" path="/var/lib/kubelet/pods/bf556761-5303-4c4b-a91c-972065ca11b7/volumes"
Sep 10 21:15:03.809250 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:03.809214    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8" event={"ID":"b4994cae-5984-4f3f-bce1-a6142ed7f466","Type":"ContainerStarted","Data":"7a1c27fab208628df1cdbecbf3e3fa72024cac55eea2b4bbae050e7973f43083"}
Sep 10 21:15:03.809652 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:03.809267    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8" event={"ID":"b4994cae-5984-4f3f-bce1-a6142ed7f466","Type":"ContainerStarted","Data":"dbe1aec26b61a90026c0b1b543ecf83575bb4c9e9174a9bfb818fe9beaf6fd33"}
Sep 10 21:15:03.811027 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:03.811000    2572 generic.go:356] "Generic (PLEG): container finished" podID="1ddd7181-205c-465e-a978-3fa74b7b2c78" containerID="62abe8827c2fc080a9eac5663f63aa692cdba4c497be9f248a26b0701dc8d749" exitCode=0
Sep 10 21:15:03.811123 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:03.811083    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-tvvlp" event={"ID":"1ddd7181-205c-465e-a978-3fa74b7b2c78","Type":"ContainerDied","Data":"62abe8827c2fc080a9eac5663f63aa692cdba4c497be9f248a26b0701dc8d749"}
Sep 10 21:15:03.827228 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:03.827186    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8" podStartSLOduration=1.827172061 podStartE2EDuration="1.827172061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:03.82612415 +0000 UTC m=+1089.038589072" watchObservedRunningTime="2026-09-10 21:15:03.827172061 +0000 UTC m=+1089.039636979"
Sep 10 21:15:04.950797 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:04.950775    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-tvvlp"
Sep 10 21:15:05.751784 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:05.751703    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"]
Sep 10 21:15:05.751922 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:05.751899    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk" podUID="e35a47b7-668d-4c95-b2b8-446d5e762a8c" containerName="istio-proxy" containerID="cri-o://30f3436cca127fe33e24548cb7222df89890e1e6d37fc052844336191eebce65" gracePeriod=30
Sep 10 21:15:05.821708 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:05.821678    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-tvvlp"
Sep 10 21:15:05.821833 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:05.821732    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-tvvlp" event={"ID":"1ddd7181-205c-465e-a978-3fa74b7b2c78","Type":"ContainerDied","Data":"c4eff2d2f34d456470fab0d8af200d056fb50861c0fd1a800c997c419bf78936"}
Sep 10 21:15:05.821833 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:05.821778    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4eff2d2f34d456470fab0d8af200d056fb50861c0fd1a800c997c419bf78936"
Sep 10 21:15:06.325022 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:06.324976    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:15:09.112228 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:09.112199    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"]
Sep 10 21:15:11.007505 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.007480    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:15:11.175673 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.175646    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6s84h\" (UniqueName: \"kubernetes.io/projected/e35a47b7-668d-4c95-b2b8-446d5e762a8c-kube-api-access-6s84h\") pod \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") "
Sep 10 21:15:11.175819 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.175687    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-workload-socket\") pod \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") "
Sep 10 21:15:11.175819 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.175705    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-credential-socket\") pod \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") "
Sep 10 21:15:11.175899 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.175840    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istio-podinfo\") pod \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") "
Sep 10 21:15:11.175899 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.175882    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istio-token\") pod \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") "
Sep 10 21:15:11.176019 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.175915    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-workload-certs\") pod \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") "
Sep 10 21:15:11.176019 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.175951    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istio-envoy\") pod \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") "
Sep 10 21:15:11.176019 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.175957    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e35a47b7-668d-4c95-b2b8-446d5e762a8c" (UID: "e35a47b7-668d-4c95-b2b8-446d5e762a8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:11.176019 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.175974    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e35a47b7-668d-4c95-b2b8-446d5e762a8c" (UID: "e35a47b7-668d-4c95-b2b8-446d5e762a8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:11.176019 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.176007    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istio-data\") pod \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") "
Sep 10 21:15:11.176280 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.176074    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istiod-ca-cert\") pod \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\" (UID: \"e35a47b7-668d-4c95-b2b8-446d5e762a8c\") "
Sep 10 21:15:11.176280 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.176219    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e35a47b7-668d-4c95-b2b8-446d5e762a8c" (UID: "e35a47b7-668d-4c95-b2b8-446d5e762a8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:11.176388 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.176277    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istio-data" (OuterVolumeSpecName: "istio-data") pod "e35a47b7-668d-4c95-b2b8-446d5e762a8c" (UID: "e35a47b7-668d-4c95-b2b8-446d5e762a8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:11.176443 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.176373    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-workload-certs\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:11.176443 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.176418    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istio-data\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:11.176443 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.176432    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-workload-socket\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:11.176765 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.176446    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-credential-socket\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:11.176765 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.176712    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e35a47b7-668d-4c95-b2b8-446d5e762a8c" (UID: "e35a47b7-668d-4c95-b2b8-446d5e762a8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:15:11.178081 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.178050    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e35a47b7-668d-4c95-b2b8-446d5e762a8c-kube-api-access-6s84h" (OuterVolumeSpecName: "kube-api-access-6s84h") pod "e35a47b7-668d-4c95-b2b8-446d5e762a8c" (UID: "e35a47b7-668d-4c95-b2b8-446d5e762a8c"). InnerVolumeSpecName "kube-api-access-6s84h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:15:11.178194 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.178167    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e35a47b7-668d-4c95-b2b8-446d5e762a8c" (UID: "e35a47b7-668d-4c95-b2b8-446d5e762a8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:11.178243 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.178224    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e35a47b7-668d-4c95-b2b8-446d5e762a8c" (UID: "e35a47b7-668d-4c95-b2b8-446d5e762a8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:15:11.178284 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.178271    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istio-token" (OuterVolumeSpecName: "istio-token") pod "e35a47b7-668d-4c95-b2b8-446d5e762a8c" (UID: "e35a47b7-668d-4c95-b2b8-446d5e762a8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:15:11.277238 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.277215    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istio-podinfo\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:11.277238 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.277236    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istio-token\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:11.277359 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.277246    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istio-envoy\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:11.277359 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.277255    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35a47b7-668d-4c95-b2b8-446d5e762a8c-istiod-ca-cert\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:11.277359 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.277263    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s84h\" (UniqueName: \"kubernetes.io/projected/e35a47b7-668d-4c95-b2b8-446d5e762a8c-kube-api-access-6s84h\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:11.854674 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.854637    2572 generic.go:356] "Generic (PLEG): container finished" podID="e35a47b7-668d-4c95-b2b8-446d5e762a8c" containerID="30f3436cca127fe33e24548cb7222df89890e1e6d37fc052844336191eebce65" exitCode=0
Sep 10 21:15:11.854814 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.854701    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"
Sep 10 21:15:11.854814 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.854733    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk" event={"ID":"e35a47b7-668d-4c95-b2b8-446d5e762a8c","Type":"ContainerDied","Data":"30f3436cca127fe33e24548cb7222df89890e1e6d37fc052844336191eebce65"}
Sep 10 21:15:11.854814 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.854801    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk" event={"ID":"e35a47b7-668d-4c95-b2b8-446d5e762a8c","Type":"ContainerDied","Data":"5ea9f16c5fefdd6e3e199277ce56948cdf0bce42a22be5deb6e75e0232c07f49"}
Sep 10 21:15:11.854929 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.854829    2572 scope.go:117] "RemoveContainer" containerID="30f3436cca127fe33e24548cb7222df89890e1e6d37fc052844336191eebce65"
Sep 10 21:15:11.865313 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.865279    2572 scope.go:117] "RemoveContainer" containerID="c42e42d0caa02bcdb3dd7789d1c054c14d0682df50abd4781b74dc535fd78b7e"
Sep 10 21:15:11.875023 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.874813    2572 scope.go:117] "RemoveContainer" containerID="30f3436cca127fe33e24548cb7222df89890e1e6d37fc052844336191eebce65"
Sep 10 21:15:11.875130 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.875105    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"]
Sep 10 21:15:11.875172 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:15:11.875138    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f3436cca127fe33e24548cb7222df89890e1e6d37fc052844336191eebce65\": container with ID starting with 30f3436cca127fe33e24548cb7222df89890e1e6d37fc052844336191eebce65 not found: ID does not exist" containerID="30f3436cca127fe33e24548cb7222df89890e1e6d37fc052844336191eebce65"
Sep 10 21:15:11.875206 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.875164    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f3436cca127fe33e24548cb7222df89890e1e6d37fc052844336191eebce65"} err="failed to get container status \"30f3436cca127fe33e24548cb7222df89890e1e6d37fc052844336191eebce65\": rpc error: code = NotFound desc = could not find container \"30f3436cca127fe33e24548cb7222df89890e1e6d37fc052844336191eebce65\": container with ID starting with 30f3436cca127fe33e24548cb7222df89890e1e6d37fc052844336191eebce65 not found: ID does not exist"
Sep 10 21:15:11.875206 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.875187    2572 scope.go:117] "RemoveContainer" containerID="c42e42d0caa02bcdb3dd7789d1c054c14d0682df50abd4781b74dc535fd78b7e"
Sep 10 21:15:11.875473 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:15:11.875451    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c42e42d0caa02bcdb3dd7789d1c054c14d0682df50abd4781b74dc535fd78b7e\": container with ID starting with c42e42d0caa02bcdb3dd7789d1c054c14d0682df50abd4781b74dc535fd78b7e not found: ID does not exist" containerID="c42e42d0caa02bcdb3dd7789d1c054c14d0682df50abd4781b74dc535fd78b7e"
Sep 10 21:15:11.875538 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.875483    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c42e42d0caa02bcdb3dd7789d1c054c14d0682df50abd4781b74dc535fd78b7e"} err="failed to get container status \"c42e42d0caa02bcdb3dd7789d1c054c14d0682df50abd4781b74dc535fd78b7e\": rpc error: code = NotFound desc = could not find container \"c42e42d0caa02bcdb3dd7789d1c054c14d0682df50abd4781b74dc535fd78b7e\": container with ID starting with c42e42d0caa02bcdb3dd7789d1c054c14d0682df50abd4781b74dc535fd78b7e not found: ID does not exist"
Sep 10 21:15:11.880049 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:11.880026    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-04db70-openshift-default-799fdccb4d-cm7mk"]
Sep 10 21:15:12.399208 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:12.399184    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"
Sep 10 21:15:13.438790 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:13.438747    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e35a47b7-668d-4c95-b2b8-446d5e762a8c" path="/var/lib/kubelet/pods/e35a47b7-668d-4c95-b2b8-446d5e762a8c/volumes"
Sep 10 21:15:14.211100 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.211071    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"]
Sep 10 21:15:14.211407 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.211393    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e35a47b7-668d-4c95-b2b8-446d5e762a8c" containerName="istio-proxy"
Sep 10 21:15:14.211454 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.211408    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e35a47b7-668d-4c95-b2b8-446d5e762a8c" containerName="istio-proxy"
Sep 10 21:15:14.211454 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.211417    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ddd7181-205c-465e-a978-3fa74b7b2c78" containerName="cleanup"
Sep 10 21:15:14.211454 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.211436    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ddd7181-205c-465e-a978-3fa74b7b2c78" containerName="cleanup"
Sep 10 21:15:14.211548 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.211460    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b5faa84-5402-40e6-b011-1f4cfe719278" containerName="main"
Sep 10 21:15:14.211548 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.211466    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b5faa84-5402-40e6-b011-1f4cfe719278" containerName="main"
Sep 10 21:15:14.211548 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.211528    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ddd7181-205c-465e-a978-3fa74b7b2c78" containerName="cleanup"
Sep 10 21:15:14.211548 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.211540    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b5faa84-5402-40e6-b011-1f4cfe719278" containerName="main"
Sep 10 21:15:14.211548 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.211547    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e35a47b7-668d-4c95-b2b8-446d5e762a8c" containerName="istio-proxy"
Sep 10 21:15:14.211706 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.211552    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e35a47b7-668d-4c95-b2b8-446d5e762a8c" containerName="istio-proxy"
Sep 10 21:15:14.216142 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.216125    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.219064 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.219039    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-899bf7e2-openshift-default-dockercfg-vtzsx\""
Sep 10 21:15:14.228318 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.228298    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"]
Sep 10 21:15:14.401730 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.401705    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdtfl\" (UniqueName: \"kubernetes.io/projected/454b3bae-5f97-42d4-b202-5a983a492099-kube-api-access-gdtfl\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.401884 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.401752    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-workload-certs\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.401884 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.401785    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-istio-envoy\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.401884 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.401836    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/454b3bae-5f97-42d4-b202-5a983a492099-istio-token\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.401884 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.401879    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/454b3bae-5f97-42d4-b202-5a983a492099-istiod-ca-cert\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.402084 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.401909    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-credential-socket\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.402084 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.401926    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-istio-data\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.402084 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.402041    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-workload-socket\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.402084 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.402071    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/454b3bae-5f97-42d4-b202-5a983a492099-istio-podinfo\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.502844 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.502783    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-workload-socket\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.502844 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.502809    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/454b3bae-5f97-42d4-b202-5a983a492099-istio-podinfo\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.502844 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.502833    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdtfl\" (UniqueName: \"kubernetes.io/projected/454b3bae-5f97-42d4-b202-5a983a492099-kube-api-access-gdtfl\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.503384 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.502855    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-workload-certs\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.503384 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.502905    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-istio-envoy\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.503384 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.502953    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/454b3bae-5f97-42d4-b202-5a983a492099-istio-token\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.503384 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.503062    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/454b3bae-5f97-42d4-b202-5a983a492099-istiod-ca-cert\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.503384 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.503109    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-credential-socket\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.503384 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.503138    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-istio-data\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.503384 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.503350    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-workload-certs\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.503719 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.503501    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-istio-data\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.503719 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.503590    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-credential-socket\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.503719 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.503623    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-workload-socket\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.504027 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.504003    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/454b3bae-5f97-42d4-b202-5a983a492099-istiod-ca-cert\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.505334 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.505307    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-istio-envoy\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.505660 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.505637    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/454b3bae-5f97-42d4-b202-5a983a492099-istio-podinfo\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.511645 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.511627    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/454b3bae-5f97-42d4-b202-5a983a492099-istio-token\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.511733 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.511661    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdtfl\" (UniqueName: \"kubernetes.io/projected/454b3bae-5f97-42d4-b202-5a983a492099-kube-api-access-gdtfl\") pod \"e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") " pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.526119 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.526103    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:14.820849 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.820827    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"]
Sep 10 21:15:14.823505 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:15:14.823477    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454b3bae_5f97_42d4_b202_5a983a492099.slice/crio-c75f05273bc8d5b9a4f1d9eeed46311af5984c5ef923b7b25260697821bdc1b7 WatchSource:0}: Error finding container c75f05273bc8d5b9a4f1d9eeed46311af5984c5ef923b7b25260697821bdc1b7: Status 404 returned error can't find the container with id c75f05273bc8d5b9a4f1d9eeed46311af5984c5ef923b7b25260697821bdc1b7
Sep 10 21:15:14.826130 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.826096    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:15:14.826219 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.826166    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:15:14.826219 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.826198    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:15:14.882522 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:14.882498    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65" event={"ID":"454b3bae-5f97-42d4-b202-5a983a492099","Type":"ContainerStarted","Data":"c75f05273bc8d5b9a4f1d9eeed46311af5984c5ef923b7b25260697821bdc1b7"}
Sep 10 21:15:15.888541 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:15.888505    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65" event={"ID":"454b3bae-5f97-42d4-b202-5a983a492099","Type":"ContainerStarted","Data":"c80ba4f83593398727e28f924a3a0bbe949a799cd42212d89bb51c5e07c6d8fb"}
Sep 10 21:15:15.916959 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:15.916910    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65" podStartSLOduration=1.916894122 podStartE2EDuration="1.916894122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:15.91517202 +0000 UTC m=+1101.127636938" watchObservedRunningTime="2026-09-10 21:15:15.916894122 +0000 UTC m=+1101.129359045"
Sep 10 21:15:16.432571 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:16.432536    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:15:16.526217 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:16.526189    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:16.531304 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:16.531281    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:16.894117 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:16.894088    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:16.895099 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:16.895075    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:24.134596 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.134529    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8" podUID="b4994cae-5984-4f3f-bce1-a6142ed7f466" containerName="main" containerID="cri-o://7a1c27fab208628df1cdbecbf3e3fa72024cac55eea2b4bbae050e7973f43083" gracePeriod=45
Sep 10 21:15:24.241436 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.237905    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"]
Sep 10 21:15:24.241436 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.238250    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl" podUID="9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04" containerName="istio-proxy" containerID="cri-o://523b4d2e352411d602dac1297c69cff971d38868c334f6a1de6237183da1636d" gracePeriod=30
Sep 10 21:15:24.403228 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.403204    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"
Sep 10 21:15:24.483420 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.483391    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4994cae-5984-4f3f-bce1-a6142ed7f466-dshm\") pod \"b4994cae-5984-4f3f-bce1-a6142ed7f466\" (UID: \"b4994cae-5984-4f3f-bce1-a6142ed7f466\") "
Sep 10 21:15:24.483420 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.483421    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4994cae-5984-4f3f-bce1-a6142ed7f466-model-cache\") pod \"b4994cae-5984-4f3f-bce1-a6142ed7f466\" (UID: \"b4994cae-5984-4f3f-bce1-a6142ed7f466\") "
Sep 10 21:15:24.483591 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.483454    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msd76\" (UniqueName: \"kubernetes.io/projected/b4994cae-5984-4f3f-bce1-a6142ed7f466-kube-api-access-msd76\") pod \"b4994cae-5984-4f3f-bce1-a6142ed7f466\" (UID: \"b4994cae-5984-4f3f-bce1-a6142ed7f466\") "
Sep 10 21:15:24.483591 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.483477    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4994cae-5984-4f3f-bce1-a6142ed7f466-tmp-dir\") pod \"b4994cae-5984-4f3f-bce1-a6142ed7f466\" (UID: \"b4994cae-5984-4f3f-bce1-a6142ed7f466\") "
Sep 10 21:15:24.483591 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.483506    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4994cae-5984-4f3f-bce1-a6142ed7f466-tls-certs\") pod \"b4994cae-5984-4f3f-bce1-a6142ed7f466\" (UID: \"b4994cae-5984-4f3f-bce1-a6142ed7f466\") "
Sep 10 21:15:24.483591 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.483586    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4994cae-5984-4f3f-bce1-a6142ed7f466-home\") pod \"b4994cae-5984-4f3f-bce1-a6142ed7f466\" (UID: \"b4994cae-5984-4f3f-bce1-a6142ed7f466\") "
Sep 10 21:15:24.483800 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.483729    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4994cae-5984-4f3f-bce1-a6142ed7f466-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b4994cae-5984-4f3f-bce1-a6142ed7f466" (UID: "b4994cae-5984-4f3f-bce1-a6142ed7f466"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:24.483880 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.483860    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4994cae-5984-4f3f-bce1-a6142ed7f466-tmp-dir\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:24.483998 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.483954    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4994cae-5984-4f3f-bce1-a6142ed7f466-model-cache" (OuterVolumeSpecName: "model-cache") pod "b4994cae-5984-4f3f-bce1-a6142ed7f466" (UID: "b4994cae-5984-4f3f-bce1-a6142ed7f466"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:24.484047 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.483999    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4994cae-5984-4f3f-bce1-a6142ed7f466-home" (OuterVolumeSpecName: "home") pod "b4994cae-5984-4f3f-bce1-a6142ed7f466" (UID: "b4994cae-5984-4f3f-bce1-a6142ed7f466"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:24.485839 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.485818    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4994cae-5984-4f3f-bce1-a6142ed7f466-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b4994cae-5984-4f3f-bce1-a6142ed7f466" (UID: "b4994cae-5984-4f3f-bce1-a6142ed7f466"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:15:24.485932 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.485913    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4994cae-5984-4f3f-bce1-a6142ed7f466-kube-api-access-msd76" (OuterVolumeSpecName: "kube-api-access-msd76") pod "b4994cae-5984-4f3f-bce1-a6142ed7f466" (UID: "b4994cae-5984-4f3f-bce1-a6142ed7f466"). InnerVolumeSpecName "kube-api-access-msd76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:15:24.486017 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.486003    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4994cae-5984-4f3f-bce1-a6142ed7f466-dshm" (OuterVolumeSpecName: "dshm") pod "b4994cae-5984-4f3f-bce1-a6142ed7f466" (UID: "b4994cae-5984-4f3f-bce1-a6142ed7f466"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:24.584827 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.584798    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4994cae-5984-4f3f-bce1-a6142ed7f466-home\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:24.584827 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.584823    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4994cae-5984-4f3f-bce1-a6142ed7f466-dshm\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:24.585009 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.584832    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4994cae-5984-4f3f-bce1-a6142ed7f466-model-cache\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:24.585009 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.584843    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msd76\" (UniqueName: \"kubernetes.io/projected/b4994cae-5984-4f3f-bce1-a6142ed7f466-kube-api-access-msd76\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:24.585009 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.584864    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4994cae-5984-4f3f-bce1-a6142ed7f466-tls-certs\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:24.938934 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.938899    2572 generic.go:356] "Generic (PLEG): container finished" podID="b4994cae-5984-4f3f-bce1-a6142ed7f466" containerID="7a1c27fab208628df1cdbecbf3e3fa72024cac55eea2b4bbae050e7973f43083" exitCode=0
Sep 10 21:15:24.938934 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.938934    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8" event={"ID":"b4994cae-5984-4f3f-bce1-a6142ed7f466","Type":"ContainerDied","Data":"7a1c27fab208628df1cdbecbf3e3fa72024cac55eea2b4bbae050e7973f43083"}
Sep 10 21:15:24.939228 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.938962    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8" event={"ID":"b4994cae-5984-4f3f-bce1-a6142ed7f466","Type":"ContainerDied","Data":"dbe1aec26b61a90026c0b1b543ecf83575bb4c9e9174a9bfb818fe9beaf6fd33"}
Sep 10 21:15:24.939228 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.938998    2572 scope.go:117] "RemoveContainer" containerID="7a1c27fab208628df1cdbecbf3e3fa72024cac55eea2b4bbae050e7973f43083"
Sep 10 21:15:24.939228 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.939112    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"
Sep 10 21:15:24.949317 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.949299    2572 scope.go:117] "RemoveContainer" containerID="7a1c27fab208628df1cdbecbf3e3fa72024cac55eea2b4bbae050e7973f43083"
Sep 10 21:15:24.949602 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:15:24.949583    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a1c27fab208628df1cdbecbf3e3fa72024cac55eea2b4bbae050e7973f43083\": container with ID starting with 7a1c27fab208628df1cdbecbf3e3fa72024cac55eea2b4bbae050e7973f43083 not found: ID does not exist" containerID="7a1c27fab208628df1cdbecbf3e3fa72024cac55eea2b4bbae050e7973f43083"
Sep 10 21:15:24.949649 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.949611    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a1c27fab208628df1cdbecbf3e3fa72024cac55eea2b4bbae050e7973f43083"} err="failed to get container status \"7a1c27fab208628df1cdbecbf3e3fa72024cac55eea2b4bbae050e7973f43083\": rpc error: code = NotFound desc = could not find container \"7a1c27fab208628df1cdbecbf3e3fa72024cac55eea2b4bbae050e7973f43083\": container with ID starting with 7a1c27fab208628df1cdbecbf3e3fa72024cac55eea2b4bbae050e7973f43083 not found: ID does not exist"
Sep 10 21:15:24.960536 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.960513    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"]
Sep 10 21:15:24.964068 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:24.964048    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-6324ec/noten-model-6324ec-kserve-789bc5b979-cf2z8"]
Sep 10 21:15:25.443832 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:25.443802    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4994cae-5984-4f3f-bce1-a6142ed7f466" path="/var/lib/kubelet/pods/b4994cae-5984-4f3f-bce1-a6142ed7f466/volumes"
Sep 10 21:15:27.527577 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:27.527524    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:15:29.505934 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.505906    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:15:29.527755 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.527678    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istio-envoy\") pod \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") "
Sep 10 21:15:29.527755 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.527734    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fb5pt\" (UniqueName: \"kubernetes.io/projected/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-kube-api-access-fb5pt\") pod \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") "
Sep 10 21:15:29.527966 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.527761    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istio-podinfo\") pod \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") "
Sep 10 21:15:29.527966 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.527796    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istiod-ca-cert\") pod \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") "
Sep 10 21:15:29.527966 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.527821    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-workload-certs\") pod \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") "
Sep 10 21:15:29.527966 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.527880    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-workload-socket\") pod \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") "
Sep 10 21:15:29.527966 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.527934    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istio-data\") pod \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") "
Sep 10 21:15:29.527966 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.527958    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-credential-socket\") pod \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") "
Sep 10 21:15:29.528300 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.528000    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istio-token\") pod \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\" (UID: \"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04\") "
Sep 10 21:15:29.528300 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.528173    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04" (UID: "9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:29.528300 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.528230    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04" (UID: "9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:29.528300 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.528269    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04" (UID: "9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:15:29.528503 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.528383    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istiod-ca-cert\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:29.528503 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.528405    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-workload-certs\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:29.528503 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.528418    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-workload-socket\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:29.528662 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.528622    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04" (UID: "9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:29.528712 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.528651    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istio-data" (OuterVolumeSpecName: "istio-data") pod "9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04" (UID: "9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:29.530404 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.530372    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04" (UID: "9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:29.530610 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.530558    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istio-token" (OuterVolumeSpecName: "istio-token") pod "9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04" (UID: "9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:15:29.530734 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.530707    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-kube-api-access-fb5pt" (OuterVolumeSpecName: "kube-api-access-fb5pt") pod "9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04" (UID: "9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04"). InnerVolumeSpecName "kube-api-access-fb5pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:15:29.531531 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.530888    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04" (UID: "9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:15:29.629750 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.629722    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istio-data\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:29.629750 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.629748    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-credential-socket\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:29.629967 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.629761    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istio-token\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:29.629967 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.629771    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istio-envoy\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:29.629967 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.629781    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb5pt\" (UniqueName: \"kubernetes.io/projected/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-kube-api-access-fb5pt\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:29.629967 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.629792    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04-istio-podinfo\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:29.966553 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.966514    2572 generic.go:356] "Generic (PLEG): container finished" podID="9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04" containerID="523b4d2e352411d602dac1297c69cff971d38868c334f6a1de6237183da1636d" exitCode=0
Sep 10 21:15:29.966716 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.966616    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"
Sep 10 21:15:29.966716 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.966655    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl" event={"ID":"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04","Type":"ContainerDied","Data":"523b4d2e352411d602dac1297c69cff971d38868c334f6a1de6237183da1636d"}
Sep 10 21:15:29.966716 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.966715    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl" event={"ID":"9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04","Type":"ContainerDied","Data":"120b7c4a9580b0f91c6416bbab3a360514ead45343bef005d7d645af67466646"}
Sep 10 21:15:29.966833 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.966740    2572 scope.go:117] "RemoveContainer" containerID="523b4d2e352411d602dac1297c69cff971d38868c334f6a1de6237183da1636d"
Sep 10 21:15:29.976234 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.976220    2572 scope.go:117] "RemoveContainer" containerID="47343e050aa1cd3af2286ac1ed77e1d875c6970911d227ef2adc782298891d31"
Sep 10 21:15:29.984320 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.984302    2572 scope.go:117] "RemoveContainer" containerID="523b4d2e352411d602dac1297c69cff971d38868c334f6a1de6237183da1636d"
Sep 10 21:15:29.984618 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:15:29.984597    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"523b4d2e352411d602dac1297c69cff971d38868c334f6a1de6237183da1636d\": container with ID starting with 523b4d2e352411d602dac1297c69cff971d38868c334f6a1de6237183da1636d not found: ID does not exist" containerID="523b4d2e352411d602dac1297c69cff971d38868c334f6a1de6237183da1636d"
Sep 10 21:15:29.984744 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.984621    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"523b4d2e352411d602dac1297c69cff971d38868c334f6a1de6237183da1636d"} err="failed to get container status \"523b4d2e352411d602dac1297c69cff971d38868c334f6a1de6237183da1636d\": rpc error: code = NotFound desc = could not find container \"523b4d2e352411d602dac1297c69cff971d38868c334f6a1de6237183da1636d\": container with ID starting with 523b4d2e352411d602dac1297c69cff971d38868c334f6a1de6237183da1636d not found: ID does not exist"
Sep 10 21:15:29.984744 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.984636    2572 scope.go:117] "RemoveContainer" containerID="47343e050aa1cd3af2286ac1ed77e1d875c6970911d227ef2adc782298891d31"
Sep 10 21:15:29.984911 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:15:29.984893    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47343e050aa1cd3af2286ac1ed77e1d875c6970911d227ef2adc782298891d31\": container with ID starting with 47343e050aa1cd3af2286ac1ed77e1d875c6970911d227ef2adc782298891d31 not found: ID does not exist" containerID="47343e050aa1cd3af2286ac1ed77e1d875c6970911d227ef2adc782298891d31"
Sep 10 21:15:29.985000 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:29.984916    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47343e050aa1cd3af2286ac1ed77e1d875c6970911d227ef2adc782298891d31"} err="failed to get container status \"47343e050aa1cd3af2286ac1ed77e1d875c6970911d227ef2adc782298891d31\": rpc error: code = NotFound desc = could not find container \"47343e050aa1cd3af2286ac1ed77e1d875c6970911d227ef2adc782298891d31\": container with ID starting with 47343e050aa1cd3af2286ac1ed77e1d875c6970911d227ef2adc782298891d31 not found: ID does not exist"
Sep 10 21:15:30.019211 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.019189    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"]
Sep 10 21:15:30.026210 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.026189    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7eaf1f-openshift-default-845cf49dcb-cjmwl"]
Sep 10 21:15:30.524014 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.523965    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw"]
Sep 10 21:15:30.524467 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.524295    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4994cae-5984-4f3f-bce1-a6142ed7f466" containerName="main"
Sep 10 21:15:30.524467 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.524306    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4994cae-5984-4f3f-bce1-a6142ed7f466" containerName="main"
Sep 10 21:15:30.524467 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.524324    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04" containerName="istio-proxy"
Sep 10 21:15:30.524467 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.524330    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04" containerName="istio-proxy"
Sep 10 21:15:30.524467 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.524339    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e35a47b7-668d-4c95-b2b8-446d5e762a8c" containerName="istio-proxy"
Sep 10 21:15:30.524467 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.524344    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e35a47b7-668d-4c95-b2b8-446d5e762a8c" containerName="istio-proxy"
Sep 10 21:15:30.524467 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.524354    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04" containerName="istio-proxy"
Sep 10 21:15:30.524467 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.524359    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04" containerName="istio-proxy"
Sep 10 21:15:30.524467 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.524415    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04" containerName="istio-proxy"
Sep 10 21:15:30.524467 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.524422    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04" containerName="istio-proxy"
Sep 10 21:15:30.524467 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.524427    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4994cae-5984-4f3f-bce1-a6142ed7f466" containerName="main"
Sep 10 21:15:30.528699 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.528676    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw"
Sep 10 21:15:30.532138 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.532118    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-af7150\""
Sep 10 21:15:30.533147 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.533128    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-af7150-dockercfg-mfh6j\""
Sep 10 21:15:30.540389 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.540367    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd"]
Sep 10 21:15:30.543803 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.543785    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw"]
Sep 10 21:15:30.543899 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.543860    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd"
Sep 10 21:15:30.559183 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.559160    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd"]
Sep 10 21:15:30.638254 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.638225    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crm9b\" (UniqueName: \"kubernetes.io/projected/a87ef12f-1556-4db9-9c07-5c655e6dbd80-kube-api-access-crm9b\") pod \"payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd\" (UID: \"a87ef12f-1556-4db9-9c07-5c655e6dbd80\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd"
Sep 10 21:15:30.638356 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.638272    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87ef12f-1556-4db9-9c07-5c655e6dbd80-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd\" (UID: \"a87ef12f-1556-4db9-9c07-5c655e6dbd80\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd"
Sep 10 21:15:30.638414 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.638372    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c984132-b24c-46f5-8808-4557e7cd8bc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw\" (UID: \"2c984132-b24c-46f5-8808-4557e7cd8bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw"
Sep 10 21:15:30.638456 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.638439    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj8v5\" (UniqueName: \"kubernetes.io/projected/2c984132-b24c-46f5-8808-4557e7cd8bc7-kube-api-access-bj8v5\") pod \"payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw\" (UID: \"2c984132-b24c-46f5-8808-4557e7cd8bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw"
Sep 10 21:15:30.739233 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.739203    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj8v5\" (UniqueName: \"kubernetes.io/projected/2c984132-b24c-46f5-8808-4557e7cd8bc7-kube-api-access-bj8v5\") pod \"payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw\" (UID: \"2c984132-b24c-46f5-8808-4557e7cd8bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw"
Sep 10 21:15:30.739338 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.739287    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crm9b\" (UniqueName: \"kubernetes.io/projected/a87ef12f-1556-4db9-9c07-5c655e6dbd80-kube-api-access-crm9b\") pod \"payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd\" (UID: \"a87ef12f-1556-4db9-9c07-5c655e6dbd80\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd"
Sep 10 21:15:30.739338 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.739326    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87ef12f-1556-4db9-9c07-5c655e6dbd80-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd\" (UID: \"a87ef12f-1556-4db9-9c07-5c655e6dbd80\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd"
Sep 10 21:15:30.739489 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.739470    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c984132-b24c-46f5-8808-4557e7cd8bc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw\" (UID: \"2c984132-b24c-46f5-8808-4557e7cd8bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw"
Sep 10 21:15:30.740006 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.739973    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87ef12f-1556-4db9-9c07-5c655e6dbd80-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd\" (UID: \"a87ef12f-1556-4db9-9c07-5c655e6dbd80\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd"
Sep 10 21:15:30.740103 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.740088    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c984132-b24c-46f5-8808-4557e7cd8bc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw\" (UID: \"2c984132-b24c-46f5-8808-4557e7cd8bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw"
Sep 10 21:15:30.747609 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.747588    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj8v5\" (UniqueName: \"kubernetes.io/projected/2c984132-b24c-46f5-8808-4557e7cd8bc7-kube-api-access-bj8v5\") pod \"payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw\" (UID: \"2c984132-b24c-46f5-8808-4557e7cd8bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw"
Sep 10 21:15:30.747707 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.747650    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crm9b\" (UniqueName: \"kubernetes.io/projected/a87ef12f-1556-4db9-9c07-5c655e6dbd80-kube-api-access-crm9b\") pod \"payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd\" (UID: \"a87ef12f-1556-4db9-9c07-5c655e6dbd80\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd"
Sep 10 21:15:30.839442 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.839418    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw"
Sep 10 21:15:30.856824 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:30.856802    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd"
Sep 10 21:15:31.119079 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:31.119009    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw"]
Sep 10 21:15:31.122464 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:15:31.122440    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c984132_b24c_46f5_8808_4557e7cd8bc7.slice/crio-cbcc983d622b864b8b6d26bfbdd54a43421562df7059b58e367dbf83d330c2af WatchSource:0}: Error finding container cbcc983d622b864b8b6d26bfbdd54a43421562df7059b58e367dbf83d330c2af: Status 404 returned error can't find the container with id cbcc983d622b864b8b6d26bfbdd54a43421562df7059b58e367dbf83d330c2af
Sep 10 21:15:31.148398 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:31.148376    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd"]
Sep 10 21:15:31.154032 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:15:31.154006    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87ef12f_1556_4db9_9c07_5c655e6dbd80.slice/crio-b1edd721abeb751077547d5b007fabf7bd0e1b8c285f2b43963a6c44dbbb6427 WatchSource:0}: Error finding container b1edd721abeb751077547d5b007fabf7bd0e1b8c285f2b43963a6c44dbbb6427: Status 404 returned error can't find the container with id b1edd721abeb751077547d5b007fabf7bd0e1b8c285f2b43963a6c44dbbb6427
Sep 10 21:15:31.439235 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:31.439170    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04" path="/var/lib/kubelet/pods/9b28f1bf-6fbf-4a14-80fa-990f7ea4ff04/volumes"
Sep 10 21:15:31.895156 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:31.895126    2572 patch_prober.go:28] interesting pod/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 10 21:15:31.895520 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:31.895175    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65" podUID="454b3bae-5f97-42d4-b202-5a983a492099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 10 21:15:31.980506 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:31.980477    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw" event={"ID":"2c984132-b24c-46f5-8808-4557e7cd8bc7","Type":"ContainerStarted","Data":"5e97d3f7e1e7544d6a18f5ae82f688c337e3aad49775b0a88a0ddfa7166eba06"}
Sep 10 21:15:31.980622 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:31.980527    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw" event={"ID":"2c984132-b24c-46f5-8808-4557e7cd8bc7","Type":"ContainerStarted","Data":"cbcc983d622b864b8b6d26bfbdd54a43421562df7059b58e367dbf83d330c2af"}
Sep 10 21:15:31.980622 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:31.980563    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw"
Sep 10 21:15:31.982125 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:31.982105    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd" event={"ID":"a87ef12f-1556-4db9-9c07-5c655e6dbd80","Type":"ContainerStarted","Data":"b827cf4a110abb139d3bf533e75f5970645e24c466e861b7b6d91f3bcc097d89"}
Sep 10 21:15:31.982221 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:31.982148    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd"
Sep 10 21:15:31.982221 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:31.982164    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd" event={"ID":"a87ef12f-1556-4db9-9c07-5c655e6dbd80","Type":"ContainerStarted","Data":"b1edd721abeb751077547d5b007fabf7bd0e1b8c285f2b43963a6c44dbbb6427"}
Sep 10 21:15:31.983649 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:31.983631    2572 generic.go:356] "Generic (PLEG): container finished" podID="454b3bae-5f97-42d4-b202-5a983a492099" containerID="c80ba4f83593398727e28f924a3a0bbe949a799cd42212d89bb51c5e07c6d8fb" exitCode=0
Sep 10 21:15:31.983725 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:31.983673    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65" event={"ID":"454b3bae-5f97-42d4-b202-5a983a492099","Type":"ContainerDied","Data":"c80ba4f83593398727e28f924a3a0bbe949a799cd42212d89bb51c5e07c6d8fb"}
Sep 10 21:15:31.983949 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:31.983935    2572 scope.go:117] "RemoveContainer" containerID="c80ba4f83593398727e28f924a3a0bbe949a799cd42212d89bb51c5e07c6d8fb"
Sep 10 21:15:31.985083 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:31.985050    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:15:31.985164 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:31.985136    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:15:31.985206 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:31.985179    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:15:31.997830 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:31.997787    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw" podStartSLOduration=1.997773992 podStartE2EDuration="1.997773992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:31.99695644 +0000 UTC m=+1117.209421360" watchObservedRunningTime="2026-09-10 21:15:31.997773992 +0000 UTC m=+1117.210238911"
Sep 10 21:15:32.040666 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:32.040115    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd" podStartSLOduration=2.040089685 podStartE2EDuration="2.040089685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:32.032231119 +0000 UTC m=+1117.244696039" watchObservedRunningTime="2026-09-10 21:15:32.040089685 +0000 UTC m=+1117.252554606"
Sep 10 21:15:32.990603 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:32.990570    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65" event={"ID":"454b3bae-5f97-42d4-b202-5a983a492099","Type":"ContainerStarted","Data":"902da23eab9ba51238fdf1f13eb3fbea35662756e04c3600b58636923c9a9c5c"}
Sep 10 21:15:33.526651 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:33.526613    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:33.527174 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:33.526850    2572 patch_prober.go:28] interesting pod/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 10 21:15:33.527174 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:33.526912    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65" podUID="454b3bae-5f97-42d4-b202-5a983a492099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 10 21:15:34.527620 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:34.527527    2572 patch_prober.go:28] interesting pod/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 10 21:15:34.527620 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:34.527606    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65" podUID="454b3bae-5f97-42d4-b202-5a983a492099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 10 21:15:34.561661 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:34.561620    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:15:35.527358 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:35.527317    2572 patch_prober.go:28] interesting pod/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 10 21:15:35.527558 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:35.527381    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65" podUID="454b3bae-5f97-42d4-b202-5a983a492099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 10 21:15:36.527411 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:36.527378    2572 patch_prober.go:28] interesting pod/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 10 21:15:36.528028 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:36.527968    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65" podUID="454b3bae-5f97-42d4-b202-5a983a492099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 10 21:15:37.527475 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:37.527431    2572 patch_prober.go:28] interesting pod/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 10 21:15:37.528005 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:37.527499    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65" podUID="454b3bae-5f97-42d4-b202-5a983a492099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 10 21:15:38.530968 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:38.530937    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:38.531484 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:38.531291    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:38.532016 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:38.531976    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:15:42.993040 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:42.993002    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw"
Sep 10 21:15:42.993478 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:42.993211    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd"
Sep 10 21:15:50.257351 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:50.257322    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-899bf7e2-n2gvp"]
Sep 10 21:15:50.261787 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:50.261771    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-899bf7e2-n2gvp"
Sep 10 21:15:50.264104 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:50.264086    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5c7x\""
Sep 10 21:15:50.264104 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:50.264096    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:15:50.268360 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:50.268337    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-899bf7e2-n2gvp"]
Sep 10 21:15:50.399610 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:50.399583    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bfc611-b27a-4542-97b0-267de3e2053a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-899bf7e2-n2gvp\" (UID: \"b6bfc611-b27a-4542-97b0-267de3e2053a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-899bf7e2-n2gvp"
Sep 10 21:15:50.500777 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:50.500745    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bfc611-b27a-4542-97b0-267de3e2053a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-899bf7e2-n2gvp\" (UID: \"b6bfc611-b27a-4542-97b0-267de3e2053a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-899bf7e2-n2gvp"
Sep 10 21:15:50.501360 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:50.501340    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bfc611-b27a-4542-97b0-267de3e2053a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-899bf7e2-n2gvp\" (UID: \"b6bfc611-b27a-4542-97b0-267de3e2053a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-899bf7e2-n2gvp"
Sep 10 21:15:50.572956 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:50.572936    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-899bf7e2-n2gvp"
Sep 10 21:15:50.846620 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:50.846598    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-899bf7e2-n2gvp"]
Sep 10 21:15:50.848853 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:15:50.848824    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6bfc611_b27a_4542_97b0_267de3e2053a.slice/crio-448e1c665fdc69e352bedbe4830788debbc45830ebbc3f078ba37acff5a54168 WatchSource:0}: Error finding container 448e1c665fdc69e352bedbe4830788debbc45830ebbc3f078ba37acff5a54168: Status 404 returned error can't find the container with id 448e1c665fdc69e352bedbe4830788debbc45830ebbc3f078ba37acff5a54168
Sep 10 21:15:51.084542 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:51.084510    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-899bf7e2-n2gvp" event={"ID":"b6bfc611-b27a-4542-97b0-267de3e2053a","Type":"ContainerStarted","Data":"448e1c665fdc69e352bedbe4830788debbc45830ebbc3f078ba37acff5a54168"}
Sep 10 21:15:52.090903 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:52.090868    2572 generic.go:356] "Generic (PLEG): container finished" podID="b6bfc611-b27a-4542-97b0-267de3e2053a" containerID="89dfd12ba19c600c2daa71a636507629d79302f5493668c332987b8824f30c5c" exitCode=0
Sep 10 21:15:52.091290 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:52.090914    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-899bf7e2-n2gvp" event={"ID":"b6bfc611-b27a-4542-97b0-267de3e2053a","Type":"ContainerDied","Data":"89dfd12ba19c600c2daa71a636507629d79302f5493668c332987b8824f30c5c"}
Sep 10 21:15:53.225854 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:53.225828    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-899bf7e2-n2gvp"
Sep 10 21:15:53.323047 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:53.323020    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bfc611-b27a-4542-97b0-267de3e2053a-maas-api-service-ca\") pod \"b6bfc611-b27a-4542-97b0-267de3e2053a\" (UID: \"b6bfc611-b27a-4542-97b0-267de3e2053a\") "
Sep 10 21:15:53.323371 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:53.323345    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bfc611-b27a-4542-97b0-267de3e2053a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b6bfc611-b27a-4542-97b0-267de3e2053a" (UID: "b6bfc611-b27a-4542-97b0-267de3e2053a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:15:53.423584 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:53.423522    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6bfc611-b27a-4542-97b0-267de3e2053a-maas-api-service-ca\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:15:54.101489 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:54.101461    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-899bf7e2-n2gvp"
Sep 10 21:15:54.101710 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:54.101465    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-899bf7e2-n2gvp" event={"ID":"b6bfc611-b27a-4542-97b0-267de3e2053a","Type":"ContainerDied","Data":"448e1c665fdc69e352bedbe4830788debbc45830ebbc3f078ba37acff5a54168"}
Sep 10 21:15:54.101842 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:54.101738    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="448e1c665fdc69e352bedbe4830788debbc45830ebbc3f078ba37acff5a54168"
Sep 10 21:15:58.621853 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.621816    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f"]
Sep 10 21:15:58.622295 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.622185    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6bfc611-b27a-4542-97b0-267de3e2053a" containerName="revoke-keys"
Sep 10 21:15:58.622295 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.622197    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6bfc611-b27a-4542-97b0-267de3e2053a" containerName="revoke-keys"
Sep 10 21:15:58.622295 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.622258    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6bfc611-b27a-4542-97b0-267de3e2053a" containerName="revoke-keys"
Sep 10 21:15:58.626645 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.626628    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f"
Sep 10 21:15:58.629279 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.629256    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-7810bd-dockercfg-j9mjv\""
Sep 10 21:15:58.629619 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.629602    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-7810bd\""
Sep 10 21:15:58.636767 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.636741    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f"]
Sep 10 21:15:58.660067 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.660042    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd"]
Sep 10 21:15:58.663438 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.663423    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd"
Sep 10 21:15:58.673268 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.673250    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd"]
Sep 10 21:15:58.762899 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.762872    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sx27\" (UniqueName: \"kubernetes.io/projected/9e97ece3-54ca-4af5-baa3-7a1472975414-kube-api-access-5sx27\") pod \"payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f\" (UID: \"9e97ece3-54ca-4af5-baa3-7a1472975414\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f"
Sep 10 21:15:58.763062 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.762908    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd\" (UID: \"766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd"
Sep 10 21:15:58.763062 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.762968    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzxf8\" (UniqueName: \"kubernetes.io/projected/766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c-kube-api-access-nzxf8\") pod \"payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd\" (UID: \"766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd"
Sep 10 21:15:58.763154 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.763112    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e97ece3-54ca-4af5-baa3-7a1472975414-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f\" (UID: \"9e97ece3-54ca-4af5-baa3-7a1472975414\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f"
Sep 10 21:15:58.864372 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.864337    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e97ece3-54ca-4af5-baa3-7a1472975414-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f\" (UID: \"9e97ece3-54ca-4af5-baa3-7a1472975414\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f"
Sep 10 21:15:58.864372 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.864377    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sx27\" (UniqueName: \"kubernetes.io/projected/9e97ece3-54ca-4af5-baa3-7a1472975414-kube-api-access-5sx27\") pod \"payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f\" (UID: \"9e97ece3-54ca-4af5-baa3-7a1472975414\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f"
Sep 10 21:15:58.864558 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.864401    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd\" (UID: \"766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd"
Sep 10 21:15:58.864558 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.864439    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzxf8\" (UniqueName: \"kubernetes.io/projected/766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c-kube-api-access-nzxf8\") pod \"payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd\" (UID: \"766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd"
Sep 10 21:15:58.865068 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.865047    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e97ece3-54ca-4af5-baa3-7a1472975414-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f\" (UID: \"9e97ece3-54ca-4af5-baa3-7a1472975414\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f"
Sep 10 21:15:58.865194 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.865174    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd\" (UID: \"766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd"
Sep 10 21:15:58.873321 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.873268    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzxf8\" (UniqueName: \"kubernetes.io/projected/766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c-kube-api-access-nzxf8\") pod \"payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd\" (UID: \"766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd"
Sep 10 21:15:58.873401 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.873330    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sx27\" (UniqueName: \"kubernetes.io/projected/9e97ece3-54ca-4af5-baa3-7a1472975414-kube-api-access-5sx27\") pod \"payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f\" (UID: \"9e97ece3-54ca-4af5-baa3-7a1472975414\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f"
Sep 10 21:15:58.937661 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.937643    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f"
Sep 10 21:15:58.972813 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:58.972787    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd"
Sep 10 21:15:59.213685 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:59.213664    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f"]
Sep 10 21:15:59.215596 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:15:59.215574    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e97ece3_54ca_4af5_baa3_7a1472975414.slice/crio-d5de4c86e6dcfd00fb33d073b4b668475dc7e3df4bf11bc2703dcc9f5a1d229f WatchSource:0}: Error finding container d5de4c86e6dcfd00fb33d073b4b668475dc7e3df4bf11bc2703dcc9f5a1d229f: Status 404 returned error can't find the container with id d5de4c86e6dcfd00fb33d073b4b668475dc7e3df4bf11bc2703dcc9f5a1d229f
Sep 10 21:15:59.259805 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:15:59.259782    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd"]
Sep 10 21:15:59.262505 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:15:59.262483    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766dfa93_eeda_4c2f_8b32_52d8f5dd0b9c.slice/crio-98e5ad485d4426a9143177dbbeed2a114a6dcdd4afceaddeb7572fb8b10c5787 WatchSource:0}: Error finding container 98e5ad485d4426a9143177dbbeed2a114a6dcdd4afceaddeb7572fb8b10c5787: Status 404 returned error can't find the container with id 98e5ad485d4426a9143177dbbeed2a114a6dcdd4afceaddeb7572fb8b10c5787
Sep 10 21:16:00.138590 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:00.138557    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f" event={"ID":"9e97ece3-54ca-4af5-baa3-7a1472975414","Type":"ContainerStarted","Data":"604c6e202566496e6240cad3776b6aec7c916e939ecd207673e8f83e7345e5c4"}
Sep 10 21:16:00.139055 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:00.138623    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f" event={"ID":"9e97ece3-54ca-4af5-baa3-7a1472975414","Type":"ContainerStarted","Data":"d5de4c86e6dcfd00fb33d073b4b668475dc7e3df4bf11bc2703dcc9f5a1d229f"}
Sep 10 21:16:00.139055 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:00.138655    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f"
Sep 10 21:16:00.140100 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:00.140078    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd" event={"ID":"766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c","Type":"ContainerStarted","Data":"1d225bf84adac9024fdcc477dca362a467fa0423d6cf46db326c1989c6940cd0"}
Sep 10 21:16:00.140216 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:00.140118    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd" event={"ID":"766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c","Type":"ContainerStarted","Data":"98e5ad485d4426a9143177dbbeed2a114a6dcdd4afceaddeb7572fb8b10c5787"}
Sep 10 21:16:00.140216 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:00.140194    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd"
Sep 10 21:16:00.156291 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:00.156251    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f" podStartSLOduration=2.156240071 podStartE2EDuration="2.156240071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:16:00.154348036 +0000 UTC m=+1145.366812955" watchObservedRunningTime="2026-09-10 21:16:00.156240071 +0000 UTC m=+1145.368705019"
Sep 10 21:16:00.172518 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:00.172481    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd" podStartSLOduration=2.172468312 podStartE2EDuration="2.172468312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:16:00.170518019 +0000 UTC m=+1145.382982938" watchObservedRunningTime="2026-09-10 21:16:00.172468312 +0000 UTC m=+1145.384933230"
Sep 10 21:16:00.630250 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:00.630218    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-899bf7e2-n2gvp"]
Sep 10 21:16:00.633559 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:00.633539    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-899bf7e2-n2gvp"]
Sep 10 21:16:01.441029 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:01.440998    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6bfc611-b27a-4542-97b0-267de3e2053a" path="/var/lib/kubelet/pods/b6bfc611-b27a-4542-97b0-267de3e2053a/volumes"
Sep 10 21:16:02.712682 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:02.712650    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:16:07.353049 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:07.353008    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"]
Sep 10 21:16:07.353457 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:07.353270    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65" podUID="454b3bae-5f97-42d4-b202-5a983a492099" containerName="istio-proxy" containerID="cri-o://902da23eab9ba51238fdf1f13eb3fbea35662756e04c3600b58636923c9a9c5c" gracePeriod=30
Sep 10 21:16:08.533536 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:08.533498    2572 patch_prober.go:28] interesting pod/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 21:16:08.533932 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:08.533580    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65" podUID="454b3bae-5f97-42d4-b202-5a983a492099" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 21:16:11.147677 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:11.147640    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f"
Sep 10 21:16:11.148180 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:11.148118    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd"
Sep 10 21:16:12.612383 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.612360    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:16:12.775830 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.775763    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-istio-data\") pod \"454b3bae-5f97-42d4-b202-5a983a492099\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") "
Sep 10 21:16:12.775830 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.775801    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/454b3bae-5f97-42d4-b202-5a983a492099-istio-token\") pod \"454b3bae-5f97-42d4-b202-5a983a492099\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") "
Sep 10 21:16:12.776060 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.775883    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdtfl\" (UniqueName: \"kubernetes.io/projected/454b3bae-5f97-42d4-b202-5a983a492099-kube-api-access-gdtfl\") pod \"454b3bae-5f97-42d4-b202-5a983a492099\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") "
Sep 10 21:16:12.776060 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.775909    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/454b3bae-5f97-42d4-b202-5a983a492099-istiod-ca-cert\") pod \"454b3bae-5f97-42d4-b202-5a983a492099\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") "
Sep 10 21:16:12.776060 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.776014    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-credential-socket\") pod \"454b3bae-5f97-42d4-b202-5a983a492099\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") "
Sep 10 21:16:12.776060 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.776028    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-istio-data" (OuterVolumeSpecName: "istio-data") pod "454b3bae-5f97-42d4-b202-5a983a492099" (UID: "454b3bae-5f97-42d4-b202-5a983a492099"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:12.776275 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.776065    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-workload-certs\") pod \"454b3bae-5f97-42d4-b202-5a983a492099\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") "
Sep 10 21:16:12.776275 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.776091    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-workload-socket\") pod \"454b3bae-5f97-42d4-b202-5a983a492099\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") "
Sep 10 21:16:12.776275 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.776134    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-istio-envoy\") pod \"454b3bae-5f97-42d4-b202-5a983a492099\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") "
Sep 10 21:16:12.776275 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.776175    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/454b3bae-5f97-42d4-b202-5a983a492099-istio-podinfo\") pod \"454b3bae-5f97-42d4-b202-5a983a492099\" (UID: \"454b3bae-5f97-42d4-b202-5a983a492099\") "
Sep 10 21:16:12.776457 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.776273    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "454b3bae-5f97-42d4-b202-5a983a492099" (UID: "454b3bae-5f97-42d4-b202-5a983a492099"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:12.776457 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.776360    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "454b3bae-5f97-42d4-b202-5a983a492099" (UID: "454b3bae-5f97-42d4-b202-5a983a492099"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:12.776457 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.776374    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "454b3bae-5f97-42d4-b202-5a983a492099" (UID: "454b3bae-5f97-42d4-b202-5a983a492099"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:12.776457 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.776396    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/454b3bae-5f97-42d4-b202-5a983a492099-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "454b3bae-5f97-42d4-b202-5a983a492099" (UID: "454b3bae-5f97-42d4-b202-5a983a492099"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:16:12.776606 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.776495    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-istio-data\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:16:12.776697 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.776682    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/454b3bae-5f97-42d4-b202-5a983a492099-istiod-ca-cert\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:16:12.776739 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.776702    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-credential-socket\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:16:12.776739 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.776717    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-workload-certs\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:16:12.776739 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.776730    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-workload-socket\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:16:12.778038 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.778016    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454b3bae-5f97-42d4-b202-5a983a492099-istio-token" (OuterVolumeSpecName: "istio-token") pod "454b3bae-5f97-42d4-b202-5a983a492099" (UID: "454b3bae-5f97-42d4-b202-5a983a492099"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:16:12.778189 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.778171    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454b3bae-5f97-42d4-b202-5a983a492099-kube-api-access-gdtfl" (OuterVolumeSpecName: "kube-api-access-gdtfl") pod "454b3bae-5f97-42d4-b202-5a983a492099" (UID: "454b3bae-5f97-42d4-b202-5a983a492099"). InnerVolumeSpecName "kube-api-access-gdtfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:16:12.778304 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.778285    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "454b3bae-5f97-42d4-b202-5a983a492099" (UID: "454b3bae-5f97-42d4-b202-5a983a492099"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:12.778517 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.778501    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/454b3bae-5f97-42d4-b202-5a983a492099-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "454b3bae-5f97-42d4-b202-5a983a492099" (UID: "454b3bae-5f97-42d4-b202-5a983a492099"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:16:12.877652 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.877631    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/454b3bae-5f97-42d4-b202-5a983a492099-istio-token\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:16:12.877652 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.877650    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdtfl\" (UniqueName: \"kubernetes.io/projected/454b3bae-5f97-42d4-b202-5a983a492099-kube-api-access-gdtfl\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:16:12.877781 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.877662    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/454b3bae-5f97-42d4-b202-5a983a492099-istio-envoy\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:16:12.877781 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:12.877670    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/454b3bae-5f97-42d4-b202-5a983a492099-istio-podinfo\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:16:13.207155 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:13.207124    2572 generic.go:356] "Generic (PLEG): container finished" podID="454b3bae-5f97-42d4-b202-5a983a492099" containerID="902da23eab9ba51238fdf1f13eb3fbea35662756e04c3600b58636923c9a9c5c" exitCode=0
Sep 10 21:16:13.207285 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:13.207176    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"
Sep 10 21:16:13.207285 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:13.207211    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65" event={"ID":"454b3bae-5f97-42d4-b202-5a983a492099","Type":"ContainerDied","Data":"902da23eab9ba51238fdf1f13eb3fbea35662756e04c3600b58636923c9a9c5c"}
Sep 10 21:16:13.207285 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:13.207265    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65" event={"ID":"454b3bae-5f97-42d4-b202-5a983a492099","Type":"ContainerDied","Data":"c75f05273bc8d5b9a4f1d9eeed46311af5984c5ef923b7b25260697821bdc1b7"}
Sep 10 21:16:13.207393 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:13.207291    2572 scope.go:117] "RemoveContainer" containerID="902da23eab9ba51238fdf1f13eb3fbea35662756e04c3600b58636923c9a9c5c"
Sep 10 21:16:13.216561 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:13.216548    2572 scope.go:117] "RemoveContainer" containerID="c80ba4f83593398727e28f924a3a0bbe949a799cd42212d89bb51c5e07c6d8fb"
Sep 10 21:16:13.224595 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:13.224577    2572 scope.go:117] "RemoveContainer" containerID="902da23eab9ba51238fdf1f13eb3fbea35662756e04c3600b58636923c9a9c5c"
Sep 10 21:16:13.224868 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:16:13.224849    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"902da23eab9ba51238fdf1f13eb3fbea35662756e04c3600b58636923c9a9c5c\": container with ID starting with 902da23eab9ba51238fdf1f13eb3fbea35662756e04c3600b58636923c9a9c5c not found: ID does not exist" containerID="902da23eab9ba51238fdf1f13eb3fbea35662756e04c3600b58636923c9a9c5c"
Sep 10 21:16:13.224956 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:13.224877    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902da23eab9ba51238fdf1f13eb3fbea35662756e04c3600b58636923c9a9c5c"} err="failed to get container status \"902da23eab9ba51238fdf1f13eb3fbea35662756e04c3600b58636923c9a9c5c\": rpc error: code = NotFound desc = could not find container \"902da23eab9ba51238fdf1f13eb3fbea35662756e04c3600b58636923c9a9c5c\": container with ID starting with 902da23eab9ba51238fdf1f13eb3fbea35662756e04c3600b58636923c9a9c5c not found: ID does not exist"
Sep 10 21:16:13.224956 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:13.224894    2572 scope.go:117] "RemoveContainer" containerID="c80ba4f83593398727e28f924a3a0bbe949a799cd42212d89bb51c5e07c6d8fb"
Sep 10 21:16:13.225194 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:16:13.225174    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c80ba4f83593398727e28f924a3a0bbe949a799cd42212d89bb51c5e07c6d8fb\": container with ID starting with c80ba4f83593398727e28f924a3a0bbe949a799cd42212d89bb51c5e07c6d8fb not found: ID does not exist" containerID="c80ba4f83593398727e28f924a3a0bbe949a799cd42212d89bb51c5e07c6d8fb"
Sep 10 21:16:13.225249 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:13.225207    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c80ba4f83593398727e28f924a3a0bbe949a799cd42212d89bb51c5e07c6d8fb"} err="failed to get container status \"c80ba4f83593398727e28f924a3a0bbe949a799cd42212d89bb51c5e07c6d8fb\": rpc error: code = NotFound desc = could not find container \"c80ba4f83593398727e28f924a3a0bbe949a799cd42212d89bb51c5e07c6d8fb\": container with ID starting with c80ba4f83593398727e28f924a3a0bbe949a799cd42212d89bb51c5e07c6d8fb not found: ID does not exist"
Sep 10 21:16:13.228106 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:13.228080    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"]
Sep 10 21:16:13.231760 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:13.231738    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-899bf7e2-openshift-default-c89bf9fc6-5rx65"]
Sep 10 21:16:13.438144 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:13.438120    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="454b3bae-5f97-42d4-b202-5a983a492099" path="/var/lib/kubelet/pods/454b3bae-5f97-42d4-b202-5a983a492099/volumes"
Sep 10 21:16:26.756783 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:26.756711    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8"]
Sep 10 21:16:26.757224 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:26.757155    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="454b3bae-5f97-42d4-b202-5a983a492099" containerName="istio-proxy"
Sep 10 21:16:26.757224 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:26.757170    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="454b3bae-5f97-42d4-b202-5a983a492099" containerName="istio-proxy"
Sep 10 21:16:26.757224 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:26.757181    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="454b3bae-5f97-42d4-b202-5a983a492099" containerName="istio-proxy"
Sep 10 21:16:26.757224 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:26.757186    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="454b3bae-5f97-42d4-b202-5a983a492099" containerName="istio-proxy"
Sep 10 21:16:26.757359 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:26.757256    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="454b3bae-5f97-42d4-b202-5a983a492099" containerName="istio-proxy"
Sep 10 21:16:26.760297 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:26.760281    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8"
Sep 10 21:16:26.762692 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:26.762674    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-9cce25\""
Sep 10 21:16:26.762826 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:26.762811    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-9cce25-dockercfg-q95cz\""
Sep 10 21:16:26.773696 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:26.773677    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8"]
Sep 10 21:16:26.787970 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:26.787936    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2rfw\" (UniqueName: \"kubernetes.io/projected/6d19acfd-20c6-4d82-8b56-e84daef72d89-kube-api-access-p2rfw\") pod \"payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8\" (UID: \"6d19acfd-20c6-4d82-8b56-e84daef72d89\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8"
Sep 10 21:16:26.788080 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:26.788004    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d19acfd-20c6-4d82-8b56-e84daef72d89-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8\" (UID: \"6d19acfd-20c6-4d82-8b56-e84daef72d89\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8"
Sep 10 21:16:26.888675 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:26.888654    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2rfw\" (UniqueName: \"kubernetes.io/projected/6d19acfd-20c6-4d82-8b56-e84daef72d89-kube-api-access-p2rfw\") pod \"payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8\" (UID: \"6d19acfd-20c6-4d82-8b56-e84daef72d89\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8"
Sep 10 21:16:26.888773 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:26.888689    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d19acfd-20c6-4d82-8b56-e84daef72d89-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8\" (UID: \"6d19acfd-20c6-4d82-8b56-e84daef72d89\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8"
Sep 10 21:16:26.889302 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:26.889283    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d19acfd-20c6-4d82-8b56-e84daef72d89-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8\" (UID: \"6d19acfd-20c6-4d82-8b56-e84daef72d89\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8"
Sep 10 21:16:26.900825 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:26.900799    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2rfw\" (UniqueName: \"kubernetes.io/projected/6d19acfd-20c6-4d82-8b56-e84daef72d89-kube-api-access-p2rfw\") pod \"payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8\" (UID: \"6d19acfd-20c6-4d82-8b56-e84daef72d89\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8"
Sep 10 21:16:27.070938 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:27.070912    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8"
Sep 10 21:16:27.350486 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:27.350461    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8"]
Sep 10 21:16:27.352178 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:16:27.352153    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d19acfd_20c6_4d82_8b56_e84daef72d89.slice/crio-6cbe481c4f86ca8bee9d0a8979b580a804a4f70a350e24cfcf22ad3e155bfa6f WatchSource:0}: Error finding container 6cbe481c4f86ca8bee9d0a8979b580a804a4f70a350e24cfcf22ad3e155bfa6f: Status 404 returned error can't find the container with id 6cbe481c4f86ca8bee9d0a8979b580a804a4f70a350e24cfcf22ad3e155bfa6f
Sep 10 21:16:28.284345 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:28.284311    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8" event={"ID":"6d19acfd-20c6-4d82-8b56-e84daef72d89","Type":"ContainerStarted","Data":"d0743f5e23fb6844146d7bbaeb1b1880321e6ea4fca0a66b5ad0449ac6158fd7"}
Sep 10 21:16:28.284791 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:28.284378    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8" event={"ID":"6d19acfd-20c6-4d82-8b56-e84daef72d89","Type":"ContainerStarted","Data":"6cbe481c4f86ca8bee9d0a8979b580a804a4f70a350e24cfcf22ad3e155bfa6f"}
Sep 10 21:16:28.284791 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:28.284414    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8"
Sep 10 21:16:28.304102 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:28.304055    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8" podStartSLOduration=2.304040613 podStartE2EDuration="2.304040613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:16:28.303171195 +0000 UTC m=+1173.515636117" watchObservedRunningTime="2026-09-10 21:16:28.304040613 +0000 UTC m=+1173.516505522"
Sep 10 21:16:29.635035 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:29.635001    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:16:35.913172 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:35.913141    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"]
Sep 10 21:16:35.913550 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:35.913538    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="454b3bae-5f97-42d4-b202-5a983a492099" containerName="istio-proxy"
Sep 10 21:16:35.917625 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:35.917608    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:16:35.921076 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:35.921057    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-af7150\"/\"kube-root-ca.crt\""
Sep 10 21:16:35.921800 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:35.921784    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-af7150\"/\"default-dockercfg-cwcjx\""
Sep 10 21:16:35.921902 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:35.921788    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-af7150\"/\"openshift-service-ca.crt\""
Sep 10 21:16:35.921969 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:35.921937    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-af7150\"/\"auth-test-model-af7150-kserve-self-signed-certs\""
Sep 10 21:16:35.937438 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:35.937418    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"]
Sep 10 21:16:35.949749 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:35.949727    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e148d4-91bb-45aa-813f-e4aba8bd1952-model-cache\") pod \"auth-test-model-af7150-kserve-b8c9b7547-bld2f\" (UID: \"42e148d4-91bb-45aa-813f-e4aba8bd1952\") " pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:16:35.949855 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:35.949769    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e148d4-91bb-45aa-813f-e4aba8bd1952-tls-certs\") pod \"auth-test-model-af7150-kserve-b8c9b7547-bld2f\" (UID: \"42e148d4-91bb-45aa-813f-e4aba8bd1952\") " pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:16:35.949855 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:35.949786    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e148d4-91bb-45aa-813f-e4aba8bd1952-tmp-dir\") pod \"auth-test-model-af7150-kserve-b8c9b7547-bld2f\" (UID: \"42e148d4-91bb-45aa-813f-e4aba8bd1952\") " pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:16:35.949855 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:35.949800    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42e148d4-91bb-45aa-813f-e4aba8bd1952-home\") pod \"auth-test-model-af7150-kserve-b8c9b7547-bld2f\" (UID: \"42e148d4-91bb-45aa-813f-e4aba8bd1952\") " pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:16:35.949855 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:35.949816    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fz75\" (UniqueName: \"kubernetes.io/projected/42e148d4-91bb-45aa-813f-e4aba8bd1952-kube-api-access-8fz75\") pod \"auth-test-model-af7150-kserve-b8c9b7547-bld2f\" (UID: \"42e148d4-91bb-45aa-813f-e4aba8bd1952\") " pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:16:35.950039 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:35.949954    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e148d4-91bb-45aa-813f-e4aba8bd1952-dshm\") pod \"auth-test-model-af7150-kserve-b8c9b7547-bld2f\" (UID: \"42e148d4-91bb-45aa-813f-e4aba8bd1952\") " pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:16:36.050452 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:36.050428    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e148d4-91bb-45aa-813f-e4aba8bd1952-tls-certs\") pod \"auth-test-model-af7150-kserve-b8c9b7547-bld2f\" (UID: \"42e148d4-91bb-45aa-813f-e4aba8bd1952\") " pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:16:36.050544 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:36.050455    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e148d4-91bb-45aa-813f-e4aba8bd1952-tmp-dir\") pod \"auth-test-model-af7150-kserve-b8c9b7547-bld2f\" (UID: \"42e148d4-91bb-45aa-813f-e4aba8bd1952\") " pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:16:36.050544 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:36.050471    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42e148d4-91bb-45aa-813f-e4aba8bd1952-home\") pod \"auth-test-model-af7150-kserve-b8c9b7547-bld2f\" (UID: \"42e148d4-91bb-45aa-813f-e4aba8bd1952\") " pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:16:36.050544 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:36.050487    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fz75\" (UniqueName: \"kubernetes.io/projected/42e148d4-91bb-45aa-813f-e4aba8bd1952-kube-api-access-8fz75\") pod \"auth-test-model-af7150-kserve-b8c9b7547-bld2f\" (UID: \"42e148d4-91bb-45aa-813f-e4aba8bd1952\") " pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:16:36.050544 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:36.050537    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e148d4-91bb-45aa-813f-e4aba8bd1952-dshm\") pod \"auth-test-model-af7150-kserve-b8c9b7547-bld2f\" (UID: \"42e148d4-91bb-45aa-813f-e4aba8bd1952\") " pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:16:36.050715 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:36.050558    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e148d4-91bb-45aa-813f-e4aba8bd1952-model-cache\") pod \"auth-test-model-af7150-kserve-b8c9b7547-bld2f\" (UID: \"42e148d4-91bb-45aa-813f-e4aba8bd1952\") " pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:16:36.050854 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:36.050832    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e148d4-91bb-45aa-813f-e4aba8bd1952-tmp-dir\") pod \"auth-test-model-af7150-kserve-b8c9b7547-bld2f\" (UID: \"42e148d4-91bb-45aa-813f-e4aba8bd1952\") " pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:16:36.050924 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:36.050897    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42e148d4-91bb-45aa-813f-e4aba8bd1952-home\") pod \"auth-test-model-af7150-kserve-b8c9b7547-bld2f\" (UID: \"42e148d4-91bb-45aa-813f-e4aba8bd1952\") " pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:16:36.051051 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:36.050937    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e148d4-91bb-45aa-813f-e4aba8bd1952-model-cache\") pod \"auth-test-model-af7150-kserve-b8c9b7547-bld2f\" (UID: \"42e148d4-91bb-45aa-813f-e4aba8bd1952\") " pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:16:36.052845 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:36.052826    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e148d4-91bb-45aa-813f-e4aba8bd1952-dshm\") pod \"auth-test-model-af7150-kserve-b8c9b7547-bld2f\" (UID: \"42e148d4-91bb-45aa-813f-e4aba8bd1952\") " pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:16:36.053029 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:36.053012    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e148d4-91bb-45aa-813f-e4aba8bd1952-tls-certs\") pod \"auth-test-model-af7150-kserve-b8c9b7547-bld2f\" (UID: \"42e148d4-91bb-45aa-813f-e4aba8bd1952\") " pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:16:36.059947 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:36.059925    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fz75\" (UniqueName: \"kubernetes.io/projected/42e148d4-91bb-45aa-813f-e4aba8bd1952-kube-api-access-8fz75\") pod \"auth-test-model-af7150-kserve-b8c9b7547-bld2f\" (UID: \"42e148d4-91bb-45aa-813f-e4aba8bd1952\") " pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:16:36.226806 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:36.226750    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:16:36.497686 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:36.497664    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"]
Sep 10 21:16:36.500174 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:16:36.500135    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42e148d4_91bb_45aa_813f_e4aba8bd1952.slice/crio-adadfbbdb6cf1a83bf08404400024a224d620f8216a74272ee6bdfc1855724ef WatchSource:0}: Error finding container adadfbbdb6cf1a83bf08404400024a224d620f8216a74272ee6bdfc1855724ef: Status 404 returned error can't find the container with id adadfbbdb6cf1a83bf08404400024a224d620f8216a74272ee6bdfc1855724ef
Sep 10 21:16:37.341707 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:37.341668    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f" event={"ID":"42e148d4-91bb-45aa-813f-e4aba8bd1952","Type":"ContainerStarted","Data":"3093823d5d14a8b47643e91afb6c0b5a8963863044a5b2fa16e446b824e77e1f"}
Sep 10 21:16:37.342132 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:37.341725    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f" event={"ID":"42e148d4-91bb-45aa-813f-e4aba8bd1952","Type":"ContainerStarted","Data":"adadfbbdb6cf1a83bf08404400024a224d620f8216a74272ee6bdfc1855724ef"}
Sep 10 21:16:37.363081 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:37.363032    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f" podStartSLOduration=2.363014466 podStartE2EDuration="2.363014466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:16:37.361643511 +0000 UTC m=+1182.574108430" watchObservedRunningTime="2026-09-10 21:16:37.363014466 +0000 UTC m=+1182.575479385"
Sep 10 21:16:39.291182 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:39.291152    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8"
Sep 10 21:16:46.227556 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:46.227520    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:16:46.228010 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:46.227608    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:16:46.240174 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:46.240151    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:16:46.399692 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:46.399664    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:16:48.326858 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:48.326813    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:16:55.062345 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:55.062315    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9"]
Sep 10 21:16:55.065704 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:55.065688    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9"
Sep 10 21:16:55.067973 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:55.067955    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-95d905-dockercfg-4dl29\""
Sep 10 21:16:55.068159 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:55.068142    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-95d905\""
Sep 10 21:16:55.077519 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:55.077494    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9"]
Sep 10 21:16:55.101043 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:55.101018    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf70725b-8a40-4a17-aba0-4c844c611558-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9\" (UID: \"bf70725b-8a40-4a17-aba0-4c844c611558\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9"
Sep 10 21:16:55.101160 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:55.101049    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl8xb\" (UniqueName: \"kubernetes.io/projected/bf70725b-8a40-4a17-aba0-4c844c611558-kube-api-access-jl8xb\") pod \"payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9\" (UID: \"bf70725b-8a40-4a17-aba0-4c844c611558\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9"
Sep 10 21:16:55.201620 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:55.201597    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf70725b-8a40-4a17-aba0-4c844c611558-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9\" (UID: \"bf70725b-8a40-4a17-aba0-4c844c611558\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9"
Sep 10 21:16:55.201713 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:55.201626    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl8xb\" (UniqueName: \"kubernetes.io/projected/bf70725b-8a40-4a17-aba0-4c844c611558-kube-api-access-jl8xb\") pod \"payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9\" (UID: \"bf70725b-8a40-4a17-aba0-4c844c611558\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9"
Sep 10 21:16:55.202270 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:55.202252    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf70725b-8a40-4a17-aba0-4c844c611558-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9\" (UID: \"bf70725b-8a40-4a17-aba0-4c844c611558\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9"
Sep 10 21:16:55.210014 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:55.209999    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl8xb\" (UniqueName: \"kubernetes.io/projected/bf70725b-8a40-4a17-aba0-4c844c611558-kube-api-access-jl8xb\") pod \"payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9\" (UID: \"bf70725b-8a40-4a17-aba0-4c844c611558\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9"
Sep 10 21:16:55.379176 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:55.379128    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-95d905-dockercfg-4dl29\""
Sep 10 21:16:55.387498 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:55.387476    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9"
Sep 10 21:16:55.664411 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:55.664386    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9"]
Sep 10 21:16:55.666241 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:16:55.666219    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf70725b_8a40_4a17_aba0_4c844c611558.slice/crio-e1cb8d644aacf34e7fcc93e2792d81fb8045cd9e3169775c91245411a6b9f07a WatchSource:0}: Error finding container e1cb8d644aacf34e7fcc93e2792d81fb8045cd9e3169775c91245411a6b9f07a: Status 404 returned error can't find the container with id e1cb8d644aacf34e7fcc93e2792d81fb8045cd9e3169775c91245411a6b9f07a
Sep 10 21:16:56.443601 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:56.443550    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9" event={"ID":"bf70725b-8a40-4a17-aba0-4c844c611558","Type":"ContainerStarted","Data":"f7ec62f628fe54a342ef20d5bc27622e17ef867a4fed7e2e1acd6e61675f4c98"}
Sep 10 21:16:56.444059 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:56.443621    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9"
Sep 10 21:16:56.444059 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:56.443646    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9" event={"ID":"bf70725b-8a40-4a17-aba0-4c844c611558","Type":"ContainerStarted","Data":"e1cb8d644aacf34e7fcc93e2792d81fb8045cd9e3169775c91245411a6b9f07a"}
Sep 10 21:16:56.465307 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:16:56.465265    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9" podStartSLOduration=1.465250726 podStartE2EDuration="1.465250726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:16:56.463411963 +0000 UTC m=+1201.675876884" watchObservedRunningTime="2026-09-10 21:16:56.465250726 +0000 UTC m=+1201.677715645"
Sep 10 21:17:01.628834 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:01.628795    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:17:06.189281 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.189245    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"]
Sep 10 21:17:06.194307 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.194280    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:17:06.197705 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.197673    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7810bd\"/\"kube-root-ca.crt\""
Sep 10 21:17:06.197705 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.197697    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7810bd\"/\"default-dockercfg-lxqpv\""
Sep 10 21:17:06.197917 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.197718    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7810bd\"/\"auth-test-model-7810bd-kserve-self-signed-certs\""
Sep 10 21:17:06.197917 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.197673    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7810bd\"/\"openshift-service-ca.crt\""
Sep 10 21:17:06.203032 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.203007    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"]
Sep 10 21:17:06.285877 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.285843    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-449jn\" (UniqueName: \"kubernetes.io/projected/8e983c34-05ee-4402-a29c-43a82181c24d-kube-api-access-449jn\") pod \"auth-test-model-7810bd-kserve-56cf77d95f-k5z4h\" (UID: \"8e983c34-05ee-4402-a29c-43a82181c24d\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:17:06.286015 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.285890    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e983c34-05ee-4402-a29c-43a82181c24d-model-cache\") pod \"auth-test-model-7810bd-kserve-56cf77d95f-k5z4h\" (UID: \"8e983c34-05ee-4402-a29c-43a82181c24d\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:17:06.286015 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.285950    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e983c34-05ee-4402-a29c-43a82181c24d-dshm\") pod \"auth-test-model-7810bd-kserve-56cf77d95f-k5z4h\" (UID: \"8e983c34-05ee-4402-a29c-43a82181c24d\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:17:06.286154 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.286040    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e983c34-05ee-4402-a29c-43a82181c24d-tmp-dir\") pod \"auth-test-model-7810bd-kserve-56cf77d95f-k5z4h\" (UID: \"8e983c34-05ee-4402-a29c-43a82181c24d\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:17:06.286154 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.286065    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e983c34-05ee-4402-a29c-43a82181c24d-home\") pod \"auth-test-model-7810bd-kserve-56cf77d95f-k5z4h\" (UID: \"8e983c34-05ee-4402-a29c-43a82181c24d\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:17:06.286264 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.286151    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e983c34-05ee-4402-a29c-43a82181c24d-tls-certs\") pod \"auth-test-model-7810bd-kserve-56cf77d95f-k5z4h\" (UID: \"8e983c34-05ee-4402-a29c-43a82181c24d\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:17:06.386698 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.386673    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e983c34-05ee-4402-a29c-43a82181c24d-tmp-dir\") pod \"auth-test-model-7810bd-kserve-56cf77d95f-k5z4h\" (UID: \"8e983c34-05ee-4402-a29c-43a82181c24d\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:17:06.386783 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.386702    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e983c34-05ee-4402-a29c-43a82181c24d-home\") pod \"auth-test-model-7810bd-kserve-56cf77d95f-k5z4h\" (UID: \"8e983c34-05ee-4402-a29c-43a82181c24d\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:17:06.386783 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.386721    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e983c34-05ee-4402-a29c-43a82181c24d-tls-certs\") pod \"auth-test-model-7810bd-kserve-56cf77d95f-k5z4h\" (UID: \"8e983c34-05ee-4402-a29c-43a82181c24d\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:17:06.386864 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.386842    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-449jn\" (UniqueName: \"kubernetes.io/projected/8e983c34-05ee-4402-a29c-43a82181c24d-kube-api-access-449jn\") pod \"auth-test-model-7810bd-kserve-56cf77d95f-k5z4h\" (UID: \"8e983c34-05ee-4402-a29c-43a82181c24d\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:17:06.386904 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.386893    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e983c34-05ee-4402-a29c-43a82181c24d-model-cache\") pod \"auth-test-model-7810bd-kserve-56cf77d95f-k5z4h\" (UID: \"8e983c34-05ee-4402-a29c-43a82181c24d\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:17:06.386945 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.386928    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e983c34-05ee-4402-a29c-43a82181c24d-dshm\") pod \"auth-test-model-7810bd-kserve-56cf77d95f-k5z4h\" (UID: \"8e983c34-05ee-4402-a29c-43a82181c24d\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:17:06.387077 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.387057    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e983c34-05ee-4402-a29c-43a82181c24d-tmp-dir\") pod \"auth-test-model-7810bd-kserve-56cf77d95f-k5z4h\" (UID: \"8e983c34-05ee-4402-a29c-43a82181c24d\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:17:06.387128 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.387078    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e983c34-05ee-4402-a29c-43a82181c24d-home\") pod \"auth-test-model-7810bd-kserve-56cf77d95f-k5z4h\" (UID: \"8e983c34-05ee-4402-a29c-43a82181c24d\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:17:06.387255 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.387231    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e983c34-05ee-4402-a29c-43a82181c24d-model-cache\") pod \"auth-test-model-7810bd-kserve-56cf77d95f-k5z4h\" (UID: \"8e983c34-05ee-4402-a29c-43a82181c24d\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:17:06.389300 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.389280    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e983c34-05ee-4402-a29c-43a82181c24d-dshm\") pod \"auth-test-model-7810bd-kserve-56cf77d95f-k5z4h\" (UID: \"8e983c34-05ee-4402-a29c-43a82181c24d\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:17:06.389480 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.389392    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e983c34-05ee-4402-a29c-43a82181c24d-tls-certs\") pod \"auth-test-model-7810bd-kserve-56cf77d95f-k5z4h\" (UID: \"8e983c34-05ee-4402-a29c-43a82181c24d\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:17:06.396121 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.396099    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-449jn\" (UniqueName: \"kubernetes.io/projected/8e983c34-05ee-4402-a29c-43a82181c24d-kube-api-access-449jn\") pod \"auth-test-model-7810bd-kserve-56cf77d95f-k5z4h\" (UID: \"8e983c34-05ee-4402-a29c-43a82181c24d\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:17:06.505524 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.505469    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:17:06.785549 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:06.785521    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"]
Sep 10 21:17:06.787579 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:17:06.787558    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e983c34_05ee_4402_a29c_43a82181c24d.slice/crio-57b2717c6e55abbcdc74008b5f9d8505b63cbefdb0af5bf05c5e0f6b6bdd7401 WatchSource:0}: Error finding container 57b2717c6e55abbcdc74008b5f9d8505b63cbefdb0af5bf05c5e0f6b6bdd7401: Status 404 returned error can't find the container with id 57b2717c6e55abbcdc74008b5f9d8505b63cbefdb0af5bf05c5e0f6b6bdd7401
Sep 10 21:17:07.449882 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:07.449848    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9"
Sep 10 21:17:07.506228 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:07.506196    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h" event={"ID":"8e983c34-05ee-4402-a29c-43a82181c24d","Type":"ContainerStarted","Data":"6a52fba73ba0049ac5c2c255d8351e678dc407d1d017edc313503e47e8d4a5b4"}
Sep 10 21:17:07.506390 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:07.506244    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h" event={"ID":"8e983c34-05ee-4402-a29c-43a82181c24d","Type":"ContainerStarted","Data":"57b2717c6e55abbcdc74008b5f9d8505b63cbefdb0af5bf05c5e0f6b6bdd7401"}
Sep 10 21:17:07.529803 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:07.529757    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h" podStartSLOduration=1.529740109 podStartE2EDuration="1.529740109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:17:07.527090763 +0000 UTC m=+1212.739555679" watchObservedRunningTime="2026-09-10 21:17:07.529740109 +0000 UTC m=+1212.742205029"
Sep 10 21:17:16.505920 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:16.505878    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:17:16.506381 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:16.505935    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:17:16.518390 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:16.518368    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:17:16.563489 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:16.563462    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:17:18.870735 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:18.870703    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7906-b6e4e9"]
Sep 10 21:17:18.874350 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:18.874335    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7906-b6e4e9"
Sep 10 21:17:18.878284 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:18.878260    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zd8vg\""
Sep 10 21:17:18.884253 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:18.884231    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7906-b6e4e9"]
Sep 10 21:17:18.985434 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:18.985404    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmntc\" (UniqueName: \"kubernetes.io/projected/02eb6b3d-4bdb-430b-be82-aa16a1337b40-kube-api-access-tmntc\") pod \"test-curl-7906-b6e4e9\" (UID: \"02eb6b3d-4bdb-430b-be82-aa16a1337b40\") " pod="opendatahub/test-curl-7906-b6e4e9"
Sep 10 21:17:19.086290 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:19.086263    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmntc\" (UniqueName: \"kubernetes.io/projected/02eb6b3d-4bdb-430b-be82-aa16a1337b40-kube-api-access-tmntc\") pod \"test-curl-7906-b6e4e9\" (UID: \"02eb6b3d-4bdb-430b-be82-aa16a1337b40\") " pod="opendatahub/test-curl-7906-b6e4e9"
Sep 10 21:17:19.095745 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:19.095723    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmntc\" (UniqueName: \"kubernetes.io/projected/02eb6b3d-4bdb-430b-be82-aa16a1337b40-kube-api-access-tmntc\") pod \"test-curl-7906-b6e4e9\" (UID: \"02eb6b3d-4bdb-430b-be82-aa16a1337b40\") " pod="opendatahub/test-curl-7906-b6e4e9"
Sep 10 21:17:19.184943 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:19.184891    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7906-b6e4e9"
Sep 10 21:17:19.455225 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:19.455200    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7906-b6e4e9"]
Sep 10 21:17:19.457033 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:17:19.456996    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02eb6b3d_4bdb_430b_be82_aa16a1337b40.slice/crio-700f6685782e037c4c9bbc1dc7f96ca3844c1bfcde22664d72c8ad0b040beeae WatchSource:0}: Error finding container 700f6685782e037c4c9bbc1dc7f96ca3844c1bfcde22664d72c8ad0b040beeae: Status 404 returned error can't find the container with id 700f6685782e037c4c9bbc1dc7f96ca3844c1bfcde22664d72c8ad0b040beeae
Sep 10 21:17:19.569867 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:19.569830    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7906-b6e4e9" event={"ID":"02eb6b3d-4bdb-430b-be82-aa16a1337b40","Type":"ContainerStarted","Data":"7aafe5325c49c4170f73eb11f234917d33bd5d60c86629a029ee18e7529cf05b"}
Sep 10 21:17:19.569965 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:19.569881    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7906-b6e4e9" event={"ID":"02eb6b3d-4bdb-430b-be82-aa16a1337b40","Type":"ContainerStarted","Data":"700f6685782e037c4c9bbc1dc7f96ca3844c1bfcde22664d72c8ad0b040beeae"}
Sep 10 21:17:19.586840 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:19.586802    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7906-b6e4e9" podStartSLOduration=1.586787965 podStartE2EDuration="1.586787965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:17:19.585345261 +0000 UTC m=+1224.797810181" watchObservedRunningTime="2026-09-10 21:17:19.586787965 +0000 UTC m=+1224.799252934"
Sep 10 21:17:20.263458 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:20.263428    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7906-b6e4e9"]
Sep 10 21:17:20.267622 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:20.267596    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7906-b6e4e9"]
Sep 10 21:17:20.382423 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:20.382359    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7906-3f9927"]
Sep 10 21:17:20.382719 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:20.382707    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02eb6b3d-4bdb-430b-be82-aa16a1337b40" containerName="test-curl-7906-b6e4e9"
Sep 10 21:17:20.382766 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:20.382721    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="02eb6b3d-4bdb-430b-be82-aa16a1337b40" containerName="test-curl-7906-b6e4e9"
Sep 10 21:17:20.382806 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:20.382797    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="02eb6b3d-4bdb-430b-be82-aa16a1337b40" containerName="test-curl-7906-b6e4e9"
Sep 10 21:17:20.385825 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:20.385811    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7906-3f9927"
Sep 10 21:17:20.395021 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:20.395000    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7906-3f9927"]
Sep 10 21:17:20.498382 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:20.498339    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76glm\" (UniqueName: \"kubernetes.io/projected/7439a16c-bddc-4622-9712-8a3f973145df-kube-api-access-76glm\") pod \"test-curl-7906-3f9927\" (UID: \"7439a16c-bddc-4622-9712-8a3f973145df\") " pod="opendatahub/test-curl-7906-3f9927"
Sep 10 21:17:20.575100 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:17:20.575071    2572 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7906-b6e4e9" container="test-curl-7906-b6e4e9"
Sep 10 21:17:20.599250 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:20.599226    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76glm\" (UniqueName: \"kubernetes.io/projected/7439a16c-bddc-4622-9712-8a3f973145df-kube-api-access-76glm\") pod \"test-curl-7906-3f9927\" (UID: \"7439a16c-bddc-4622-9712-8a3f973145df\") " pod="opendatahub/test-curl-7906-3f9927"
Sep 10 21:17:20.608395 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:20.608358    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76glm\" (UniqueName: \"kubernetes.io/projected/7439a16c-bddc-4622-9712-8a3f973145df-kube-api-access-76glm\") pod \"test-curl-7906-3f9927\" (UID: \"7439a16c-bddc-4622-9712-8a3f973145df\") " pod="opendatahub/test-curl-7906-3f9927"
Sep 10 21:17:20.695134 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:20.695071    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7906-3f9927"
Sep 10 21:17:20.965834 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:20.965762    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7906-3f9927"]
Sep 10 21:17:20.967959 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:17:20.967932    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7439a16c_bddc_4622_9712_8a3f973145df.slice/crio-3d7c8ce95f20ce9113e0ff0b619fba37befc2ffcb79afd5cda9adaa107179c37 WatchSource:0}: Error finding container 3d7c8ce95f20ce9113e0ff0b619fba37befc2ffcb79afd5cda9adaa107179c37: Status 404 returned error can't find the container with id 3d7c8ce95f20ce9113e0ff0b619fba37befc2ffcb79afd5cda9adaa107179c37
Sep 10 21:17:21.582052 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:21.582015    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7906-3f9927" event={"ID":"7439a16c-bddc-4622-9712-8a3f973145df","Type":"ContainerStarted","Data":"97dfb9eeb3a354eaf8257ce40f7a72c8dcbd1c1f322f73e072b282970e406108"}
Sep 10 21:17:21.582398 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:21.582065    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7906-3f9927" event={"ID":"7439a16c-bddc-4622-9712-8a3f973145df","Type":"ContainerStarted","Data":"3d7c8ce95f20ce9113e0ff0b619fba37befc2ffcb79afd5cda9adaa107179c37"}
Sep 10 21:17:21.582398 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:21.582210    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7906-b6e4e9" podUID="02eb6b3d-4bdb-430b-be82-aa16a1337b40" containerName="test-curl-7906-b6e4e9" containerID="cri-o://7aafe5325c49c4170f73eb11f234917d33bd5d60c86629a029ee18e7529cf05b" gracePeriod=2
Sep 10 21:17:21.597599 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:21.597545    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7906-3f9927" podStartSLOduration=1.597528163 podStartE2EDuration="1.597528163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:17:21.596291146 +0000 UTC m=+1226.808756064" watchObservedRunningTime="2026-09-10 21:17:21.597528163 +0000 UTC m=+1226.809993159"
Sep 10 21:17:22.206394 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:22.206368    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7906-3f9927"]
Sep 10 21:17:22.209993 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:22.209959    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7906-3f9927"]
Sep 10 21:17:22.324777 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:22.324754    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7906-a86c8f"]
Sep 10 21:17:22.325144 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:22.325130    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7439a16c-bddc-4622-9712-8a3f973145df" containerName="test-curl-7906-3f9927"
Sep 10 21:17:22.325207 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:22.325149    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7439a16c-bddc-4622-9712-8a3f973145df" containerName="test-curl-7906-3f9927"
Sep 10 21:17:22.325244 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:22.325230    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7439a16c-bddc-4622-9712-8a3f973145df" containerName="test-curl-7906-3f9927"
Sep 10 21:17:22.328334 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:22.328319    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7906-a86c8f"
Sep 10 21:17:22.339310 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:22.339286    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7906-a86c8f"]
Sep 10 21:17:22.412588 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:22.412564    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jw8x\" (UniqueName: \"kubernetes.io/projected/ad5d7340-7843-4b8b-92ef-f878c8161e98-kube-api-access-8jw8x\") pod \"test-curl-7906-a86c8f\" (UID: \"ad5d7340-7843-4b8b-92ef-f878c8161e98\") " pod="opendatahub/test-curl-7906-a86c8f"
Sep 10 21:17:22.513040 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:22.513014    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jw8x\" (UniqueName: \"kubernetes.io/projected/ad5d7340-7843-4b8b-92ef-f878c8161e98-kube-api-access-8jw8x\") pod \"test-curl-7906-a86c8f\" (UID: \"ad5d7340-7843-4b8b-92ef-f878c8161e98\") " pod="opendatahub/test-curl-7906-a86c8f"
Sep 10 21:17:22.522195 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:22.522170    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jw8x\" (UniqueName: \"kubernetes.io/projected/ad5d7340-7843-4b8b-92ef-f878c8161e98-kube-api-access-8jw8x\") pod \"test-curl-7906-a86c8f\" (UID: \"ad5d7340-7843-4b8b-92ef-f878c8161e98\") " pod="opendatahub/test-curl-7906-a86c8f"
Sep 10 21:17:22.586693 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:17:22.586626    2572 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7906-3f9927" container="test-curl-7906-3f9927"
Sep 10 21:17:22.637204 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:22.637184    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7906-a86c8f"
Sep 10 21:17:22.825911 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:22.825879    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:17:22.909696 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:22.909673    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7906-a86c8f"]
Sep 10 21:17:22.911527 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:17:22.911505    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad5d7340_7843_4b8b_92ef_f878c8161e98.slice/crio-fc9a2703dfc5e5c7ec1195eabc9c7fadeec59f11e1a76f239d1b15d07c597173 WatchSource:0}: Error finding container fc9a2703dfc5e5c7ec1195eabc9c7fadeec59f11e1a76f239d1b15d07c597173: Status 404 returned error can't find the container with id fc9a2703dfc5e5c7ec1195eabc9c7fadeec59f11e1a76f239d1b15d07c597173
Sep 10 21:17:23.598395 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:23.598367    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7906-a86c8f" event={"ID":"ad5d7340-7843-4b8b-92ef-f878c8161e98","Type":"ContainerStarted","Data":"2cd23669e226ea320777e18e2f4644ac9498614681dd3e1e60e4dd2a08574973"}
Sep 10 21:17:23.598769 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:23.598421    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7906-a86c8f" event={"ID":"ad5d7340-7843-4b8b-92ef-f878c8161e98","Type":"ContainerStarted","Data":"fc9a2703dfc5e5c7ec1195eabc9c7fadeec59f11e1a76f239d1b15d07c597173"}
Sep 10 21:17:23.598769 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:23.598474    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7906-3f9927" podUID="7439a16c-bddc-4622-9712-8a3f973145df" containerName="test-curl-7906-3f9927" containerID="cri-o://97dfb9eeb3a354eaf8257ce40f7a72c8dcbd1c1f322f73e072b282970e406108" gracePeriod=2
Sep 10 21:17:23.613864 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:23.613810    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7906-a86c8f" podStartSLOduration=1.613795467 podStartE2EDuration="1.613795467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:17:23.6137099 +0000 UTC m=+1228.826174820" watchObservedRunningTime="2026-09-10 21:17:23.613795467 +0000 UTC m=+1228.826260382"
Sep 10 21:17:23.762825 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:23.762804    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7906-b6e4e9"
Sep 10 21:17:23.926345 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:23.926280    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmntc\" (UniqueName: \"kubernetes.io/projected/02eb6b3d-4bdb-430b-be82-aa16a1337b40-kube-api-access-tmntc\") pod \"02eb6b3d-4bdb-430b-be82-aa16a1337b40\" (UID: \"02eb6b3d-4bdb-430b-be82-aa16a1337b40\") "
Sep 10 21:17:23.928294 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:23.928268    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02eb6b3d-4bdb-430b-be82-aa16a1337b40-kube-api-access-tmntc" (OuterVolumeSpecName: "kube-api-access-tmntc") pod "02eb6b3d-4bdb-430b-be82-aa16a1337b40" (UID: "02eb6b3d-4bdb-430b-be82-aa16a1337b40"). InnerVolumeSpecName "kube-api-access-tmntc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:17:24.027428 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:24.027410    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmntc\" (UniqueName: \"kubernetes.io/projected/02eb6b3d-4bdb-430b-be82-aa16a1337b40-kube-api-access-tmntc\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:17:24.202330 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:24.202273    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7906-a86c8f"]
Sep 10 21:17:24.209195 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:24.209173    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7906-a86c8f"]
Sep 10 21:17:24.329483 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:24.326633    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7906-a3134c"]
Sep 10 21:17:24.329483 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:24.327515    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad5d7340-7843-4b8b-92ef-f878c8161e98" containerName="test-curl-7906-a86c8f"
Sep 10 21:17:24.329483 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:24.327542    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad5d7340-7843-4b8b-92ef-f878c8161e98" containerName="test-curl-7906-a86c8f"
Sep 10 21:17:24.329483 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:24.327731    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad5d7340-7843-4b8b-92ef-f878c8161e98" containerName="test-curl-7906-a86c8f"
Sep 10 21:17:24.332130 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:24.332106    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7906-a3134c"
Sep 10 21:17:24.333321 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:24.333294    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7906-a3134c"]
Sep 10 21:17:24.431080 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:24.431054    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsjfh\" (UniqueName: \"kubernetes.io/projected/6cc20cb1-bce6-412f-b312-e01042e7b6c0-kube-api-access-tsjfh\") pod \"test-curl-7906-a3134c\" (UID: \"6cc20cb1-bce6-412f-b312-e01042e7b6c0\") " pod="opendatahub/test-curl-7906-a3134c"
Sep 10 21:17:24.531603 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:24.531579    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsjfh\" (UniqueName: \"kubernetes.io/projected/6cc20cb1-bce6-412f-b312-e01042e7b6c0-kube-api-access-tsjfh\") pod \"test-curl-7906-a3134c\" (UID: \"6cc20cb1-bce6-412f-b312-e01042e7b6c0\") " pod="opendatahub/test-curl-7906-a3134c"
Sep 10 21:17:24.543253 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:24.543223    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsjfh\" (UniqueName: \"kubernetes.io/projected/6cc20cb1-bce6-412f-b312-e01042e7b6c0-kube-api-access-tsjfh\") pod \"test-curl-7906-a3134c\" (UID: \"6cc20cb1-bce6-412f-b312-e01042e7b6c0\") " pod="opendatahub/test-curl-7906-a3134c"
Sep 10 21:17:24.604947 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:24.604922    2572 generic.go:356] "Generic (PLEG): container finished" podID="02eb6b3d-4bdb-430b-be82-aa16a1337b40" containerID="7aafe5325c49c4170f73eb11f234917d33bd5d60c86629a029ee18e7529cf05b" exitCode=137
Sep 10 21:17:24.605301 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:24.604973    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7906-b6e4e9"
Sep 10 21:17:24.605301 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:24.605008    2572 scope.go:117] "RemoveContainer" containerID="7aafe5325c49c4170f73eb11f234917d33bd5d60c86629a029ee18e7529cf05b"
Sep 10 21:17:24.605301 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:17:24.605199    2572 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7906-a86c8f" container="test-curl-7906-a86c8f"
Sep 10 21:17:24.613923 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:24.613906    2572 scope.go:117] "RemoveContainer" containerID="7aafe5325c49c4170f73eb11f234917d33bd5d60c86629a029ee18e7529cf05b"
Sep 10 21:17:24.614211 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:17:24.614191    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aafe5325c49c4170f73eb11f234917d33bd5d60c86629a029ee18e7529cf05b\": container with ID starting with 7aafe5325c49c4170f73eb11f234917d33bd5d60c86629a029ee18e7529cf05b not found: ID does not exist" containerID="7aafe5325c49c4170f73eb11f234917d33bd5d60c86629a029ee18e7529cf05b"
Sep 10 21:17:24.614266 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:24.614218    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aafe5325c49c4170f73eb11f234917d33bd5d60c86629a029ee18e7529cf05b"} err="failed to get container status \"7aafe5325c49c4170f73eb11f234917d33bd5d60c86629a029ee18e7529cf05b\": rpc error: code = NotFound desc = could not find container \"7aafe5325c49c4170f73eb11f234917d33bd5d60c86629a029ee18e7529cf05b\": container with ID starting with 7aafe5325c49c4170f73eb11f234917d33bd5d60c86629a029ee18e7529cf05b not found: ID does not exist"
Sep 10 21:17:24.615860 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:17:24.615836    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7906-b6e4e9\" is forbidden: User \"system:node:ip-10-0-138-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-232.ec2.internal' and this object" podUID="02eb6b3d-4bdb-430b-be82-aa16a1337b40" pod="opendatahub/test-curl-7906-b6e4e9"
Sep 10 21:17:24.644133 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:24.644103    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7906-a3134c"
Sep 10 21:17:24.914886 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:24.914861    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7906-a3134c"]
Sep 10 21:17:24.916671 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:17:24.916643    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cc20cb1_bce6_412f_b312_e01042e7b6c0.slice/crio-1cf4a2d344e1ee1d2d23d6bbde615c1ec44f64fb9c89a87456ab71629ed7540b WatchSource:0}: Error finding container 1cf4a2d344e1ee1d2d23d6bbde615c1ec44f64fb9c89a87456ab71629ed7540b: Status 404 returned error can't find the container with id 1cf4a2d344e1ee1d2d23d6bbde615c1ec44f64fb9c89a87456ab71629ed7540b
Sep 10 21:17:25.436826 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:17:25.436794    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7906-b6e4e9\" is forbidden: User \"system:node:ip-10-0-138-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-232.ec2.internal' and this object" podUID="02eb6b3d-4bdb-430b-be82-aa16a1337b40" pod="opendatahub/test-curl-7906-b6e4e9"
Sep 10 21:17:25.438243 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:25.438219    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02eb6b3d-4bdb-430b-be82-aa16a1337b40" path="/var/lib/kubelet/pods/02eb6b3d-4bdb-430b-be82-aa16a1337b40/volumes"
Sep 10 21:17:25.612354 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:25.612327    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7906-a3134c" event={"ID":"6cc20cb1-bce6-412f-b312-e01042e7b6c0","Type":"ContainerStarted","Data":"a7d2505423a7ae7af5c2e03c627fdedccac470308ea3be821eec7236f33aebd1"}
Sep 10 21:17:25.612708 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:25.612366    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7906-a3134c" event={"ID":"6cc20cb1-bce6-412f-b312-e01042e7b6c0","Type":"ContainerStarted","Data":"1cf4a2d344e1ee1d2d23d6bbde615c1ec44f64fb9c89a87456ab71629ed7540b"}
Sep 10 21:17:25.612708 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:25.612483    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7906-a86c8f" podUID="ad5d7340-7843-4b8b-92ef-f878c8161e98" containerName="test-curl-7906-a86c8f" containerID="cri-o://2cd23669e226ea320777e18e2f4644ac9498614681dd3e1e60e4dd2a08574973" gracePeriod=2
Sep 10 21:17:25.647972 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:25.647926    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7906-a3134c" podStartSLOduration=1.64790999 podStartE2EDuration="1.64790999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:17:25.646000983 +0000 UTC m=+1230.858465910" watchObservedRunningTime="2026-09-10 21:17:25.64790999 +0000 UTC m=+1230.860374909"
Sep 10 21:17:25.744846 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:25.744826    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7906-3f9927"
Sep 10 21:17:25.840611 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:25.840587    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76glm\" (UniqueName: \"kubernetes.io/projected/7439a16c-bddc-4622-9712-8a3f973145df-kube-api-access-76glm\") pod \"7439a16c-bddc-4622-9712-8a3f973145df\" (UID: \"7439a16c-bddc-4622-9712-8a3f973145df\") "
Sep 10 21:17:25.842717 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:25.842684    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7439a16c-bddc-4622-9712-8a3f973145df-kube-api-access-76glm" (OuterVolumeSpecName: "kube-api-access-76glm") pod "7439a16c-bddc-4622-9712-8a3f973145df" (UID: "7439a16c-bddc-4622-9712-8a3f973145df"). InnerVolumeSpecName "kube-api-access-76glm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:17:25.942052 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:25.942025    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76glm\" (UniqueName: \"kubernetes.io/projected/7439a16c-bddc-4622-9712-8a3f973145df-kube-api-access-76glm\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:17:26.237440 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:26.237416    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7906-a3134c"]
Sep 10 21:17:26.242658 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:26.242633    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7906-a3134c"]
Sep 10 21:17:26.618727 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:26.618695    2572 generic.go:356] "Generic (PLEG): container finished" podID="7439a16c-bddc-4622-9712-8a3f973145df" containerID="97dfb9eeb3a354eaf8257ce40f7a72c8dcbd1c1f322f73e072b282970e406108" exitCode=137
Sep 10 21:17:26.619128 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:26.618731    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7906-3f9927"
Sep 10 21:17:26.619128 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:26.618763    2572 scope.go:117] "RemoveContainer" containerID="97dfb9eeb3a354eaf8257ce40f7a72c8dcbd1c1f322f73e072b282970e406108"
Sep 10 21:17:26.619128 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:17:26.619100    2572 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7906-a3134c" container="test-curl-7906-a3134c"
Sep 10 21:17:26.628191 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:26.628177    2572 scope.go:117] "RemoveContainer" containerID="97dfb9eeb3a354eaf8257ce40f7a72c8dcbd1c1f322f73e072b282970e406108"
Sep 10 21:17:26.628478 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:17:26.628459    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97dfb9eeb3a354eaf8257ce40f7a72c8dcbd1c1f322f73e072b282970e406108\": container with ID starting with 97dfb9eeb3a354eaf8257ce40f7a72c8dcbd1c1f322f73e072b282970e406108 not found: ID does not exist" containerID="97dfb9eeb3a354eaf8257ce40f7a72c8dcbd1c1f322f73e072b282970e406108"
Sep 10 21:17:26.628518 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:26.628489    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97dfb9eeb3a354eaf8257ce40f7a72c8dcbd1c1f322f73e072b282970e406108"} err="failed to get container status \"97dfb9eeb3a354eaf8257ce40f7a72c8dcbd1c1f322f73e072b282970e406108\": rpc error: code = NotFound desc = could not find container \"97dfb9eeb3a354eaf8257ce40f7a72c8dcbd1c1f322f73e072b282970e406108\": container with ID starting with 97dfb9eeb3a354eaf8257ce40f7a72c8dcbd1c1f322f73e072b282970e406108 not found: ID does not exist"
Sep 10 21:17:26.629910 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:17:26.629887    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7906-3f9927\" is forbidden: User \"system:node:ip-10-0-138-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-232.ec2.internal' and this object" podUID="7439a16c-bddc-4622-9712-8a3f973145df" pod="opendatahub/test-curl-7906-3f9927"
Sep 10 21:17:27.438261 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:27.438227    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7439a16c-bddc-4622-9712-8a3f973145df" path="/var/lib/kubelet/pods/7439a16c-bddc-4622-9712-8a3f973145df/volumes"
Sep 10 21:17:27.625383 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:27.625345    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7906-a3134c" podUID="6cc20cb1-bce6-412f-b312-e01042e7b6c0" containerName="test-curl-7906-a3134c" containerID="cri-o://a7d2505423a7ae7af5c2e03c627fdedccac470308ea3be821eec7236f33aebd1" gracePeriod=2
Sep 10 21:17:27.760047 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:27.760026    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7906-a86c8f"
Sep 10 21:17:27.857617 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:27.857593    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jw8x\" (UniqueName: \"kubernetes.io/projected/ad5d7340-7843-4b8b-92ef-f878c8161e98-kube-api-access-8jw8x\") pod \"ad5d7340-7843-4b8b-92ef-f878c8161e98\" (UID: \"ad5d7340-7843-4b8b-92ef-f878c8161e98\") "
Sep 10 21:17:27.859678 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:27.859658    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5d7340-7843-4b8b-92ef-f878c8161e98-kube-api-access-8jw8x" (OuterVolumeSpecName: "kube-api-access-8jw8x") pod "ad5d7340-7843-4b8b-92ef-f878c8161e98" (UID: "ad5d7340-7843-4b8b-92ef-f878c8161e98"). InnerVolumeSpecName "kube-api-access-8jw8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:17:27.958688 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:27.958663    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jw8x\" (UniqueName: \"kubernetes.io/projected/ad5d7340-7843-4b8b-92ef-f878c8161e98-kube-api-access-8jw8x\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:17:28.632436 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:28.632403    2572 generic.go:356] "Generic (PLEG): container finished" podID="ad5d7340-7843-4b8b-92ef-f878c8161e98" containerID="2cd23669e226ea320777e18e2f4644ac9498614681dd3e1e60e4dd2a08574973" exitCode=137
Sep 10 21:17:28.632846 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:28.632454    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7906-a86c8f"
Sep 10 21:17:28.632846 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:28.632530    2572 scope.go:117] "RemoveContainer" containerID="2cd23669e226ea320777e18e2f4644ac9498614681dd3e1e60e4dd2a08574973"
Sep 10 21:17:28.642900 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:28.642882    2572 scope.go:117] "RemoveContainer" containerID="2cd23669e226ea320777e18e2f4644ac9498614681dd3e1e60e4dd2a08574973"
Sep 10 21:17:28.643281 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:17:28.643246    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7906-a86c8f\" is forbidden: User \"system:node:ip-10-0-138-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-232.ec2.internal' and this object" podUID="ad5d7340-7843-4b8b-92ef-f878c8161e98" pod="opendatahub/test-curl-7906-a86c8f"
Sep 10 21:17:28.643361 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:17:28.643280    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cd23669e226ea320777e18e2f4644ac9498614681dd3e1e60e4dd2a08574973\": container with ID starting with 2cd23669e226ea320777e18e2f4644ac9498614681dd3e1e60e4dd2a08574973 not found: ID does not exist" containerID="2cd23669e226ea320777e18e2f4644ac9498614681dd3e1e60e4dd2a08574973"
Sep 10 21:17:28.643361 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:28.643310    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd23669e226ea320777e18e2f4644ac9498614681dd3e1e60e4dd2a08574973"} err="failed to get container status \"2cd23669e226ea320777e18e2f4644ac9498614681dd3e1e60e4dd2a08574973\": rpc error: code = NotFound desc = could not find container \"2cd23669e226ea320777e18e2f4644ac9498614681dd3e1e60e4dd2a08574973\": container with ID starting with 2cd23669e226ea320777e18e2f4644ac9498614681dd3e1e60e4dd2a08574973 not found: ID does not exist"
Sep 10 21:17:29.439074 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:29.439044    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad5d7340-7843-4b8b-92ef-f878c8161e98" path="/var/lib/kubelet/pods/ad5d7340-7843-4b8b-92ef-f878c8161e98/volumes"
Sep 10 21:17:29.802835 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:29.802811    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7906-a3134c"
Sep 10 21:17:29.973188 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:29.973154    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsjfh\" (UniqueName: \"kubernetes.io/projected/6cc20cb1-bce6-412f-b312-e01042e7b6c0-kube-api-access-tsjfh\") pod \"6cc20cb1-bce6-412f-b312-e01042e7b6c0\" (UID: \"6cc20cb1-bce6-412f-b312-e01042e7b6c0\") "
Sep 10 21:17:29.975260 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:29.975234    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc20cb1-bce6-412f-b312-e01042e7b6c0-kube-api-access-tsjfh" (OuterVolumeSpecName: "kube-api-access-tsjfh") pod "6cc20cb1-bce6-412f-b312-e01042e7b6c0" (UID: "6cc20cb1-bce6-412f-b312-e01042e7b6c0"). InnerVolumeSpecName "kube-api-access-tsjfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:17:30.074349 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:30.074325    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsjfh\" (UniqueName: \"kubernetes.io/projected/6cc20cb1-bce6-412f-b312-e01042e7b6c0-kube-api-access-tsjfh\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:17:30.644974 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:30.644941    2572 generic.go:356] "Generic (PLEG): container finished" podID="6cc20cb1-bce6-412f-b312-e01042e7b6c0" containerID="a7d2505423a7ae7af5c2e03c627fdedccac470308ea3be821eec7236f33aebd1" exitCode=137
Sep 10 21:17:30.645134 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:30.645008    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7906-a3134c"
Sep 10 21:17:30.645134 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:30.645073    2572 scope.go:117] "RemoveContainer" containerID="a7d2505423a7ae7af5c2e03c627fdedccac470308ea3be821eec7236f33aebd1"
Sep 10 21:17:30.655177 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:30.655158    2572 scope.go:117] "RemoveContainer" containerID="a7d2505423a7ae7af5c2e03c627fdedccac470308ea3be821eec7236f33aebd1"
Sep 10 21:17:30.655443 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:17:30.655424    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d2505423a7ae7af5c2e03c627fdedccac470308ea3be821eec7236f33aebd1\": container with ID starting with a7d2505423a7ae7af5c2e03c627fdedccac470308ea3be821eec7236f33aebd1 not found: ID does not exist" containerID="a7d2505423a7ae7af5c2e03c627fdedccac470308ea3be821eec7236f33aebd1"
Sep 10 21:17:30.655491 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:30.655449    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d2505423a7ae7af5c2e03c627fdedccac470308ea3be821eec7236f33aebd1"} err="failed to get container status \"a7d2505423a7ae7af5c2e03c627fdedccac470308ea3be821eec7236f33aebd1\": rpc error: code = NotFound desc = could not find container \"a7d2505423a7ae7af5c2e03c627fdedccac470308ea3be821eec7236f33aebd1\": container with ID starting with a7d2505423a7ae7af5c2e03c627fdedccac470308ea3be821eec7236f33aebd1 not found: ID does not exist"
Sep 10 21:17:30.656660 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:17:30.656638    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7906-a3134c\" is forbidden: User \"system:node:ip-10-0-138-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-232.ec2.internal' and this object" podUID="6cc20cb1-bce6-412f-b312-e01042e7b6c0" pod="opendatahub/test-curl-7906-a3134c"
Sep 10 21:17:31.438666 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:31.438634    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cc20cb1-bce6-412f-b312-e01042e7b6c0" path="/var/lib/kubelet/pods/6cc20cb1-bce6-412f-b312-e01042e7b6c0/volumes"
Sep 10 21:17:32.130476 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:32.130439    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs"]
Sep 10 21:17:32.130893 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:32.130874    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cc20cb1-bce6-412f-b312-e01042e7b6c0" containerName="test-curl-7906-a3134c"
Sep 10 21:17:32.131011 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:32.130896    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cc20cb1-bce6-412f-b312-e01042e7b6c0" containerName="test-curl-7906-a3134c"
Sep 10 21:17:32.131076 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:32.131040    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cc20cb1-bce6-412f-b312-e01042e7b6c0" containerName="test-curl-7906-a3134c"
Sep 10 21:17:32.135545 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:32.135524    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs"
Sep 10 21:17:32.138024 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:32.137971    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fc0caf1f\""
Sep 10 21:17:32.138132 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:32.138040    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fc0caf1f-dockercfg-cf2p2\""
Sep 10 21:17:32.144626 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:32.144603    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs"]
Sep 10 21:17:32.293795 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:32.293762    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp2gt\" (UniqueName: \"kubernetes.io/projected/8b59fc6e-58c6-4ea8-b522-43405f330861-kube-api-access-sp2gt\") pod \"payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs\" (UID: \"8b59fc6e-58c6-4ea8-b522-43405f330861\") " pod="openshift-ingress/payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs"
Sep 10 21:17:32.293925 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:32.293891    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b59fc6e-58c6-4ea8-b522-43405f330861-plugins-config-volume\") pod \"payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs\" (UID: \"8b59fc6e-58c6-4ea8-b522-43405f330861\") " pod="openshift-ingress/payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs"
Sep 10 21:17:32.394732 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:32.394674    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp2gt\" (UniqueName: \"kubernetes.io/projected/8b59fc6e-58c6-4ea8-b522-43405f330861-kube-api-access-sp2gt\") pod \"payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs\" (UID: \"8b59fc6e-58c6-4ea8-b522-43405f330861\") " pod="openshift-ingress/payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs"
Sep 10 21:17:32.394891 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:32.394738    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b59fc6e-58c6-4ea8-b522-43405f330861-plugins-config-volume\") pod \"payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs\" (UID: \"8b59fc6e-58c6-4ea8-b522-43405f330861\") " pod="openshift-ingress/payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs"
Sep 10 21:17:32.395360 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:32.395342    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b59fc6e-58c6-4ea8-b522-43405f330861-plugins-config-volume\") pod \"payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs\" (UID: \"8b59fc6e-58c6-4ea8-b522-43405f330861\") " pod="openshift-ingress/payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs"
Sep 10 21:17:32.403496 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:32.403477    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp2gt\" (UniqueName: \"kubernetes.io/projected/8b59fc6e-58c6-4ea8-b522-43405f330861-kube-api-access-sp2gt\") pod \"payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs\" (UID: \"8b59fc6e-58c6-4ea8-b522-43405f330861\") " pod="openshift-ingress/payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs"
Sep 10 21:17:32.447050 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:32.447030    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs"
Sep 10 21:17:32.725453 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:32.725417    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs"]
Sep 10 21:17:32.727379 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:17:32.727352    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b59fc6e_58c6_4ea8_b522_43405f330861.slice/crio-71e0b7b0c16845aef0c2ca4b91d78dbff7ce69ba56ca080960db61c65bd6b0d8 WatchSource:0}: Error finding container 71e0b7b0c16845aef0c2ca4b91d78dbff7ce69ba56ca080960db61c65bd6b0d8: Status 404 returned error can't find the container with id 71e0b7b0c16845aef0c2ca4b91d78dbff7ce69ba56ca080960db61c65bd6b0d8
Sep 10 21:17:33.662699 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:33.662664    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs" event={"ID":"8b59fc6e-58c6-4ea8-b522-43405f330861","Type":"ContainerStarted","Data":"607cf75925a71f661a9812ac525e2d9a6c999ff3eb76a47f35667d3f190d9f16"}
Sep 10 21:17:33.663161 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:33.662728    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs" event={"ID":"8b59fc6e-58c6-4ea8-b522-43405f330861","Type":"ContainerStarted","Data":"71e0b7b0c16845aef0c2ca4b91d78dbff7ce69ba56ca080960db61c65bd6b0d8"}
Sep 10 21:17:33.663161 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:33.662753    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs"
Sep 10 21:17:33.691423 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:33.691370    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs" podStartSLOduration=1.691353908 podStartE2EDuration="1.691353908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:17:33.689564984 +0000 UTC m=+1238.902029937" watchObservedRunningTime="2026-09-10 21:17:33.691353908 +0000 UTC m=+1238.903818827"
Sep 10 21:17:39.039050 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:39.039008    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:17:44.670076 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:44.670036    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs"
Sep 10 21:17:59.136786 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:17:59.136735    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:18:02.518658 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.518623    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"]
Sep 10 21:18:02.523950 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.523932    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:18:02.527627 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.527602    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fc0caf1f\"/\"kube-root-ca.crt\""
Sep 10 21:18:02.527764 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.527608    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fc0caf1f\"/\"e2e-lifecycle-model-fc0caf1f-kserve-self-signed-certs\""
Sep 10 21:18:02.527764 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.527608    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fc0caf1f\"/\"default-dockercfg-f8lg9\""
Sep 10 21:18:02.527764 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.527637    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fc0caf1f\"/\"openshift-service-ca.crt\""
Sep 10 21:18:02.536439 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.536416    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"]
Sep 10 21:18:02.631031 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.631004    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8566c1a-43b7-4295-ad43-736c51b4f9f3-home\") pod \"e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf\" (UID: \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\") " pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:18:02.631141 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.631057    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8566c1a-43b7-4295-ad43-736c51b4f9f3-dshm\") pod \"e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf\" (UID: \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\") " pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:18:02.631141 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.631086    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8566c1a-43b7-4295-ad43-736c51b4f9f3-tmp-dir\") pod \"e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf\" (UID: \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\") " pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:18:02.631141 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.631109    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8566c1a-43b7-4295-ad43-736c51b4f9f3-tls-certs\") pod \"e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf\" (UID: \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\") " pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:18:02.631141 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.631133    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq7rd\" (UniqueName: \"kubernetes.io/projected/e8566c1a-43b7-4295-ad43-736c51b4f9f3-kube-api-access-fq7rd\") pod \"e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf\" (UID: \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\") " pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:18:02.631330 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.631189    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8566c1a-43b7-4295-ad43-736c51b4f9f3-model-cache\") pod \"e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf\" (UID: \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\") " pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:18:02.732174 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.732151    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8566c1a-43b7-4295-ad43-736c51b4f9f3-dshm\") pod \"e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf\" (UID: \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\") " pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:18:02.732288 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.732179    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8566c1a-43b7-4295-ad43-736c51b4f9f3-tmp-dir\") pod \"e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf\" (UID: \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\") " pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:18:02.732288 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.732197    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8566c1a-43b7-4295-ad43-736c51b4f9f3-tls-certs\") pod \"e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf\" (UID: \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\") " pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:18:02.732288 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.732214    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq7rd\" (UniqueName: \"kubernetes.io/projected/e8566c1a-43b7-4295-ad43-736c51b4f9f3-kube-api-access-fq7rd\") pod \"e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf\" (UID: \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\") " pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:18:02.732288 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.732260    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8566c1a-43b7-4295-ad43-736c51b4f9f3-model-cache\") pod \"e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf\" (UID: \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\") " pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:18:02.732508 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.732348    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8566c1a-43b7-4295-ad43-736c51b4f9f3-home\") pod \"e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf\" (UID: \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\") " pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:18:02.732640 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.732617    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8566c1a-43b7-4295-ad43-736c51b4f9f3-tmp-dir\") pod \"e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf\" (UID: \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\") " pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:18:02.732818 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.732793    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8566c1a-43b7-4295-ad43-736c51b4f9f3-model-cache\") pod \"e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf\" (UID: \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\") " pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:18:02.732933 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.732674    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8566c1a-43b7-4295-ad43-736c51b4f9f3-home\") pod \"e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf\" (UID: \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\") " pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:18:02.734669 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.734648    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8566c1a-43b7-4295-ad43-736c51b4f9f3-dshm\") pod \"e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf\" (UID: \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\") " pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:18:02.735037 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.735017    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8566c1a-43b7-4295-ad43-736c51b4f9f3-tls-certs\") pod \"e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf\" (UID: \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\") " pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:18:02.744954 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.744931    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq7rd\" (UniqueName: \"kubernetes.io/projected/e8566c1a-43b7-4295-ad43-736c51b4f9f3-kube-api-access-fq7rd\") pod \"e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf\" (UID: \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\") " pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:18:02.833202 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:02.833169    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:18:03.110190 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:03.110165    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"]
Sep 10 21:18:03.111953 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:18:03.111931    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8566c1a_43b7_4295_ad43_736c51b4f9f3.slice/crio-4f424b266ae97b3473220eedbd52d1624f1ae9652262ed35d1c8d7eeb2d92e93 WatchSource:0}: Error finding container 4f424b266ae97b3473220eedbd52d1624f1ae9652262ed35d1c8d7eeb2d92e93: Status 404 returned error can't find the container with id 4f424b266ae97b3473220eedbd52d1624f1ae9652262ed35d1c8d7eeb2d92e93
Sep 10 21:18:03.827335 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:03.827294    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf" event={"ID":"e8566c1a-43b7-4295-ad43-736c51b4f9f3","Type":"ContainerStarted","Data":"e716f13a0ea69369eab173538a7aac6ef0376c9653573990844d8d679eb5af92"}
Sep 10 21:18:03.827335 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:03.827340    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf" event={"ID":"e8566c1a-43b7-4295-ad43-736c51b4f9f3","Type":"ContainerStarted","Data":"4f424b266ae97b3473220eedbd52d1624f1ae9652262ed35d1c8d7eeb2d92e93"}
Sep 10 21:18:03.846531 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:03.846477    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf" podStartSLOduration=1.846460443 podStartE2EDuration="1.846460443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:18:03.844942964 +0000 UTC m=+1269.057407884" watchObservedRunningTime="2026-09-10 21:18:03.846460443 +0000 UTC m=+1269.058925442"
Sep 10 21:18:12.833373 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:12.833341    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:18:12.833834 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:12.833383    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:18:12.845971 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:12.845945    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:18:12.885627 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:12.885602    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:18:17.840975 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:17.840923    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:18:36.140237 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:36.140203    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:18:45.208276 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:45.208238    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs"]
Sep 10 21:18:45.208718 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:45.208521    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs" podUID="8b59fc6e-58c6-4ea8-b522-43405f330861" containerName="payload-processing" containerID="cri-o://607cf75925a71f661a9812ac525e2d9a6c999ff3eb76a47f35667d3f190d9f16" gracePeriod=30
Sep 10 21:18:45.461457 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:45.461400    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs"
Sep 10 21:18:45.587376 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:45.587334    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sp2gt\" (UniqueName: \"kubernetes.io/projected/8b59fc6e-58c6-4ea8-b522-43405f330861-kube-api-access-sp2gt\") pod \"8b59fc6e-58c6-4ea8-b522-43405f330861\" (UID: \"8b59fc6e-58c6-4ea8-b522-43405f330861\") "
Sep 10 21:18:45.587516 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:45.587452    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b59fc6e-58c6-4ea8-b522-43405f330861-plugins-config-volume\") pod \"8b59fc6e-58c6-4ea8-b522-43405f330861\" (UID: \"8b59fc6e-58c6-4ea8-b522-43405f330861\") "
Sep 10 21:18:45.587801 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:45.587777    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b59fc6e-58c6-4ea8-b522-43405f330861-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b59fc6e-58c6-4ea8-b522-43405f330861" (UID: "8b59fc6e-58c6-4ea8-b522-43405f330861"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:18:45.589525 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:45.589504    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b59fc6e-58c6-4ea8-b522-43405f330861-kube-api-access-sp2gt" (OuterVolumeSpecName: "kube-api-access-sp2gt") pod "8b59fc6e-58c6-4ea8-b522-43405f330861" (UID: "8b59fc6e-58c6-4ea8-b522-43405f330861"). InnerVolumeSpecName "kube-api-access-sp2gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:18:45.688861 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:45.688840    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp2gt\" (UniqueName: \"kubernetes.io/projected/8b59fc6e-58c6-4ea8-b522-43405f330861-kube-api-access-sp2gt\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:18:45.688861 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:45.688860    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b59fc6e-58c6-4ea8-b522-43405f330861-plugins-config-volume\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:18:46.045503 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:46.045479    2572 generic.go:356] "Generic (PLEG): container finished" podID="8b59fc6e-58c6-4ea8-b522-43405f330861" containerID="607cf75925a71f661a9812ac525e2d9a6c999ff3eb76a47f35667d3f190d9f16" exitCode=0
Sep 10 21:18:46.045611 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:46.045526    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs"
Sep 10 21:18:46.045611 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:46.045533    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs" event={"ID":"8b59fc6e-58c6-4ea8-b522-43405f330861","Type":"ContainerDied","Data":"607cf75925a71f661a9812ac525e2d9a6c999ff3eb76a47f35667d3f190d9f16"}
Sep 10 21:18:46.045611 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:46.045589    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs" event={"ID":"8b59fc6e-58c6-4ea8-b522-43405f330861","Type":"ContainerDied","Data":"71e0b7b0c16845aef0c2ca4b91d78dbff7ce69ba56ca080960db61c65bd6b0d8"}
Sep 10 21:18:46.046018 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:46.045617    2572 scope.go:117] "RemoveContainer" containerID="607cf75925a71f661a9812ac525e2d9a6c999ff3eb76a47f35667d3f190d9f16"
Sep 10 21:18:46.055370 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:46.055351    2572 scope.go:117] "RemoveContainer" containerID="607cf75925a71f661a9812ac525e2d9a6c999ff3eb76a47f35667d3f190d9f16"
Sep 10 21:18:46.055650 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:18:46.055630    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"607cf75925a71f661a9812ac525e2d9a6c999ff3eb76a47f35667d3f190d9f16\": container with ID starting with 607cf75925a71f661a9812ac525e2d9a6c999ff3eb76a47f35667d3f190d9f16 not found: ID does not exist" containerID="607cf75925a71f661a9812ac525e2d9a6c999ff3eb76a47f35667d3f190d9f16"
Sep 10 21:18:46.055705 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:46.055656    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"607cf75925a71f661a9812ac525e2d9a6c999ff3eb76a47f35667d3f190d9f16"} err="failed to get container status \"607cf75925a71f661a9812ac525e2d9a6c999ff3eb76a47f35667d3f190d9f16\": rpc error: code = NotFound desc = could not find container \"607cf75925a71f661a9812ac525e2d9a6c999ff3eb76a47f35667d3f190d9f16\": container with ID starting with 607cf75925a71f661a9812ac525e2d9a6c999ff3eb76a47f35667d3f190d9f16 not found: ID does not exist"
Sep 10 21:18:46.080218 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:46.080192    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs"]
Sep 10 21:18:46.084146 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:46.084128    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fc0caf1f-7cf54d5bbf-bggcs"]
Sep 10 21:18:47.439834 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:47.439797    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b59fc6e-58c6-4ea8-b522-43405f330861" path="/var/lib/kubelet/pods/8b59fc6e-58c6-4ea8-b522-43405f330861/volumes"
Sep 10 21:18:53.584700 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:53.584663    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"]
Sep 10 21:18:55.033655 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:18:55.033609    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:19:08.604650 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:08.604586    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf" podUID="e8566c1a-43b7-4295-ad43-736c51b4f9f3" containerName="main" containerID="cri-o://e716f13a0ea69369eab173538a7aac6ef0376c9653573990844d8d679eb5af92" gracePeriod=45
Sep 10 21:19:08.657567 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:19:08.657542    2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8566c1a_43b7_4295_ad43_736c51b4f9f3.slice/crio-conmon-e716f13a0ea69369eab173538a7aac6ef0376c9653573990844d8d679eb5af92.scope\": RecentStats: unable to find data in memory cache]"
Sep 10 21:19:08.849735 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:08.849706    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:19:08.969587 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:08.969520    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8566c1a-43b7-4295-ad43-736c51b4f9f3-tls-certs\") pod \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\" (UID: \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\") "
Sep 10 21:19:08.969587 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:08.969557    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8566c1a-43b7-4295-ad43-736c51b4f9f3-home\") pod \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\" (UID: \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\") "
Sep 10 21:19:08.969587 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:08.969577    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8566c1a-43b7-4295-ad43-736c51b4f9f3-dshm\") pod \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\" (UID: \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\") "
Sep 10 21:19:08.969846 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:08.969594    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fq7rd\" (UniqueName: \"kubernetes.io/projected/e8566c1a-43b7-4295-ad43-736c51b4f9f3-kube-api-access-fq7rd\") pod \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\" (UID: \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\") "
Sep 10 21:19:08.969846 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:08.969633    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8566c1a-43b7-4295-ad43-736c51b4f9f3-tmp-dir\") pod \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\" (UID: \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\") "
Sep 10 21:19:08.969846 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:08.969669    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8566c1a-43b7-4295-ad43-736c51b4f9f3-model-cache\") pod \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\" (UID: \"e8566c1a-43b7-4295-ad43-736c51b4f9f3\") "
Sep 10 21:19:08.969846 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:08.969819    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8566c1a-43b7-4295-ad43-736c51b4f9f3-home" (OuterVolumeSpecName: "home") pod "e8566c1a-43b7-4295-ad43-736c51b4f9f3" (UID: "e8566c1a-43b7-4295-ad43-736c51b4f9f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:19:08.970080 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:08.969954    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8566c1a-43b7-4295-ad43-736c51b4f9f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8566c1a-43b7-4295-ad43-736c51b4f9f3" (UID: "e8566c1a-43b7-4295-ad43-736c51b4f9f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:19:08.970080 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:08.969975    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8566c1a-43b7-4295-ad43-736c51b4f9f3-home\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:19:08.970080 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:08.970023    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8566c1a-43b7-4295-ad43-736c51b4f9f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8566c1a-43b7-4295-ad43-736c51b4f9f3" (UID: "e8566c1a-43b7-4295-ad43-736c51b4f9f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:19:08.971964 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:08.971934    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8566c1a-43b7-4295-ad43-736c51b4f9f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8566c1a-43b7-4295-ad43-736c51b4f9f3" (UID: "e8566c1a-43b7-4295-ad43-736c51b4f9f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:19:08.972106 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:08.971965    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8566c1a-43b7-4295-ad43-736c51b4f9f3-dshm" (OuterVolumeSpecName: "dshm") pod "e8566c1a-43b7-4295-ad43-736c51b4f9f3" (UID: "e8566c1a-43b7-4295-ad43-736c51b4f9f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:19:08.972106 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:08.971970    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8566c1a-43b7-4295-ad43-736c51b4f9f3-kube-api-access-fq7rd" (OuterVolumeSpecName: "kube-api-access-fq7rd") pod "e8566c1a-43b7-4295-ad43-736c51b4f9f3" (UID: "e8566c1a-43b7-4295-ad43-736c51b4f9f3"). InnerVolumeSpecName "kube-api-access-fq7rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:19:09.071045 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:09.071020    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8566c1a-43b7-4295-ad43-736c51b4f9f3-tmp-dir\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:19:09.071045 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:09.071041    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8566c1a-43b7-4295-ad43-736c51b4f9f3-model-cache\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:19:09.071185 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:09.071050    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8566c1a-43b7-4295-ad43-736c51b4f9f3-tls-certs\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:19:09.071185 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:09.071058    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8566c1a-43b7-4295-ad43-736c51b4f9f3-dshm\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:19:09.071185 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:09.071067    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq7rd\" (UniqueName: \"kubernetes.io/projected/e8566c1a-43b7-4295-ad43-736c51b4f9f3-kube-api-access-fq7rd\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:19:09.173500 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:09.173473    2572 generic.go:356] "Generic (PLEG): container finished" podID="e8566c1a-43b7-4295-ad43-736c51b4f9f3" containerID="e716f13a0ea69369eab173538a7aac6ef0376c9653573990844d8d679eb5af92" exitCode=0
Sep 10 21:19:09.173609 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:09.173510    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf" event={"ID":"e8566c1a-43b7-4295-ad43-736c51b4f9f3","Type":"ContainerDied","Data":"e716f13a0ea69369eab173538a7aac6ef0376c9653573990844d8d679eb5af92"}
Sep 10 21:19:09.173609 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:09.173527    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"
Sep 10 21:19:09.173609 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:09.173539    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf" event={"ID":"e8566c1a-43b7-4295-ad43-736c51b4f9f3","Type":"ContainerDied","Data":"4f424b266ae97b3473220eedbd52d1624f1ae9652262ed35d1c8d7eeb2d92e93"}
Sep 10 21:19:09.173609 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:09.173555    2572 scope.go:117] "RemoveContainer" containerID="e716f13a0ea69369eab173538a7aac6ef0376c9653573990844d8d679eb5af92"
Sep 10 21:19:09.182946 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:09.182927    2572 scope.go:117] "RemoveContainer" containerID="e716f13a0ea69369eab173538a7aac6ef0376c9653573990844d8d679eb5af92"
Sep 10 21:19:09.183234 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:19:09.183216    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e716f13a0ea69369eab173538a7aac6ef0376c9653573990844d8d679eb5af92\": container with ID starting with e716f13a0ea69369eab173538a7aac6ef0376c9653573990844d8d679eb5af92 not found: ID does not exist" containerID="e716f13a0ea69369eab173538a7aac6ef0376c9653573990844d8d679eb5af92"
Sep 10 21:19:09.183306 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:09.183239    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e716f13a0ea69369eab173538a7aac6ef0376c9653573990844d8d679eb5af92"} err="failed to get container status \"e716f13a0ea69369eab173538a7aac6ef0376c9653573990844d8d679eb5af92\": rpc error: code = NotFound desc = could not find container \"e716f13a0ea69369eab173538a7aac6ef0376c9653573990844d8d679eb5af92\": container with ID starting with e716f13a0ea69369eab173538a7aac6ef0376c9653573990844d8d679eb5af92 not found: ID does not exist"
Sep 10 21:19:09.195430 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:09.195403    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"]
Sep 10 21:19:09.199522 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:09.199501    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-fc0caf1f/e2e-lifecycle-model-fc0caf1f-kserve-58f897879f-mszkf"]
Sep 10 21:19:09.439044 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:09.439015    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8566c1a-43b7-4295-ad43-736c51b4f9f3" path="/var/lib/kubelet/pods/e8566c1a-43b7-4295-ad43-736c51b4f9f3/volumes"
Sep 10 21:19:11.626355 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:11.626315    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:19:26.061536 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:26.061446    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:19:37.259675 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:37.259631    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:19:38.682533 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:38.682502    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7909-f9c439"]
Sep 10 21:19:38.682886 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:38.682847    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8566c1a-43b7-4295-ad43-736c51b4f9f3" containerName="main"
Sep 10 21:19:38.682886 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:38.682858    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8566c1a-43b7-4295-ad43-736c51b4f9f3" containerName="main"
Sep 10 21:19:38.682886 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:38.682875    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b59fc6e-58c6-4ea8-b522-43405f330861" containerName="payload-processing"
Sep 10 21:19:38.682886 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:38.682882    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b59fc6e-58c6-4ea8-b522-43405f330861" containerName="payload-processing"
Sep 10 21:19:38.683055 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:38.682947    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8566c1a-43b7-4295-ad43-736c51b4f9f3" containerName="main"
Sep 10 21:19:38.683055 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:38.682954    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b59fc6e-58c6-4ea8-b522-43405f330861" containerName="payload-processing"
Sep 10 21:19:38.686029 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:38.686014    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7909-f9c439"
Sep 10 21:19:38.689545 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:38.689526    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zd8vg\""
Sep 10 21:19:38.696869 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:38.696843    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7909-f9c439"]
Sep 10 21:19:38.805425 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:38.805397    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlj68\" (UniqueName: \"kubernetes.io/projected/e78e5bc8-14ae-48dd-a425-88d3d742e199-kube-api-access-jlj68\") pod \"test-curl-7909-f9c439\" (UID: \"e78e5bc8-14ae-48dd-a425-88d3d742e199\") " pod="opendatahub/test-curl-7909-f9c439"
Sep 10 21:19:38.905909 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:38.905884    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlj68\" (UniqueName: \"kubernetes.io/projected/e78e5bc8-14ae-48dd-a425-88d3d742e199-kube-api-access-jlj68\") pod \"test-curl-7909-f9c439\" (UID: \"e78e5bc8-14ae-48dd-a425-88d3d742e199\") " pod="opendatahub/test-curl-7909-f9c439"
Sep 10 21:19:38.917425 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:38.917403    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlj68\" (UniqueName: \"kubernetes.io/projected/e78e5bc8-14ae-48dd-a425-88d3d742e199-kube-api-access-jlj68\") pod \"test-curl-7909-f9c439\" (UID: \"e78e5bc8-14ae-48dd-a425-88d3d742e199\") " pod="opendatahub/test-curl-7909-f9c439"
Sep 10 21:19:38.995688 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:38.995638    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7909-f9c439"
Sep 10 21:19:39.295447 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:39.295423    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7909-f9c439"]
Sep 10 21:19:39.296045 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:19:39.296015    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode78e5bc8_14ae_48dd_a425_88d3d742e199.slice/crio-ced8a1bfacfb23f421144952ed66026c4ff03253b2d8a3e697e6b753f163e1f5 WatchSource:0}: Error finding container ced8a1bfacfb23f421144952ed66026c4ff03253b2d8a3e697e6b753f163e1f5: Status 404 returned error can't find the container with id ced8a1bfacfb23f421144952ed66026c4ff03253b2d8a3e697e6b753f163e1f5
Sep 10 21:19:39.333824 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:39.333795    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7909-f9c439" event={"ID":"e78e5bc8-14ae-48dd-a425-88d3d742e199","Type":"ContainerStarted","Data":"ced8a1bfacfb23f421144952ed66026c4ff03253b2d8a3e697e6b753f163e1f5"}
Sep 10 21:19:40.340646 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:40.340594    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7909-f9c439" event={"ID":"e78e5bc8-14ae-48dd-a425-88d3d742e199","Type":"ContainerStarted","Data":"0406fd5d10ba0fec92bba733aaccf6bb97f480d8e3b2a3c06e756e8626b06add"}
Sep 10 21:19:40.380899 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:40.380843    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7909-f9c439" podStartSLOduration=2.380825079 podStartE2EDuration="2.380825079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:19:40.379407282 +0000 UTC m=+1365.591872201" watchObservedRunningTime="2026-09-10 21:19:40.380825079 +0000 UTC m=+1365.593289997"
Sep 10 21:19:41.234698 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:41.234632    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7909-f9c439"]
Sep 10 21:19:41.251998 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:41.251952    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7909-f9c439"]
Sep 10 21:19:41.346478 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:19:41.346454    2572 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7909-f9c439" container="test-curl-7909-f9c439"
Sep 10 21:19:42.350834 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:42.350784    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7909-f9c439" podUID="e78e5bc8-14ae-48dd-a425-88d3d742e199" containerName="test-curl-7909-f9c439" containerID="cri-o://0406fd5d10ba0fec92bba733aaccf6bb97f480d8e3b2a3c06e756e8626b06add" gracePeriod=2
Sep 10 21:19:42.949848 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:42.949821    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"]
Sep 10 21:19:43.856151 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:43.856119    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"]
Sep 10 21:19:44.531056 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:44.531034    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7909-f9c439"
Sep 10 21:19:44.652152 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:44.652100    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlj68\" (UniqueName: \"kubernetes.io/projected/e78e5bc8-14ae-48dd-a425-88d3d742e199-kube-api-access-jlj68\") pod \"e78e5bc8-14ae-48dd-a425-88d3d742e199\" (UID: \"e78e5bc8-14ae-48dd-a425-88d3d742e199\") "
Sep 10 21:19:44.654245 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:44.654223    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78e5bc8-14ae-48dd-a425-88d3d742e199-kube-api-access-jlj68" (OuterVolumeSpecName: "kube-api-access-jlj68") pod "e78e5bc8-14ae-48dd-a425-88d3d742e199" (UID: "e78e5bc8-14ae-48dd-a425-88d3d742e199"). InnerVolumeSpecName "kube-api-access-jlj68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:19:44.752863 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:44.752824    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlj68\" (UniqueName: \"kubernetes.io/projected/e78e5bc8-14ae-48dd-a425-88d3d742e199-kube-api-access-jlj68\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:19:45.369144 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:45.369116    2572 generic.go:356] "Generic (PLEG): container finished" podID="e78e5bc8-14ae-48dd-a425-88d3d742e199" containerID="0406fd5d10ba0fec92bba733aaccf6bb97f480d8e3b2a3c06e756e8626b06add" exitCode=137
Sep 10 21:19:45.369555 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:45.369177    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7909-f9c439"
Sep 10 21:19:45.369555 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:45.369228    2572 scope.go:117] "RemoveContainer" containerID="0406fd5d10ba0fec92bba733aaccf6bb97f480d8e3b2a3c06e756e8626b06add"
Sep 10 21:19:45.378712 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:45.378696    2572 scope.go:117] "RemoveContainer" containerID="0406fd5d10ba0fec92bba733aaccf6bb97f480d8e3b2a3c06e756e8626b06add"
Sep 10 21:19:45.378996 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:19:45.378962    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0406fd5d10ba0fec92bba733aaccf6bb97f480d8e3b2a3c06e756e8626b06add\": container with ID starting with 0406fd5d10ba0fec92bba733aaccf6bb97f480d8e3b2a3c06e756e8626b06add not found: ID does not exist" containerID="0406fd5d10ba0fec92bba733aaccf6bb97f480d8e3b2a3c06e756e8626b06add"
Sep 10 21:19:45.379056 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:45.379007    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0406fd5d10ba0fec92bba733aaccf6bb97f480d8e3b2a3c06e756e8626b06add"} err="failed to get container status \"0406fd5d10ba0fec92bba733aaccf6bb97f480d8e3b2a3c06e756e8626b06add\": rpc error: code = NotFound desc = could not find container \"0406fd5d10ba0fec92bba733aaccf6bb97f480d8e3b2a3c06e756e8626b06add\": container with ID starting with 0406fd5d10ba0fec92bba733aaccf6bb97f480d8e3b2a3c06e756e8626b06add not found: ID does not exist"
Sep 10 21:19:45.380048 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:19:45.380026    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7909-f9c439\" is forbidden: User \"system:node:ip-10-0-138-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-232.ec2.internal' and this object" podUID="e78e5bc8-14ae-48dd-a425-88d3d742e199" pod="opendatahub/test-curl-7909-f9c439"
Sep 10 21:19:45.442746 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:19:45.442714    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7909-f9c439\" is forbidden: User \"system:node:ip-10-0-138-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-232.ec2.internal' and this object" podUID="e78e5bc8-14ae-48dd-a425-88d3d742e199" pod="opendatahub/test-curl-7909-f9c439"
Sep 10 21:19:45.444065 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:45.444042    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e78e5bc8-14ae-48dd-a425-88d3d742e199" path="/var/lib/kubelet/pods/e78e5bc8-14ae-48dd-a425-88d3d742e199/volumes"
Sep 10 21:19:54.252201 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:54.252159    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:19:57.968947 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:57.968897    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f" podUID="42e148d4-91bb-45aa-813f-e4aba8bd1952" containerName="main" containerID="cri-o://3093823d5d14a8b47643e91afb6c0b5a8963863044a5b2fa16e446b824e77e1f" gracePeriod=45
Sep 10 21:19:58.219475 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.219419    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:19:58.265507 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.265480    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e148d4-91bb-45aa-813f-e4aba8bd1952-tls-certs\") pod \"42e148d4-91bb-45aa-813f-e4aba8bd1952\" (UID: \"42e148d4-91bb-45aa-813f-e4aba8bd1952\") "
Sep 10 21:19:58.265629 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.265542    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e148d4-91bb-45aa-813f-e4aba8bd1952-model-cache\") pod \"42e148d4-91bb-45aa-813f-e4aba8bd1952\" (UID: \"42e148d4-91bb-45aa-813f-e4aba8bd1952\") "
Sep 10 21:19:58.265629 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.265599    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e148d4-91bb-45aa-813f-e4aba8bd1952-tmp-dir\") pod \"42e148d4-91bb-45aa-813f-e4aba8bd1952\" (UID: \"42e148d4-91bb-45aa-813f-e4aba8bd1952\") "
Sep 10 21:19:58.265755 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.265630    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e148d4-91bb-45aa-813f-e4aba8bd1952-dshm\") pod \"42e148d4-91bb-45aa-813f-e4aba8bd1952\" (UID: \"42e148d4-91bb-45aa-813f-e4aba8bd1952\") "
Sep 10 21:19:58.265755 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.265659    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fz75\" (UniqueName: \"kubernetes.io/projected/42e148d4-91bb-45aa-813f-e4aba8bd1952-kube-api-access-8fz75\") pod \"42e148d4-91bb-45aa-813f-e4aba8bd1952\" (UID: \"42e148d4-91bb-45aa-813f-e4aba8bd1952\") "
Sep 10 21:19:58.265755 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.265682    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42e148d4-91bb-45aa-813f-e4aba8bd1952-home\") pod \"42e148d4-91bb-45aa-813f-e4aba8bd1952\" (UID: \"42e148d4-91bb-45aa-813f-e4aba8bd1952\") "
Sep 10 21:19:58.265927 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.265883    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e148d4-91bb-45aa-813f-e4aba8bd1952-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "42e148d4-91bb-45aa-813f-e4aba8bd1952" (UID: "42e148d4-91bb-45aa-813f-e4aba8bd1952"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:19:58.266103 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.266071    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e148d4-91bb-45aa-813f-e4aba8bd1952-tmp-dir\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:19:58.266103 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.266083    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e148d4-91bb-45aa-813f-e4aba8bd1952-home" (OuterVolumeSpecName: "home") pod "42e148d4-91bb-45aa-813f-e4aba8bd1952" (UID: "42e148d4-91bb-45aa-813f-e4aba8bd1952"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:19:58.266382 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.266355    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e148d4-91bb-45aa-813f-e4aba8bd1952-model-cache" (OuterVolumeSpecName: "model-cache") pod "42e148d4-91bb-45aa-813f-e4aba8bd1952" (UID: "42e148d4-91bb-45aa-813f-e4aba8bd1952"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:19:58.268065 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.268028    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42e148d4-91bb-45aa-813f-e4aba8bd1952-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "42e148d4-91bb-45aa-813f-e4aba8bd1952" (UID: "42e148d4-91bb-45aa-813f-e4aba8bd1952"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:19:58.268065 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.268042    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e148d4-91bb-45aa-813f-e4aba8bd1952-kube-api-access-8fz75" (OuterVolumeSpecName: "kube-api-access-8fz75") pod "42e148d4-91bb-45aa-813f-e4aba8bd1952" (UID: "42e148d4-91bb-45aa-813f-e4aba8bd1952"). InnerVolumeSpecName "kube-api-access-8fz75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:19:58.268228 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.268109    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e148d4-91bb-45aa-813f-e4aba8bd1952-dshm" (OuterVolumeSpecName: "dshm") pod "42e148d4-91bb-45aa-813f-e4aba8bd1952" (UID: "42e148d4-91bb-45aa-813f-e4aba8bd1952"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:19:58.367428 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.367404    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e148d4-91bb-45aa-813f-e4aba8bd1952-model-cache\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:19:58.367428 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.367424    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e148d4-91bb-45aa-813f-e4aba8bd1952-dshm\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:19:58.367549 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.367433    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fz75\" (UniqueName: \"kubernetes.io/projected/42e148d4-91bb-45aa-813f-e4aba8bd1952-kube-api-access-8fz75\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:19:58.367549 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.367443    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42e148d4-91bb-45aa-813f-e4aba8bd1952-home\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:19:58.367549 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.367452    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e148d4-91bb-45aa-813f-e4aba8bd1952-tls-certs\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:19:58.438119 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.438090    2572 generic.go:356] "Generic (PLEG): container finished" podID="42e148d4-91bb-45aa-813f-e4aba8bd1952" containerID="3093823d5d14a8b47643e91afb6c0b5a8963863044a5b2fa16e446b824e77e1f" exitCode=0
Sep 10 21:19:58.438213 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.438126    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"
Sep 10 21:19:58.438213 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.438162    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f" event={"ID":"42e148d4-91bb-45aa-813f-e4aba8bd1952","Type":"ContainerDied","Data":"3093823d5d14a8b47643e91afb6c0b5a8963863044a5b2fa16e446b824e77e1f"}
Sep 10 21:19:58.438213 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.438200    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f" event={"ID":"42e148d4-91bb-45aa-813f-e4aba8bd1952","Type":"ContainerDied","Data":"adadfbbdb6cf1a83bf08404400024a224d620f8216a74272ee6bdfc1855724ef"}
Sep 10 21:19:58.438321 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.438217    2572 scope.go:117] "RemoveContainer" containerID="3093823d5d14a8b47643e91afb6c0b5a8963863044a5b2fa16e446b824e77e1f"
Sep 10 21:19:58.450473 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.450457    2572 scope.go:117] "RemoveContainer" containerID="3093823d5d14a8b47643e91afb6c0b5a8963863044a5b2fa16e446b824e77e1f"
Sep 10 21:19:58.450746 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:19:58.450729    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3093823d5d14a8b47643e91afb6c0b5a8963863044a5b2fa16e446b824e77e1f\": container with ID starting with 3093823d5d14a8b47643e91afb6c0b5a8963863044a5b2fa16e446b824e77e1f not found: ID does not exist" containerID="3093823d5d14a8b47643e91afb6c0b5a8963863044a5b2fa16e446b824e77e1f"
Sep 10 21:19:58.450789 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.450755    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3093823d5d14a8b47643e91afb6c0b5a8963863044a5b2fa16e446b824e77e1f"} err="failed to get container status \"3093823d5d14a8b47643e91afb6c0b5a8963863044a5b2fa16e446b824e77e1f\": rpc error: code = NotFound desc = could not find container \"3093823d5d14a8b47643e91afb6c0b5a8963863044a5b2fa16e446b824e77e1f\": container with ID starting with 3093823d5d14a8b47643e91afb6c0b5a8963863044a5b2fa16e446b824e77e1f not found: ID does not exist"
Sep 10 21:19:58.459556 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.459531    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"]
Sep 10 21:19:58.463598 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.463578    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7150/auth-test-model-af7150-kserve-b8c9b7547-bld2f"]
Sep 10 21:19:58.876053 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:58.876026    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h" podUID="8e983c34-05ee-4402-a29c-43a82181c24d" containerName="main" containerID="cri-o://6a52fba73ba0049ac5c2c255d8351e678dc407d1d017edc313503e47e8d4a5b4" gracePeriod=45
Sep 10 21:19:59.121823 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.121802    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:19:59.173181 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.173129    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e983c34-05ee-4402-a29c-43a82181c24d-tls-certs\") pod \"8e983c34-05ee-4402-a29c-43a82181c24d\" (UID: \"8e983c34-05ee-4402-a29c-43a82181c24d\") "
Sep 10 21:19:59.173303 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.173186    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e983c34-05ee-4402-a29c-43a82181c24d-model-cache\") pod \"8e983c34-05ee-4402-a29c-43a82181c24d\" (UID: \"8e983c34-05ee-4402-a29c-43a82181c24d\") "
Sep 10 21:19:59.173303 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.173204    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e983c34-05ee-4402-a29c-43a82181c24d-home\") pod \"8e983c34-05ee-4402-a29c-43a82181c24d\" (UID: \"8e983c34-05ee-4402-a29c-43a82181c24d\") "
Sep 10 21:19:59.173303 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.173219    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e983c34-05ee-4402-a29c-43a82181c24d-tmp-dir\") pod \"8e983c34-05ee-4402-a29c-43a82181c24d\" (UID: \"8e983c34-05ee-4402-a29c-43a82181c24d\") "
Sep 10 21:19:59.173303 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.173241    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-449jn\" (UniqueName: \"kubernetes.io/projected/8e983c34-05ee-4402-a29c-43a82181c24d-kube-api-access-449jn\") pod \"8e983c34-05ee-4402-a29c-43a82181c24d\" (UID: \"8e983c34-05ee-4402-a29c-43a82181c24d\") "
Sep 10 21:19:59.173303 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.173279    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e983c34-05ee-4402-a29c-43a82181c24d-dshm\") pod \"8e983c34-05ee-4402-a29c-43a82181c24d\" (UID: \"8e983c34-05ee-4402-a29c-43a82181c24d\") "
Sep 10 21:19:59.173573 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.173490    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e983c34-05ee-4402-a29c-43a82181c24d-home" (OuterVolumeSpecName: "home") pod "8e983c34-05ee-4402-a29c-43a82181c24d" (UID: "8e983c34-05ee-4402-a29c-43a82181c24d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:19:59.173573 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.173506    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e983c34-05ee-4402-a29c-43a82181c24d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8e983c34-05ee-4402-a29c-43a82181c24d" (UID: "8e983c34-05ee-4402-a29c-43a82181c24d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:19:59.173573 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.173522    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e983c34-05ee-4402-a29c-43a82181c24d-model-cache" (OuterVolumeSpecName: "model-cache") pod "8e983c34-05ee-4402-a29c-43a82181c24d" (UID: "8e983c34-05ee-4402-a29c-43a82181c24d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:19:59.175525 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.175495    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e983c34-05ee-4402-a29c-43a82181c24d-kube-api-access-449jn" (OuterVolumeSpecName: "kube-api-access-449jn") pod "8e983c34-05ee-4402-a29c-43a82181c24d" (UID: "8e983c34-05ee-4402-a29c-43a82181c24d"). InnerVolumeSpecName "kube-api-access-449jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:19:59.175525 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.175507    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e983c34-05ee-4402-a29c-43a82181c24d-dshm" (OuterVolumeSpecName: "dshm") pod "8e983c34-05ee-4402-a29c-43a82181c24d" (UID: "8e983c34-05ee-4402-a29c-43a82181c24d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:19:59.175525 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.175501    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e983c34-05ee-4402-a29c-43a82181c24d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e983c34-05ee-4402-a29c-43a82181c24d" (UID: "8e983c34-05ee-4402-a29c-43a82181c24d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:19:59.274267 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.274244    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-449jn\" (UniqueName: \"kubernetes.io/projected/8e983c34-05ee-4402-a29c-43a82181c24d-kube-api-access-449jn\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:19:59.274267 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.274265    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e983c34-05ee-4402-a29c-43a82181c24d-dshm\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:19:59.274411 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.274276    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e983c34-05ee-4402-a29c-43a82181c24d-tls-certs\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:19:59.274411 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.274285    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e983c34-05ee-4402-a29c-43a82181c24d-model-cache\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:19:59.274411 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.274292    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e983c34-05ee-4402-a29c-43a82181c24d-home\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:19:59.274411 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.274300    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e983c34-05ee-4402-a29c-43a82181c24d-tmp-dir\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:19:59.438863 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.438789    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42e148d4-91bb-45aa-813f-e4aba8bd1952" path="/var/lib/kubelet/pods/42e148d4-91bb-45aa-813f-e4aba8bd1952/volumes"
Sep 10 21:19:59.447157 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.447130    2572 generic.go:356] "Generic (PLEG): container finished" podID="8e983c34-05ee-4402-a29c-43a82181c24d" containerID="6a52fba73ba0049ac5c2c255d8351e678dc407d1d017edc313503e47e8d4a5b4" exitCode=0
Sep 10 21:19:59.447284 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.447175    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"
Sep 10 21:19:59.447284 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.447201    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h" event={"ID":"8e983c34-05ee-4402-a29c-43a82181c24d","Type":"ContainerDied","Data":"6a52fba73ba0049ac5c2c255d8351e678dc407d1d017edc313503e47e8d4a5b4"}
Sep 10 21:19:59.447284 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.447257    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h" event={"ID":"8e983c34-05ee-4402-a29c-43a82181c24d","Type":"ContainerDied","Data":"57b2717c6e55abbcdc74008b5f9d8505b63cbefdb0af5bf05c5e0f6b6bdd7401"}
Sep 10 21:19:59.447448 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.447285    2572 scope.go:117] "RemoveContainer" containerID="6a52fba73ba0049ac5c2c255d8351e678dc407d1d017edc313503e47e8d4a5b4"
Sep 10 21:19:59.457143 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.457123    2572 scope.go:117] "RemoveContainer" containerID="6a52fba73ba0049ac5c2c255d8351e678dc407d1d017edc313503e47e8d4a5b4"
Sep 10 21:19:59.457416 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:19:59.457397    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a52fba73ba0049ac5c2c255d8351e678dc407d1d017edc313503e47e8d4a5b4\": container with ID starting with 6a52fba73ba0049ac5c2c255d8351e678dc407d1d017edc313503e47e8d4a5b4 not found: ID does not exist" containerID="6a52fba73ba0049ac5c2c255d8351e678dc407d1d017edc313503e47e8d4a5b4"
Sep 10 21:19:59.457462 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.457424    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a52fba73ba0049ac5c2c255d8351e678dc407d1d017edc313503e47e8d4a5b4"} err="failed to get container status \"6a52fba73ba0049ac5c2c255d8351e678dc407d1d017edc313503e47e8d4a5b4\": rpc error: code = NotFound desc = could not find container \"6a52fba73ba0049ac5c2c255d8351e678dc407d1d017edc313503e47e8d4a5b4\": container with ID starting with 6a52fba73ba0049ac5c2c255d8351e678dc407d1d017edc313503e47e8d4a5b4 not found: ID does not exist"
Sep 10 21:19:59.464413 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.464388    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"]
Sep 10 21:19:59.474066 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:19:59.474043    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7810bd/auth-test-model-7810bd-kserve-56cf77d95f-k5z4h"]
Sep 10 21:20:01.444785 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.444749    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e983c34-05ee-4402-a29c-43a82181c24d" path="/var/lib/kubelet/pods/8e983c34-05ee-4402-a29c-43a82181c24d/volumes"
Sep 10 21:20:01.519311 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.519281    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"]
Sep 10 21:20:01.519648 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.519634    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e983c34-05ee-4402-a29c-43a82181c24d" containerName="main"
Sep 10 21:20:01.519690 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.519650    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e983c34-05ee-4402-a29c-43a82181c24d" containerName="main"
Sep 10 21:20:01.519690 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.519662    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e78e5bc8-14ae-48dd-a425-88d3d742e199" containerName="test-curl-7909-f9c439"
Sep 10 21:20:01.519690 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.519667    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e78e5bc8-14ae-48dd-a425-88d3d742e199" containerName="test-curl-7909-f9c439"
Sep 10 21:20:01.519690 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.519675    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42e148d4-91bb-45aa-813f-e4aba8bd1952" containerName="main"
Sep 10 21:20:01.519690 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.519680    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="42e148d4-91bb-45aa-813f-e4aba8bd1952" containerName="main"
Sep 10 21:20:01.519837 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.519739    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="42e148d4-91bb-45aa-813f-e4aba8bd1952" containerName="main"
Sep 10 21:20:01.519837 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.519748    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e78e5bc8-14ae-48dd-a425-88d3d742e199" containerName="test-curl-7909-f9c439"
Sep 10 21:20:01.519837 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.519755    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e983c34-05ee-4402-a29c-43a82181c24d" containerName="main"
Sep 10 21:20:01.523899 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.523884    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:20:01.526246 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.526224    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-af7150\"/\"kube-root-ca.crt\""
Sep 10 21:20:01.526351 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.526325    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-af7150\"/\"sub-test-model-af7150-kserve-self-signed-certs\""
Sep 10 21:20:01.527219 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.527205    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-af7150\"/\"default-dockercfg-cwcjx\""
Sep 10 21:20:01.527279 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.527239    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-af7150\"/\"openshift-service-ca.crt\""
Sep 10 21:20:01.533416 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.533396    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"]
Sep 10 21:20:01.591296 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.591271    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-tls-certs\") pod \"sub-test-model-af7150-kserve-587fc5b987-w2t5c\" (UID: \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\") " pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:20:01.591406 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.591359    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-tmp-dir\") pod \"sub-test-model-af7150-kserve-587fc5b987-w2t5c\" (UID: \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\") " pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:20:01.591406 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.591381    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knhn5\" (UniqueName: \"kubernetes.io/projected/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-kube-api-access-knhn5\") pod \"sub-test-model-af7150-kserve-587fc5b987-w2t5c\" (UID: \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\") " pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:20:01.591406 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.591401    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-model-cache\") pod \"sub-test-model-af7150-kserve-587fc5b987-w2t5c\" (UID: \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\") " pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:20:01.591510 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.591474    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-dshm\") pod \"sub-test-model-af7150-kserve-587fc5b987-w2t5c\" (UID: \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\") " pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:20:01.591510 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.591498    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-home\") pod \"sub-test-model-af7150-kserve-587fc5b987-w2t5c\" (UID: \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\") " pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:20:01.692528 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.692504    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-model-cache\") pod \"sub-test-model-af7150-kserve-587fc5b987-w2t5c\" (UID: \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\") " pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:20:01.692604 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.692545    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-dshm\") pod \"sub-test-model-af7150-kserve-587fc5b987-w2t5c\" (UID: \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\") " pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:20:01.692604 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.692562    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-home\") pod \"sub-test-model-af7150-kserve-587fc5b987-w2t5c\" (UID: \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\") " pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:20:01.692604 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.692580    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-tls-certs\") pod \"sub-test-model-af7150-kserve-587fc5b987-w2t5c\" (UID: \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\") " pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:20:01.692710 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.692640    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-tmp-dir\") pod \"sub-test-model-af7150-kserve-587fc5b987-w2t5c\" (UID: \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\") " pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:20:01.692710 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.692655    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knhn5\" (UniqueName: \"kubernetes.io/projected/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-kube-api-access-knhn5\") pod \"sub-test-model-af7150-kserve-587fc5b987-w2t5c\" (UID: \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\") " pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:20:01.692923 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.692903    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-model-cache\") pod \"sub-test-model-af7150-kserve-587fc5b987-w2t5c\" (UID: \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\") " pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:20:01.693015 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.692950    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-home\") pod \"sub-test-model-af7150-kserve-587fc5b987-w2t5c\" (UID: \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\") " pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:20:01.693079 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.693042    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-tmp-dir\") pod \"sub-test-model-af7150-kserve-587fc5b987-w2t5c\" (UID: \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\") " pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:20:01.695167 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.695120    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-dshm\") pod \"sub-test-model-af7150-kserve-587fc5b987-w2t5c\" (UID: \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\") " pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:20:01.695412 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.695394    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-tls-certs\") pod \"sub-test-model-af7150-kserve-587fc5b987-w2t5c\" (UID: \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\") " pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:20:01.701212 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.701186    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knhn5\" (UniqueName: \"kubernetes.io/projected/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-kube-api-access-knhn5\") pod \"sub-test-model-af7150-kserve-587fc5b987-w2t5c\" (UID: \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\") " pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:20:01.833480 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:01.833455    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:20:02.109208 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:02.109102    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"]
Sep 10 21:20:02.112251 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:20:02.112220    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59bd12c2_ea8f_4fbf_aed7_c2a03a754310.slice/crio-7527aed4cc28ce21535cc4b86b9d5bc90002fcf1cae4c19addc08f5681360c96 WatchSource:0}: Error finding container 7527aed4cc28ce21535cc4b86b9d5bc90002fcf1cae4c19addc08f5681360c96: Status 404 returned error can't find the container with id 7527aed4cc28ce21535cc4b86b9d5bc90002fcf1cae4c19addc08f5681360c96
Sep 10 21:20:02.464941 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:02.464846    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c" event={"ID":"59bd12c2-ea8f-4fbf-aed7-c2a03a754310","Type":"ContainerStarted","Data":"4289521c03032086c43a2e6298df5a0242c34abf713ea6e51d4b99d7faaca1d9"}
Sep 10 21:20:02.464941 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:02.464890    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c" event={"ID":"59bd12c2-ea8f-4fbf-aed7-c2a03a754310","Type":"ContainerStarted","Data":"7527aed4cc28ce21535cc4b86b9d5bc90002fcf1cae4c19addc08f5681360c96"}
Sep 10 21:20:02.483784 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:02.483740    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c" podStartSLOduration=1.483723678 podStartE2EDuration="1.483723678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:02.482863462 +0000 UTC m=+1387.695328381" watchObservedRunningTime="2026-09-10 21:20:02.483723678 +0000 UTC m=+1387.696188597"
Sep 10 21:20:06.632255 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:06.632215    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:20:11.834304 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:11.834257    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:20:11.834304 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:11.834323    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:20:11.846876 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:11.846841    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:20:12.524437 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:12.524411    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:20:33.668463 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.668426    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"]
Sep 10 21:20:33.671678 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.671661    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:20:33.679448 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.679415    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7810bd\"/\"kube-root-ca.crt\""
Sep 10 21:20:33.679448 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.679443    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7810bd\"/\"default-dockercfg-lxqpv\""
Sep 10 21:20:33.679627 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.679426    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7810bd\"/\"sub-test-model-7810bd-kserve-self-signed-certs\""
Sep 10 21:20:33.679627 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.679426    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7810bd\"/\"openshift-service-ca.crt\""
Sep 10 21:20:33.710779 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.710744    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"]
Sep 10 21:20:33.749739 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.749719    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-tls-certs\") pod \"sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr\" (UID: \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:20:33.749845 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.749747    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw6v4\" (UniqueName: \"kubernetes.io/projected/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-kube-api-access-dw6v4\") pod \"sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr\" (UID: \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:20:33.749845 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.749779    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-home\") pod \"sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr\" (UID: \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:20:33.749930 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.749847    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-dshm\") pod \"sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr\" (UID: \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:20:33.749930 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.749884    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-model-cache\") pod \"sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr\" (UID: \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:20:33.750034 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.749944    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-tmp-dir\") pod \"sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr\" (UID: \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:20:33.851044 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.851011    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-dshm\") pod \"sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr\" (UID: \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:20:33.851136 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.851051    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-model-cache\") pod \"sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr\" (UID: \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:20:33.851136 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.851095    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-tmp-dir\") pod \"sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr\" (UID: \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:20:33.851136 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.851114    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-tls-certs\") pod \"sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr\" (UID: \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:20:33.851136 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.851130    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw6v4\" (UniqueName: \"kubernetes.io/projected/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-kube-api-access-dw6v4\") pod \"sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr\" (UID: \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:20:33.851335 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.851162    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-home\") pod \"sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr\" (UID: \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:20:33.851542 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.851512    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-tmp-dir\") pod \"sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr\" (UID: \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:20:33.851626 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.851547    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-home\") pod \"sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr\" (UID: \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:20:33.851626 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.851512    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-model-cache\") pod \"sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr\" (UID: \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:20:33.853572 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.853549    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-dshm\") pod \"sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr\" (UID: \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:20:33.853776 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.853758    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-tls-certs\") pod \"sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr\" (UID: \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:20:33.861050 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.861032    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw6v4\" (UniqueName: \"kubernetes.io/projected/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-kube-api-access-dw6v4\") pod \"sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr\" (UID: \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\") " pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:20:33.981682 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:33.981618    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:20:34.274794 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:34.274771    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"]
Sep 10 21:20:34.277420 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:20:34.277396    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c4d78f2_0bab_4e03_8799_4ee9f4fff6e4.slice/crio-10b09c13b3d31b2c68b1fd3f8aabd73db87dac224bc75a3b2f6be1febc9ce855 WatchSource:0}: Error finding container 10b09c13b3d31b2c68b1fd3f8aabd73db87dac224bc75a3b2f6be1febc9ce855: Status 404 returned error can't find the container with id 10b09c13b3d31b2c68b1fd3f8aabd73db87dac224bc75a3b2f6be1febc9ce855
Sep 10 21:20:34.633113 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:34.633080    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr" event={"ID":"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4","Type":"ContainerStarted","Data":"e757450dc2fe1e5d14651a39774504ec2d5c5c67c8d20fac356851d3ea5d07db"}
Sep 10 21:20:34.633276 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:34.633122    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr" event={"ID":"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4","Type":"ContainerStarted","Data":"10b09c13b3d31b2c68b1fd3f8aabd73db87dac224bc75a3b2f6be1febc9ce855"}
Sep 10 21:20:38.141786 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:38.141727    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr" podStartSLOduration=5.141707951 podStartE2EDuration="5.141707951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:34.67345749 +0000 UTC m=+1419.885922408" watchObservedRunningTime="2026-09-10 21:20:38.141707951 +0000 UTC m=+1423.354172903"
Sep 10 21:20:38.142784 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:38.142762    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:20:43.982334 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:43.982302    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:20:43.982824 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:43.982367    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:20:43.994512 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:43.994486    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:20:44.696499 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:44.696472    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:20:52.129489 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:20:52.129409    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:21:08.575036 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:08.575003    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7909-f9fd2c"]
Sep 10 21:21:08.579817 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:08.579790    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7909-f9fd2c"
Sep 10 21:21:08.583085 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:08.583068    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zd8vg\""
Sep 10 21:21:08.588156 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:08.588133    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7909-f9fd2c"]
Sep 10 21:21:08.634594 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:08.634571    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz7cc\" (UniqueName: \"kubernetes.io/projected/efdf58c2-cabc-4824-b448-517244d818ac-kube-api-access-pz7cc\") pod \"test-curl-7909-f9fd2c\" (UID: \"efdf58c2-cabc-4824-b448-517244d818ac\") " pod="opendatahub/test-curl-7909-f9fd2c"
Sep 10 21:21:08.735580 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:08.735556    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz7cc\" (UniqueName: \"kubernetes.io/projected/efdf58c2-cabc-4824-b448-517244d818ac-kube-api-access-pz7cc\") pod \"test-curl-7909-f9fd2c\" (UID: \"efdf58c2-cabc-4824-b448-517244d818ac\") " pod="opendatahub/test-curl-7909-f9fd2c"
Sep 10 21:21:08.747371 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:08.747347    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz7cc\" (UniqueName: \"kubernetes.io/projected/efdf58c2-cabc-4824-b448-517244d818ac-kube-api-access-pz7cc\") pod \"test-curl-7909-f9fd2c\" (UID: \"efdf58c2-cabc-4824-b448-517244d818ac\") " pod="opendatahub/test-curl-7909-f9fd2c"
Sep 10 21:21:08.889248 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:08.889196    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7909-f9fd2c"
Sep 10 21:21:09.158308 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:09.158270    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7909-f9fd2c"]
Sep 10 21:21:09.160612 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:21:09.160582    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefdf58c2_cabc_4824_b448_517244d818ac.slice/crio-40a8cb4e3d02e38878dee0301fa6f2e32efa789480a37dcfdc36cfce137ee7d4 WatchSource:0}: Error finding container 40a8cb4e3d02e38878dee0301fa6f2e32efa789480a37dcfdc36cfce137ee7d4: Status 404 returned error can't find the container with id 40a8cb4e3d02e38878dee0301fa6f2e32efa789480a37dcfdc36cfce137ee7d4
Sep 10 21:21:09.817434 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:09.817402    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7909-f9fd2c" event={"ID":"efdf58c2-cabc-4824-b448-517244d818ac","Type":"ContainerStarted","Data":"08c1e3df0e24606883cf8605256900657061a34d04809837ec590abbca037a24"}
Sep 10 21:21:09.817797 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:09.817444    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7909-f9fd2c" event={"ID":"efdf58c2-cabc-4824-b448-517244d818ac","Type":"ContainerStarted","Data":"40a8cb4e3d02e38878dee0301fa6f2e32efa789480a37dcfdc36cfce137ee7d4"}
Sep 10 21:21:09.845422 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:09.845377    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7909-f9fd2c" podStartSLOduration=1.845363031 podStartE2EDuration="1.845363031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:21:09.8450832 +0000 UTC m=+1455.057548120" watchObservedRunningTime="2026-09-10 21:21:09.845363031 +0000 UTC m=+1455.057827947"
Sep 10 21:21:10.474749 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:10.474721    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7909-f9fd2c"]
Sep 10 21:21:10.478688 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:10.478661    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7909-f9fd2c"]
Sep 10 21:21:10.596141 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:10.596115    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7909-bcedc5"]
Sep 10 21:21:10.596478 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:10.596466    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efdf58c2-cabc-4824-b448-517244d818ac" containerName="test-curl-7909-f9fd2c"
Sep 10 21:21:10.596522 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:10.596480    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="efdf58c2-cabc-4824-b448-517244d818ac" containerName="test-curl-7909-f9fd2c"
Sep 10 21:21:10.596564 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:10.596555    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="efdf58c2-cabc-4824-b448-517244d818ac" containerName="test-curl-7909-f9fd2c"
Sep 10 21:21:10.599573 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:10.599558    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7909-bcedc5"
Sep 10 21:21:10.605775 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:10.605714    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7909-bcedc5"]
Sep 10 21:21:10.650336 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:10.650295    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w9nm\" (UniqueName: \"kubernetes.io/projected/ff8ec78f-e3ef-45ba-b0ab-f1ba0577c1aa-kube-api-access-5w9nm\") pod \"test-curl-7909-bcedc5\" (UID: \"ff8ec78f-e3ef-45ba-b0ab-f1ba0577c1aa\") " pod="opendatahub/test-curl-7909-bcedc5"
Sep 10 21:21:10.750787 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:10.750762    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w9nm\" (UniqueName: \"kubernetes.io/projected/ff8ec78f-e3ef-45ba-b0ab-f1ba0577c1aa-kube-api-access-5w9nm\") pod \"test-curl-7909-bcedc5\" (UID: \"ff8ec78f-e3ef-45ba-b0ab-f1ba0577c1aa\") " pod="opendatahub/test-curl-7909-bcedc5"
Sep 10 21:21:10.760473 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:10.760452    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w9nm\" (UniqueName: \"kubernetes.io/projected/ff8ec78f-e3ef-45ba-b0ab-f1ba0577c1aa-kube-api-access-5w9nm\") pod \"test-curl-7909-bcedc5\" (UID: \"ff8ec78f-e3ef-45ba-b0ab-f1ba0577c1aa\") " pod="opendatahub/test-curl-7909-bcedc5"
Sep 10 21:21:10.825176 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:21:10.825153    2572 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7909-f9fd2c" container="test-curl-7909-f9fd2c"
Sep 10 21:21:10.909823 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:10.909768    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7909-bcedc5"
Sep 10 21:21:11.181650 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:11.181627    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7909-bcedc5"]
Sep 10 21:21:11.183763 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:21:11.183735    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff8ec78f_e3ef_45ba_b0ab_f1ba0577c1aa.slice/crio-1983de208b74299dc220d62060c0f7d94263ae7c564977d2494e5a41f1b78ef6 WatchSource:0}: Error finding container 1983de208b74299dc220d62060c0f7d94263ae7c564977d2494e5a41f1b78ef6: Status 404 returned error can't find the container with id 1983de208b74299dc220d62060c0f7d94263ae7c564977d2494e5a41f1b78ef6
Sep 10 21:21:11.832068 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:11.832029    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7909-bcedc5" event={"ID":"ff8ec78f-e3ef-45ba-b0ab-f1ba0577c1aa","Type":"ContainerStarted","Data":"5289b9d3d8b87e46382c69e04aa2c228d6153ff0b89a41d805e12f17e2249e77"}
Sep 10 21:21:11.832430 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:11.832080    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7909-bcedc5" event={"ID":"ff8ec78f-e3ef-45ba-b0ab-f1ba0577c1aa","Type":"ContainerStarted","Data":"1983de208b74299dc220d62060c0f7d94263ae7c564977d2494e5a41f1b78ef6"}
Sep 10 21:21:11.832430 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:11.832295    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7909-f9fd2c" podUID="efdf58c2-cabc-4824-b448-517244d818ac" containerName="test-curl-7909-f9fd2c" containerID="cri-o://08c1e3df0e24606883cf8605256900657061a34d04809837ec590abbca037a24" gracePeriod=2
Sep 10 21:21:11.852101 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:11.852052    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7909-bcedc5" podStartSLOduration=1.852037976 podStartE2EDuration="1.852037976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:21:11.850478479 +0000 UTC m=+1457.062943398" watchObservedRunningTime="2026-09-10 21:21:11.852037976 +0000 UTC m=+1457.064502894"
Sep 10 21:21:12.416661 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.416636    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"]
Sep 10 21:21:12.420313 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.420293    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.425750 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.425724    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d55386-openshift-default-dockercfg-g9658\""
Sep 10 21:21:12.428198 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.428179    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 21:21:12.465088 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.465063    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istiod-ca-cert\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.465190 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.465108    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-credential-socket\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.465190 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.465132    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istio-token\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.465190 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.465154    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istio-podinfo\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.465343 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.465189    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istio-envoy\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.465343 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.465211    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istio-data\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.465343 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.465232    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-workload-certs\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.465343 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.465257    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-workload-socket\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.465343 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.465317    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5pn2\" (UniqueName: \"kubernetes.io/projected/079a8ef0-9f45-4e5b-bdbc-946e052e440d-kube-api-access-n5pn2\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.502487 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.502461    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"]
Sep 10 21:21:12.565846 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.565818    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istiod-ca-cert\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.565969 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.565892    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-credential-socket\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.565969 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.565928    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istio-token\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.565969 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.565960    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istio-podinfo\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.566170 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.566031    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istio-envoy\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.566170 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.566058    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istio-data\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.566170 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.566098    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-workload-certs\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.566170 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.566132    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-workload-socket\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.566464 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.566438    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5pn2\" (UniqueName: \"kubernetes.io/projected/079a8ef0-9f45-4e5b-bdbc-946e052e440d-kube-api-access-n5pn2\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.566571 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.566555    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istio-data\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.566643 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.566619    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istiod-ca-cert\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.566802 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.566670    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-credential-socket\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.567088 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.567069    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-workload-certs\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.567179 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.567146    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-workload-socket\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.568752 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.568730    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istio-podinfo\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.568752 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.568743    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istio-envoy\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.575772 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.575751    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5pn2\" (UniqueName: \"kubernetes.io/projected/079a8ef0-9f45-4e5b-bdbc-946e052e440d-kube-api-access-n5pn2\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.575874 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.575805    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istio-token\") pod \"e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") " pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.694402 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.694348    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7909-bcedc5"]
Sep 10 21:21:12.702606 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.702584    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7909-bcedc5"]
Sep 10 21:21:12.730785 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:12.730763    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:12.836789 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:21:12.836762    2572 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7909-bcedc5" container="test-curl-7909-bcedc5"
Sep 10 21:21:13.020294 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:13.020271    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"]
Sep 10 21:21:13.022097 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:21:13.022072    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079a8ef0_9f45_4e5b_bdbc_946e052e440d.slice/crio-0c4fcc0fb743cd15ea519eb4f323c3ae66216dbf38a3c151b79774d8c3d8ac11 WatchSource:0}: Error finding container 0c4fcc0fb743cd15ea519eb4f323c3ae66216dbf38a3c151b79774d8c3d8ac11: Status 404 returned error can't find the container with id 0c4fcc0fb743cd15ea519eb4f323c3ae66216dbf38a3c151b79774d8c3d8ac11
Sep 10 21:21:13.024300 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:13.024268    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:21:13.024365 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:13.024330    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:21:13.024365 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:13.024361    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:21:13.847553 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:13.847518    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8" event={"ID":"079a8ef0-9f45-4e5b-bdbc-946e052e440d","Type":"ContainerStarted","Data":"263cca1650f174bef81d85f238350b4af927a1344133f6c369d9ff15ba604e4c"}
Sep 10 21:21:13.847853 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:13.847589    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8" event={"ID":"079a8ef0-9f45-4e5b-bdbc-946e052e440d","Type":"ContainerStarted","Data":"0c4fcc0fb743cd15ea519eb4f323c3ae66216dbf38a3c151b79774d8c3d8ac11"}
Sep 10 21:21:13.847853 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:13.847641    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7909-bcedc5" podUID="ff8ec78f-e3ef-45ba-b0ab-f1ba0577c1aa" containerName="test-curl-7909-bcedc5" containerID="cri-o://5289b9d3d8b87e46382c69e04aa2c228d6153ff0b89a41d805e12f17e2249e77" gracePeriod=2
Sep 10 21:21:13.879047 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:13.878960    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8" podStartSLOduration=1.878939312 podStartE2EDuration="1.878939312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:21:13.875480019 +0000 UTC m=+1459.087944937" watchObservedRunningTime="2026-09-10 21:21:13.878939312 +0000 UTC m=+1459.091404231"
Sep 10 21:21:14.027588 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:14.027565    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7909-f9fd2c"
Sep 10 21:21:14.080764 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:14.080735    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz7cc\" (UniqueName: \"kubernetes.io/projected/efdf58c2-cabc-4824-b448-517244d818ac-kube-api-access-pz7cc\") pod \"efdf58c2-cabc-4824-b448-517244d818ac\" (UID: \"efdf58c2-cabc-4824-b448-517244d818ac\") "
Sep 10 21:21:14.082857 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:14.082834    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efdf58c2-cabc-4824-b448-517244d818ac-kube-api-access-pz7cc" (OuterVolumeSpecName: "kube-api-access-pz7cc") pod "efdf58c2-cabc-4824-b448-517244d818ac" (UID: "efdf58c2-cabc-4824-b448-517244d818ac"). InnerVolumeSpecName "kube-api-access-pz7cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:14.181742 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:14.181719    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz7cc\" (UniqueName: \"kubernetes.io/projected/efdf58c2-cabc-4824-b448-517244d818ac-kube-api-access-pz7cc\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:21:14.731022 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:14.730997    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:14.736854 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:14.736831    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:14.853591 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:14.853555    2572 generic.go:356] "Generic (PLEG): container finished" podID="efdf58c2-cabc-4824-b448-517244d818ac" containerID="08c1e3df0e24606883cf8605256900657061a34d04809837ec590abbca037a24" exitCode=137
Sep 10 21:21:14.853921 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:14.853596    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7909-f9fd2c"
Sep 10 21:21:14.853921 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:14.853654    2572 scope.go:117] "RemoveContainer" containerID="08c1e3df0e24606883cf8605256900657061a34d04809837ec590abbca037a24"
Sep 10 21:21:14.854059 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:14.854044    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:14.855156 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:14.855041    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:14.863231 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:14.863216    2572 scope.go:117] "RemoveContainer" containerID="08c1e3df0e24606883cf8605256900657061a34d04809837ec590abbca037a24"
Sep 10 21:21:14.863496 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:21:14.863478    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08c1e3df0e24606883cf8605256900657061a34d04809837ec590abbca037a24\": container with ID starting with 08c1e3df0e24606883cf8605256900657061a34d04809837ec590abbca037a24 not found: ID does not exist" containerID="08c1e3df0e24606883cf8605256900657061a34d04809837ec590abbca037a24"
Sep 10 21:21:14.863547 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:14.863504    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08c1e3df0e24606883cf8605256900657061a34d04809837ec590abbca037a24"} err="failed to get container status \"08c1e3df0e24606883cf8605256900657061a34d04809837ec590abbca037a24\": rpc error: code = NotFound desc = could not find container \"08c1e3df0e24606883cf8605256900657061a34d04809837ec590abbca037a24\": container with ID starting with 08c1e3df0e24606883cf8605256900657061a34d04809837ec590abbca037a24 not found: ID does not exist"
Sep 10 21:21:14.906562 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:21:14.906536    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7909-f9fd2c\" is forbidden: User \"system:node:ip-10-0-138-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-232.ec2.internal' and this object" podUID="efdf58c2-cabc-4824-b448-517244d818ac" pod="opendatahub/test-curl-7909-f9fd2c"
Sep 10 21:21:15.401420 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:15.401395    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"]
Sep 10 21:21:15.403701 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:21:15.403673    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7909-f9fd2c\" is forbidden: User \"system:node:ip-10-0-138-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-232.ec2.internal' and this object" podUID="efdf58c2-cabc-4824-b448-517244d818ac" pod="opendatahub/test-curl-7909-f9fd2c"
Sep 10 21:21:15.438369 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:15.438345    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efdf58c2-cabc-4824-b448-517244d818ac" path="/var/lib/kubelet/pods/efdf58c2-cabc-4824-b448-517244d818ac/volumes"
Sep 10 21:21:15.438477 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:21:15.438402    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7909-f9fd2c\" is forbidden: User \"system:node:ip-10-0-138-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-232.ec2.internal' and this object" podUID="efdf58c2-cabc-4824-b448-517244d818ac" pod="opendatahub/test-curl-7909-f9fd2c"
Sep 10 21:21:16.035836 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:16.035814    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7909-bcedc5"
Sep 10 21:21:16.097762 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:16.097735    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5w9nm\" (UniqueName: \"kubernetes.io/projected/ff8ec78f-e3ef-45ba-b0ab-f1ba0577c1aa-kube-api-access-5w9nm\") pod \"ff8ec78f-e3ef-45ba-b0ab-f1ba0577c1aa\" (UID: \"ff8ec78f-e3ef-45ba-b0ab-f1ba0577c1aa\") "
Sep 10 21:21:16.099751 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:16.099726    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8ec78f-e3ef-45ba-b0ab-f1ba0577c1aa-kube-api-access-5w9nm" (OuterVolumeSpecName: "kube-api-access-5w9nm") pod "ff8ec78f-e3ef-45ba-b0ab-f1ba0577c1aa" (UID: "ff8ec78f-e3ef-45ba-b0ab-f1ba0577c1aa"). InnerVolumeSpecName "kube-api-access-5w9nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:16.198612 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:16.198592    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w9nm\" (UniqueName: \"kubernetes.io/projected/ff8ec78f-e3ef-45ba-b0ab-f1ba0577c1aa-kube-api-access-5w9nm\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:21:16.585560 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:16.585526    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"]
Sep 10 21:21:16.866857 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:16.866774    2572 generic.go:356] "Generic (PLEG): container finished" podID="ff8ec78f-e3ef-45ba-b0ab-f1ba0577c1aa" containerID="5289b9d3d8b87e46382c69e04aa2c228d6153ff0b89a41d805e12f17e2249e77" exitCode=137
Sep 10 21:21:16.866857 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:16.866827    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7909-bcedc5"
Sep 10 21:21:16.867105 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:16.866878    2572 scope.go:117] "RemoveContainer" containerID="5289b9d3d8b87e46382c69e04aa2c228d6153ff0b89a41d805e12f17e2249e77"
Sep 10 21:21:16.876347 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:16.876331    2572 scope.go:117] "RemoveContainer" containerID="5289b9d3d8b87e46382c69e04aa2c228d6153ff0b89a41d805e12f17e2249e77"
Sep 10 21:21:16.876650 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:21:16.876625    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5289b9d3d8b87e46382c69e04aa2c228d6153ff0b89a41d805e12f17e2249e77\": container with ID starting with 5289b9d3d8b87e46382c69e04aa2c228d6153ff0b89a41d805e12f17e2249e77 not found: ID does not exist" containerID="5289b9d3d8b87e46382c69e04aa2c228d6153ff0b89a41d805e12f17e2249e77"
Sep 10 21:21:16.876732 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:16.876659    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5289b9d3d8b87e46382c69e04aa2c228d6153ff0b89a41d805e12f17e2249e77"} err="failed to get container status \"5289b9d3d8b87e46382c69e04aa2c228d6153ff0b89a41d805e12f17e2249e77\": rpc error: code = NotFound desc = could not find container \"5289b9d3d8b87e46382c69e04aa2c228d6153ff0b89a41d805e12f17e2249e77\": container with ID starting with 5289b9d3d8b87e46382c69e04aa2c228d6153ff0b89a41d805e12f17e2249e77 not found: ID does not exist"
Sep 10 21:21:16.877841 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:21:16.877817    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7909-bcedc5\" is forbidden: User \"system:node:ip-10-0-138-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-232.ec2.internal' and this object" podUID="ff8ec78f-e3ef-45ba-b0ab-f1ba0577c1aa" pod="opendatahub/test-curl-7909-bcedc5"
Sep 10 21:21:17.438032 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:17.438003    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff8ec78f-e3ef-45ba-b0ab-f1ba0577c1aa" path="/var/lib/kubelet/pods/ff8ec78f-e3ef-45ba-b0ab-f1ba0577c1aa/volumes"
Sep 10 21:21:18.256414 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.256387    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:21:18.340480 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.340450    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp"]
Sep 10 21:21:18.340805 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.340794    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff8ec78f-e3ef-45ba-b0ab-f1ba0577c1aa" containerName="test-curl-7909-bcedc5"
Sep 10 21:21:18.340855 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.340808    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff8ec78f-e3ef-45ba-b0ab-f1ba0577c1aa" containerName="test-curl-7909-bcedc5"
Sep 10 21:21:18.340890 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.340883    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff8ec78f-e3ef-45ba-b0ab-f1ba0577c1aa" containerName="test-curl-7909-bcedc5"
Sep 10 21:21:18.345262 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.345246    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp"
Sep 10 21:21:18.349595 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.349573    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d55386-dockercfg-9w69p\""
Sep 10 21:21:18.349595 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.349573    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d55386\""
Sep 10 21:21:18.361993 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.361966    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp"]
Sep 10 21:21:18.418490 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.418460    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sskw\" (UniqueName: \"kubernetes.io/projected/6fcecd64-a5c2-476c-b688-92c7595f3964-kube-api-access-2sskw\") pod \"payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp\" (UID: \"6fcecd64-a5c2-476c-b688-92c7595f3964\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp"
Sep 10 21:21:18.418663 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.418497    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fcecd64-a5c2-476c-b688-92c7595f3964-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp\" (UID: \"6fcecd64-a5c2-476c-b688-92c7595f3964\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp"
Sep 10 21:21:18.440609 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.440576    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm"]
Sep 10 21:21:18.447098 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.447068    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm"
Sep 10 21:21:18.457343 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.457315    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm"]
Sep 10 21:21:18.519094 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.519035    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fcecd64-a5c2-476c-b688-92c7595f3964-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp\" (UID: \"6fcecd64-a5c2-476c-b688-92c7595f3964\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp"
Sep 10 21:21:18.519204 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.519144    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sskw\" (UniqueName: \"kubernetes.io/projected/6fcecd64-a5c2-476c-b688-92c7595f3964-kube-api-access-2sskw\") pod \"payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp\" (UID: \"6fcecd64-a5c2-476c-b688-92c7595f3964\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp"
Sep 10 21:21:18.519204 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.519179    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4663747e-50fd-498c-9d5c-681d59b52efd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm\" (UID: \"4663747e-50fd-498c-9d5c-681d59b52efd\") " pod="openshift-ingress/payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm"
Sep 10 21:21:18.519314 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.519208    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcgbb\" (UniqueName: \"kubernetes.io/projected/4663747e-50fd-498c-9d5c-681d59b52efd-kube-api-access-tcgbb\") pod \"payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm\" (UID: \"4663747e-50fd-498c-9d5c-681d59b52efd\") " pod="openshift-ingress/payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm"
Sep 10 21:21:18.519719 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.519702    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fcecd64-a5c2-476c-b688-92c7595f3964-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp\" (UID: \"6fcecd64-a5c2-476c-b688-92c7595f3964\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp"
Sep 10 21:21:18.529804 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.529781    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sskw\" (UniqueName: \"kubernetes.io/projected/6fcecd64-a5c2-476c-b688-92c7595f3964-kube-api-access-2sskw\") pod \"payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp\" (UID: \"6fcecd64-a5c2-476c-b688-92c7595f3964\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp"
Sep 10 21:21:18.620477 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.620456    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4663747e-50fd-498c-9d5c-681d59b52efd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm\" (UID: \"4663747e-50fd-498c-9d5c-681d59b52efd\") " pod="openshift-ingress/payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm"
Sep 10 21:21:18.620551 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.620485    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcgbb\" (UniqueName: \"kubernetes.io/projected/4663747e-50fd-498c-9d5c-681d59b52efd-kube-api-access-tcgbb\") pod \"payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm\" (UID: \"4663747e-50fd-498c-9d5c-681d59b52efd\") " pod="openshift-ingress/payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm"
Sep 10 21:21:18.621101 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.621083    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4663747e-50fd-498c-9d5c-681d59b52efd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm\" (UID: \"4663747e-50fd-498c-9d5c-681d59b52efd\") " pod="openshift-ingress/payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm"
Sep 10 21:21:18.633126 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.633108    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcgbb\" (UniqueName: \"kubernetes.io/projected/4663747e-50fd-498c-9d5c-681d59b52efd-kube-api-access-tcgbb\") pod \"payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm\" (UID: \"4663747e-50fd-498c-9d5c-681d59b52efd\") " pod="openshift-ingress/payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm"
Sep 10 21:21:18.655498 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.655480    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp"
Sep 10 21:21:18.759869 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.759847    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm"
Sep 10 21:21:18.930952 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:18.930925    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp"]
Sep 10 21:21:18.933474 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:21:18.933447    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fcecd64_a5c2_476c_b688_92c7595f3964.slice/crio-835885fffe3c193d86cfb55580246c204ae7754e6e22effcb47b0fcc8212d0a8 WatchSource:0}: Error finding container 835885fffe3c193d86cfb55580246c204ae7754e6e22effcb47b0fcc8212d0a8: Status 404 returned error can't find the container with id 835885fffe3c193d86cfb55580246c204ae7754e6e22effcb47b0fcc8212d0a8
Sep 10 21:21:19.055228 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:19.055207    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm"]
Sep 10 21:21:19.056936 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:21:19.056913    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4663747e_50fd_498c_9d5c_681d59b52efd.slice/crio-b321c312f1430c4125e4c2f8124b99f2ec71f0f0630c64ffff0716b7b517522d WatchSource:0}: Error finding container b321c312f1430c4125e4c2f8124b99f2ec71f0f0630c64ffff0716b7b517522d: Status 404 returned error can't find the container with id b321c312f1430c4125e4c2f8124b99f2ec71f0f0630c64ffff0716b7b517522d
Sep 10 21:21:19.883811 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:19.883771    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp" event={"ID":"6fcecd64-a5c2-476c-b688-92c7595f3964","Type":"ContainerStarted","Data":"f5edf5e62dd5562a58c6ec6fbbffb6c1579648124026841b360ba8a230500a9b"}
Sep 10 21:21:19.884236 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:19.883835    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp" event={"ID":"6fcecd64-a5c2-476c-b688-92c7595f3964","Type":"ContainerStarted","Data":"835885fffe3c193d86cfb55580246c204ae7754e6e22effcb47b0fcc8212d0a8"}
Sep 10 21:21:19.884236 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:19.883869    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp"
Sep 10 21:21:19.885505 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:19.885481    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm" event={"ID":"4663747e-50fd-498c-9d5c-681d59b52efd","Type":"ContainerStarted","Data":"683b48e6bc1dfae3cc5ff3812a11d9b0725ad63c5b66f9f004bdb795ce369797"}
Sep 10 21:21:19.885650 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:19.885527    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm" event={"ID":"4663747e-50fd-498c-9d5c-681d59b52efd","Type":"ContainerStarted","Data":"b321c312f1430c4125e4c2f8124b99f2ec71f0f0630c64ffff0716b7b517522d"}
Sep 10 21:21:19.885650 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:19.885562    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm"
Sep 10 21:21:19.914660 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:19.914611    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp" podStartSLOduration=1.914595577 podStartE2EDuration="1.914595577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:21:19.911516795 +0000 UTC m=+1465.123981713" watchObservedRunningTime="2026-09-10 21:21:19.914595577 +0000 UTC m=+1465.127060497"
Sep 10 21:21:19.937721 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:19.937674    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm" podStartSLOduration=1.937656929 podStartE2EDuration="1.937656929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:21:19.937225265 +0000 UTC m=+1465.149690184" watchObservedRunningTime="2026-09-10 21:21:19.937656929 +0000 UTC m=+1465.150121853"
Sep 10 21:21:30.425750 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.425697    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c" podUID="59bd12c2-ea8f-4fbf-aed7-c2a03a754310" containerName="main" containerID="cri-o://4289521c03032086c43a2e6298df5a0242c34abf713ea6e51d4b99d7faaca1d9" gracePeriod=45
Sep 10 21:21:30.695876 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.695851    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:21:30.839656 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.839609    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knhn5\" (UniqueName: \"kubernetes.io/projected/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-kube-api-access-knhn5\") pod \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\" (UID: \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\") "
Sep 10 21:21:30.839656 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.839659    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-model-cache\") pod \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\" (UID: \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\") "
Sep 10 21:21:30.839924 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.839684    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-tls-certs\") pod \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\" (UID: \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\") "
Sep 10 21:21:30.839924 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.839726    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-dshm\") pod \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\" (UID: \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\") "
Sep 10 21:21:30.839924 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.839815    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-home\") pod \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\" (UID: \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\") "
Sep 10 21:21:30.839924 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.839887    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-tmp-dir\") pod \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\" (UID: \"59bd12c2-ea8f-4fbf-aed7-c2a03a754310\") "
Sep 10 21:21:30.840189 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.839935    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-model-cache" (OuterVolumeSpecName: "model-cache") pod "59bd12c2-ea8f-4fbf-aed7-c2a03a754310" (UID: "59bd12c2-ea8f-4fbf-aed7-c2a03a754310"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:30.840189 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.840149    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "59bd12c2-ea8f-4fbf-aed7-c2a03a754310" (UID: "59bd12c2-ea8f-4fbf-aed7-c2a03a754310"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:30.840189 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.840139    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-home" (OuterVolumeSpecName: "home") pod "59bd12c2-ea8f-4fbf-aed7-c2a03a754310" (UID: "59bd12c2-ea8f-4fbf-aed7-c2a03a754310"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:30.840350 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.840280    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-model-cache\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:21:30.840350 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.840300    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-home\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:21:30.840454 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.840368    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-tmp-dir\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:21:30.842009 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.841967    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-kube-api-access-knhn5" (OuterVolumeSpecName: "kube-api-access-knhn5") pod "59bd12c2-ea8f-4fbf-aed7-c2a03a754310" (UID: "59bd12c2-ea8f-4fbf-aed7-c2a03a754310"). InnerVolumeSpecName "kube-api-access-knhn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:30.842552 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.842531    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-dshm" (OuterVolumeSpecName: "dshm") pod "59bd12c2-ea8f-4fbf-aed7-c2a03a754310" (UID: "59bd12c2-ea8f-4fbf-aed7-c2a03a754310"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:30.842629 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.842604    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59bd12c2-ea8f-4fbf-aed7-c2a03a754310" (UID: "59bd12c2-ea8f-4fbf-aed7-c2a03a754310"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:21:30.892437 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.892418    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp"
Sep 10 21:21:30.892552 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.892479    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm"
Sep 10 21:21:30.942123 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.942025    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knhn5\" (UniqueName: \"kubernetes.io/projected/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-kube-api-access-knhn5\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:21:30.942123 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.942059    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-tls-certs\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:21:30.942123 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.942077    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59bd12c2-ea8f-4fbf-aed7-c2a03a754310-dshm\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:21:30.947852 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.947822    2572 generic.go:356] "Generic (PLEG): container finished" podID="59bd12c2-ea8f-4fbf-aed7-c2a03a754310" containerID="4289521c03032086c43a2e6298df5a0242c34abf713ea6e51d4b99d7faaca1d9" exitCode=0
Sep 10 21:21:30.948040 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.947895    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"
Sep 10 21:21:30.948040 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.947902    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c" event={"ID":"59bd12c2-ea8f-4fbf-aed7-c2a03a754310","Type":"ContainerDied","Data":"4289521c03032086c43a2e6298df5a0242c34abf713ea6e51d4b99d7faaca1d9"}
Sep 10 21:21:30.948040 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.947950    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c" event={"ID":"59bd12c2-ea8f-4fbf-aed7-c2a03a754310","Type":"ContainerDied","Data":"7527aed4cc28ce21535cc4b86b9d5bc90002fcf1cae4c19addc08f5681360c96"}
Sep 10 21:21:30.948040 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.947967    2572 scope.go:117] "RemoveContainer" containerID="4289521c03032086c43a2e6298df5a0242c34abf713ea6e51d4b99d7faaca1d9"
Sep 10 21:21:30.959871 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.959851    2572 scope.go:117] "RemoveContainer" containerID="4289521c03032086c43a2e6298df5a0242c34abf713ea6e51d4b99d7faaca1d9"
Sep 10 21:21:30.960208 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:21:30.960189    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4289521c03032086c43a2e6298df5a0242c34abf713ea6e51d4b99d7faaca1d9\": container with ID starting with 4289521c03032086c43a2e6298df5a0242c34abf713ea6e51d4b99d7faaca1d9 not found: ID does not exist" containerID="4289521c03032086c43a2e6298df5a0242c34abf713ea6e51d4b99d7faaca1d9"
Sep 10 21:21:30.960271 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.960216    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4289521c03032086c43a2e6298df5a0242c34abf713ea6e51d4b99d7faaca1d9"} err="failed to get container status \"4289521c03032086c43a2e6298df5a0242c34abf713ea6e51d4b99d7faaca1d9\": rpc error: code = NotFound desc = could not find container \"4289521c03032086c43a2e6298df5a0242c34abf713ea6e51d4b99d7faaca1d9\": container with ID starting with 4289521c03032086c43a2e6298df5a0242c34abf713ea6e51d4b99d7faaca1d9 not found: ID does not exist"
Sep 10 21:21:30.995130 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:30.995104    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"]
Sep 10 21:21:31.029006 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:31.027335    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7150/sub-test-model-af7150-kserve-587fc5b987-w2t5c"]
Sep 10 21:21:31.440726 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:31.440698    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59bd12c2-ea8f-4fbf-aed7-c2a03a754310" path="/var/lib/kubelet/pods/59bd12c2-ea8f-4fbf-aed7-c2a03a754310/volumes"
Sep 10 21:21:31.605049 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:31.605022    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr" podUID="1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4" containerName="main" containerID="cri-o://e757450dc2fe1e5d14651a39774504ec2d5c5c67c8d20fac356851d3ea5d07db" gracePeriod=45
Sep 10 21:21:31.892190 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:31.892164    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:21:31.955071 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:31.955038    2572 generic.go:356] "Generic (PLEG): container finished" podID="1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4" containerID="e757450dc2fe1e5d14651a39774504ec2d5c5c67c8d20fac356851d3ea5d07db" exitCode=0
Sep 10 21:21:31.955223 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:31.955084    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"
Sep 10 21:21:31.955223 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:31.955123    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr" event={"ID":"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4","Type":"ContainerDied","Data":"e757450dc2fe1e5d14651a39774504ec2d5c5c67c8d20fac356851d3ea5d07db"}
Sep 10 21:21:31.955223 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:31.955166    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr" event={"ID":"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4","Type":"ContainerDied","Data":"10b09c13b3d31b2c68b1fd3f8aabd73db87dac224bc75a3b2f6be1febc9ce855"}
Sep 10 21:21:31.955223 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:31.955184    2572 scope.go:117] "RemoveContainer" containerID="e757450dc2fe1e5d14651a39774504ec2d5c5c67c8d20fac356851d3ea5d07db"
Sep 10 21:21:31.963695 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:31.963678    2572 scope.go:117] "RemoveContainer" containerID="e757450dc2fe1e5d14651a39774504ec2d5c5c67c8d20fac356851d3ea5d07db"
Sep 10 21:21:31.964008 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:21:31.963969    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e757450dc2fe1e5d14651a39774504ec2d5c5c67c8d20fac356851d3ea5d07db\": container with ID starting with e757450dc2fe1e5d14651a39774504ec2d5c5c67c8d20fac356851d3ea5d07db not found: ID does not exist" containerID="e757450dc2fe1e5d14651a39774504ec2d5c5c67c8d20fac356851d3ea5d07db"
Sep 10 21:21:31.964140 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:31.964011    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e757450dc2fe1e5d14651a39774504ec2d5c5c67c8d20fac356851d3ea5d07db"} err="failed to get container status \"e757450dc2fe1e5d14651a39774504ec2d5c5c67c8d20fac356851d3ea5d07db\": rpc error: code = NotFound desc = could not find container \"e757450dc2fe1e5d14651a39774504ec2d5c5c67c8d20fac356851d3ea5d07db\": container with ID starting with e757450dc2fe1e5d14651a39774504ec2d5c5c67c8d20fac356851d3ea5d07db not found: ID does not exist"
Sep 10 21:21:32.052483 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:32.052460    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-dshm\") pod \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\" (UID: \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\") "
Sep 10 21:21:32.052621 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:32.052492    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-tmp-dir\") pod \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\" (UID: \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\") "
Sep 10 21:21:32.052621 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:32.052575    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dw6v4\" (UniqueName: \"kubernetes.io/projected/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-kube-api-access-dw6v4\") pod \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\" (UID: \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\") "
Sep 10 21:21:32.052707 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:32.052620    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-home\") pod \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\" (UID: \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\") "
Sep 10 21:21:32.052707 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:32.052661    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-model-cache\") pod \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\" (UID: \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\") "
Sep 10 21:21:32.052707 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:32.052692    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-tls-certs\") pod \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\" (UID: \"1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4\") "
Sep 10 21:21:32.052858 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:32.052761    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4" (UID: "1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:32.052914 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:32.052871    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4" (UID: "1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:32.052914 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:32.052893    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-home" (OuterVolumeSpecName: "home") pod "1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4" (UID: "1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:32.053080 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:32.053062    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-model-cache\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:21:32.053139 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:32.053084    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-tmp-dir\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:21:32.053139 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:32.053099    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-home\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:21:32.054753 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:32.054729    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-dshm" (OuterVolumeSpecName: "dshm") pod "1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4" (UID: "1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:32.054886 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:32.054867    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4" (UID: "1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:21:32.054956 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:32.054916    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-kube-api-access-dw6v4" (OuterVolumeSpecName: "kube-api-access-dw6v4") pod "1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4" (UID: "1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4"). InnerVolumeSpecName "kube-api-access-dw6v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:32.153766 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:32.153745    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-dshm\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:21:32.153766 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:32.153766    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw6v4\" (UniqueName: \"kubernetes.io/projected/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-kube-api-access-dw6v4\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:21:32.153914 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:32.153777    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4-tls-certs\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:21:32.288458 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:32.288433    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"]
Sep 10 21:21:32.295466 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:32.295445    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7810bd/sub-test-model-7810bd-kserve-66b8c9dc75-jq9sr"]
Sep 10 21:21:33.441809 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:33.441776    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4" path="/var/lib/kubelet/pods/1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4/volumes"
Sep 10 21:21:34.654365 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:34.654331    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:21:38.992045 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:38.992013    2572 generic.go:356] "Generic (PLEG): container finished" podID="079a8ef0-9f45-4e5b-bdbc-946e052e440d" containerID="263cca1650f174bef81d85f238350b4af927a1344133f6c369d9ff15ba604e4c" exitCode=0
Sep 10 21:21:38.992467 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:38.992069    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8" event={"ID":"079a8ef0-9f45-4e5b-bdbc-946e052e440d","Type":"ContainerDied","Data":"263cca1650f174bef81d85f238350b4af927a1344133f6c369d9ff15ba604e4c"}
Sep 10 21:21:38.992523 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:38.992500    2572 scope.go:117] "RemoveContainer" containerID="263cca1650f174bef81d85f238350b4af927a1344133f6c369d9ff15ba604e4c"
Sep 10 21:21:38.993687 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:38.993654    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:21:38.993756 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:38.993741    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:21:38.993793 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:38.993781    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:21:40.002250 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:40.002214    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8" event={"ID":"079a8ef0-9f45-4e5b-bdbc-946e052e440d","Type":"ContainerStarted","Data":"048f1792f25045a8e789ebe45492ee45ba0f71a42fee35161e6dd70544c504d4"}
Sep 10 21:21:40.732679 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:40.732631    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:40.733340 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:40.733305    2572 patch_prober.go:28] interesting pod/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 10 21:21:40.733463 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:40.733377    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8" podUID="079a8ef0-9f45-4e5b-bdbc-946e052e440d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 10 21:21:41.732145 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:41.732110    2572 patch_prober.go:28] interesting pod/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 10 21:21:41.732632 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:41.732197    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8" podUID="079a8ef0-9f45-4e5b-bdbc-946e052e440d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 10 21:21:42.732020 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:42.731965    2572 patch_prober.go:28] interesting pod/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 10 21:21:42.732299 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:42.732266    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8" podUID="079a8ef0-9f45-4e5b-bdbc-946e052e440d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 10 21:21:43.731685 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:43.731632    2572 patch_prober.go:28] interesting pod/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 10 21:21:43.731959 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:43.731926    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8" podUID="079a8ef0-9f45-4e5b-bdbc-946e052e440d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 10 21:21:44.731713 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:44.731667    2572 patch_prober.go:28] interesting pod/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 10 21:21:44.732220 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:44.731769    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8" podUID="079a8ef0-9f45-4e5b-bdbc-946e052e440d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 10 21:21:44.854513 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:44.854487    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:45.735813 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:45.735783    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:45.736684 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:45.736666    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:21:48.940780 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:48.940727    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"]
Sep 10 21:21:48.941368 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:48.941347    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59bd12c2-ea8f-4fbf-aed7-c2a03a754310" containerName="main"
Sep 10 21:21:48.941368 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:48.941370    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="59bd12c2-ea8f-4fbf-aed7-c2a03a754310" containerName="main"
Sep 10 21:21:48.941614 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:48.941420    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4" containerName="main"
Sep 10 21:21:48.941614 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:48.941429    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4" containerName="main"
Sep 10 21:21:48.941614 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:48.941534    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c4d78f2-0bab-4e03-8799-4ee9f4fff6e4" containerName="main"
Sep 10 21:21:48.941614 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:48.941544    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="59bd12c2-ea8f-4fbf-aed7-c2a03a754310" containerName="main"
Sep 10 21:21:48.945874 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:48.945842    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:21:48.949341 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:48.949314    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-af7150\"/\"kube-root-ca.crt\""
Sep 10 21:21:48.950195 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:48.950175    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-af7150\"/\"openshift-service-ca.crt\""
Sep 10 21:21:48.950195 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:48.950187    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-af7150\"/\"default-dockercfg-cwcjx\""
Sep 10 21:21:48.961959 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:48.961934    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-af7150\"/\"rate-test-model-af7150-kserve-self-signed-certs\""
Sep 10 21:21:48.989674 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:48.989645    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"]
Sep 10 21:21:49.006754 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:49.006732    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28nbp\" (UniqueName: \"kubernetes.io/projected/dea3f743-8c01-49e5-890e-2d65eba3efc2-kube-api-access-28nbp\") pod \"rate-test-model-af7150-kserve-b966b47ff-mwj2r\" (UID: \"dea3f743-8c01-49e5-890e-2d65eba3efc2\") " pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:21:49.006844 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:49.006772    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea3f743-8c01-49e5-890e-2d65eba3efc2-tls-certs\") pod \"rate-test-model-af7150-kserve-b966b47ff-mwj2r\" (UID: \"dea3f743-8c01-49e5-890e-2d65eba3efc2\") " pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:21:49.006844 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:49.006791    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea3f743-8c01-49e5-890e-2d65eba3efc2-tmp-dir\") pod \"rate-test-model-af7150-kserve-b966b47ff-mwj2r\" (UID: \"dea3f743-8c01-49e5-890e-2d65eba3efc2\") " pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:21:49.006919 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:49.006843    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea3f743-8c01-49e5-890e-2d65eba3efc2-dshm\") pod \"rate-test-model-af7150-kserve-b966b47ff-mwj2r\" (UID: \"dea3f743-8c01-49e5-890e-2d65eba3efc2\") " pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:21:49.006919 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:49.006880    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea3f743-8c01-49e5-890e-2d65eba3efc2-home\") pod \"rate-test-model-af7150-kserve-b966b47ff-mwj2r\" (UID: \"dea3f743-8c01-49e5-890e-2d65eba3efc2\") " pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:21:49.006919 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:49.006915    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea3f743-8c01-49e5-890e-2d65eba3efc2-model-cache\") pod \"rate-test-model-af7150-kserve-b966b47ff-mwj2r\" (UID: \"dea3f743-8c01-49e5-890e-2d65eba3efc2\") " pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:21:49.107811 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:49.107794    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28nbp\" (UniqueName: \"kubernetes.io/projected/dea3f743-8c01-49e5-890e-2d65eba3efc2-kube-api-access-28nbp\") pod \"rate-test-model-af7150-kserve-b966b47ff-mwj2r\" (UID: \"dea3f743-8c01-49e5-890e-2d65eba3efc2\") " pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:21:49.107927 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:49.107823    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea3f743-8c01-49e5-890e-2d65eba3efc2-tls-certs\") pod \"rate-test-model-af7150-kserve-b966b47ff-mwj2r\" (UID: \"dea3f743-8c01-49e5-890e-2d65eba3efc2\") " pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:21:49.107927 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:49.107846    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea3f743-8c01-49e5-890e-2d65eba3efc2-tmp-dir\") pod \"rate-test-model-af7150-kserve-b966b47ff-mwj2r\" (UID: \"dea3f743-8c01-49e5-890e-2d65eba3efc2\") " pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:21:49.107927 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:49.107869    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea3f743-8c01-49e5-890e-2d65eba3efc2-dshm\") pod \"rate-test-model-af7150-kserve-b966b47ff-mwj2r\" (UID: \"dea3f743-8c01-49e5-890e-2d65eba3efc2\") " pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:21:49.107927 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:49.107904    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea3f743-8c01-49e5-890e-2d65eba3efc2-home\") pod \"rate-test-model-af7150-kserve-b966b47ff-mwj2r\" (UID: \"dea3f743-8c01-49e5-890e-2d65eba3efc2\") " pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:21:49.107927 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:49.107927    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea3f743-8c01-49e5-890e-2d65eba3efc2-model-cache\") pod \"rate-test-model-af7150-kserve-b966b47ff-mwj2r\" (UID: \"dea3f743-8c01-49e5-890e-2d65eba3efc2\") " pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:21:49.108293 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:49.108271    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea3f743-8c01-49e5-890e-2d65eba3efc2-tmp-dir\") pod \"rate-test-model-af7150-kserve-b966b47ff-mwj2r\" (UID: \"dea3f743-8c01-49e5-890e-2d65eba3efc2\") " pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:21:49.108349 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:49.108303    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea3f743-8c01-49e5-890e-2d65eba3efc2-model-cache\") pod \"rate-test-model-af7150-kserve-b966b47ff-mwj2r\" (UID: \"dea3f743-8c01-49e5-890e-2d65eba3efc2\") " pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:21:49.108349 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:49.108337    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea3f743-8c01-49e5-890e-2d65eba3efc2-home\") pod \"rate-test-model-af7150-kserve-b966b47ff-mwj2r\" (UID: \"dea3f743-8c01-49e5-890e-2d65eba3efc2\") " pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:21:49.110321 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:49.110304    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea3f743-8c01-49e5-890e-2d65eba3efc2-dshm\") pod \"rate-test-model-af7150-kserve-b966b47ff-mwj2r\" (UID: \"dea3f743-8c01-49e5-890e-2d65eba3efc2\") " pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:21:49.110396 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:49.110380    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea3f743-8c01-49e5-890e-2d65eba3efc2-tls-certs\") pod \"rate-test-model-af7150-kserve-b966b47ff-mwj2r\" (UID: \"dea3f743-8c01-49e5-890e-2d65eba3efc2\") " pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:21:49.122465 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:49.122446    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28nbp\" (UniqueName: \"kubernetes.io/projected/dea3f743-8c01-49e5-890e-2d65eba3efc2-kube-api-access-28nbp\") pod \"rate-test-model-af7150-kserve-b966b47ff-mwj2r\" (UID: \"dea3f743-8c01-49e5-890e-2d65eba3efc2\") " pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:21:49.255428 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:49.255370    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:21:49.531053 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:49.531029    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"]
Sep 10 21:21:49.533268 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:21:49.533246    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddea3f743_8c01_49e5_890e_2d65eba3efc2.slice/crio-1816df740198e29155e0ad3ad2f84eca6ab44907d6f4b5d820fb13a2280e3a19 WatchSource:0}: Error finding container 1816df740198e29155e0ad3ad2f84eca6ab44907d6f4b5d820fb13a2280e3a19: Status 404 returned error can't find the container with id 1816df740198e29155e0ad3ad2f84eca6ab44907d6f4b5d820fb13a2280e3a19
Sep 10 21:21:50.064843 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:50.064802    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r" event={"ID":"dea3f743-8c01-49e5-890e-2d65eba3efc2","Type":"ContainerStarted","Data":"52537db3ab16e77a1964fe64f7842c37ae7d729b3d14fbebb6ccc916d3a85d40"}
Sep 10 21:21:50.065284 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:50.064857    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r" event={"ID":"dea3f743-8c01-49e5-890e-2d65eba3efc2","Type":"ContainerStarted","Data":"1816df740198e29155e0ad3ad2f84eca6ab44907d6f4b5d820fb13a2280e3a19"}
Sep 10 21:21:50.083495 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:50.083446    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r" podStartSLOduration=2.083430983 podStartE2EDuration="2.083430983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:21:50.082296171 +0000 UTC m=+1495.294761109" watchObservedRunningTime="2026-09-10 21:21:50.083430983 +0000 UTC m=+1495.295895975"
Sep 10 21:21:53.540022 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:53.539970    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:21:55.653449 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:55.653421    2572 scope.go:117] "RemoveContainer" containerID="89dfd12ba19c600c2daa71a636507629d79302f5493668c332987b8824f30c5c"
Sep 10 21:21:59.256081 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:59.256039    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:21:59.256081 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:59.256082    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:21:59.269110 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:21:59.269076    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:22:00.134062 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:00.134035    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:22:13.025201 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:13.025167    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:22:23.887338 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:23.887246    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm"]
Sep 10 21:22:23.887829 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:23.887521    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm" podUID="4663747e-50fd-498c-9d5c-681d59b52efd" containerName="payload-processing" containerID="cri-o://683b48e6bc1dfae3cc5ff3812a11d9b0725ad63c5b66f9f004bdb795ce369797" gracePeriod=30
Sep 10 21:22:23.898560 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:23.898531    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp"]
Sep 10 21:22:23.898803 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:23.898779    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp" podUID="6fcecd64-a5c2-476c-b688-92c7595f3964" containerName="payload-pre-processing" containerID="cri-o://f5edf5e62dd5562a58c6ec6fbbffb6c1579648124026841b360ba8a230500a9b" gracePeriod=30
Sep 10 21:22:24.144261 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.144206    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm"
Sep 10 21:22:24.173413 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.173383    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp"
Sep 10 21:22:24.252841 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.252807    2572 generic.go:356] "Generic (PLEG): container finished" podID="4663747e-50fd-498c-9d5c-681d59b52efd" containerID="683b48e6bc1dfae3cc5ff3812a11d9b0725ad63c5b66f9f004bdb795ce369797" exitCode=0
Sep 10 21:22:24.253035 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.252852    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm"
Sep 10 21:22:24.253035 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.252887    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm" event={"ID":"4663747e-50fd-498c-9d5c-681d59b52efd","Type":"ContainerDied","Data":"683b48e6bc1dfae3cc5ff3812a11d9b0725ad63c5b66f9f004bdb795ce369797"}
Sep 10 21:22:24.253035 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.252941    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm" event={"ID":"4663747e-50fd-498c-9d5c-681d59b52efd","Type":"ContainerDied","Data":"b321c312f1430c4125e4c2f8124b99f2ec71f0f0630c64ffff0716b7b517522d"}
Sep 10 21:22:24.253035 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.252974    2572 scope.go:117] "RemoveContainer" containerID="683b48e6bc1dfae3cc5ff3812a11d9b0725ad63c5b66f9f004bdb795ce369797"
Sep 10 21:22:24.254382 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.254356    2572 generic.go:356] "Generic (PLEG): container finished" podID="6fcecd64-a5c2-476c-b688-92c7595f3964" containerID="f5edf5e62dd5562a58c6ec6fbbffb6c1579648124026841b360ba8a230500a9b" exitCode=0
Sep 10 21:22:24.254478 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.254393    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp"
Sep 10 21:22:24.254478 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.254449    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp" event={"ID":"6fcecd64-a5c2-476c-b688-92c7595f3964","Type":"ContainerDied","Data":"f5edf5e62dd5562a58c6ec6fbbffb6c1579648124026841b360ba8a230500a9b"}
Sep 10 21:22:24.254561 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.254497    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp" event={"ID":"6fcecd64-a5c2-476c-b688-92c7595f3964","Type":"ContainerDied","Data":"835885fffe3c193d86cfb55580246c204ae7754e6e22effcb47b0fcc8212d0a8"}
Sep 10 21:22:24.261837 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.261811    2572 scope.go:117] "RemoveContainer" containerID="683b48e6bc1dfae3cc5ff3812a11d9b0725ad63c5b66f9f004bdb795ce369797"
Sep 10 21:22:24.262196 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:22:24.262175    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"683b48e6bc1dfae3cc5ff3812a11d9b0725ad63c5b66f9f004bdb795ce369797\": container with ID starting with 683b48e6bc1dfae3cc5ff3812a11d9b0725ad63c5b66f9f004bdb795ce369797 not found: ID does not exist" containerID="683b48e6bc1dfae3cc5ff3812a11d9b0725ad63c5b66f9f004bdb795ce369797"
Sep 10 21:22:24.262291 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.262206    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"683b48e6bc1dfae3cc5ff3812a11d9b0725ad63c5b66f9f004bdb795ce369797"} err="failed to get container status \"683b48e6bc1dfae3cc5ff3812a11d9b0725ad63c5b66f9f004bdb795ce369797\": rpc error: code = NotFound desc = could not find container \"683b48e6bc1dfae3cc5ff3812a11d9b0725ad63c5b66f9f004bdb795ce369797\": container with ID starting with 683b48e6bc1dfae3cc5ff3812a11d9b0725ad63c5b66f9f004bdb795ce369797 not found: ID does not exist"
Sep 10 21:22:24.262291 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.262228    2572 scope.go:117] "RemoveContainer" containerID="f5edf5e62dd5562a58c6ec6fbbffb6c1579648124026841b360ba8a230500a9b"
Sep 10 21:22:24.270433 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.270415    2572 scope.go:117] "RemoveContainer" containerID="f5edf5e62dd5562a58c6ec6fbbffb6c1579648124026841b360ba8a230500a9b"
Sep 10 21:22:24.270728 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:22:24.270710    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5edf5e62dd5562a58c6ec6fbbffb6c1579648124026841b360ba8a230500a9b\": container with ID starting with f5edf5e62dd5562a58c6ec6fbbffb6c1579648124026841b360ba8a230500a9b not found: ID does not exist" containerID="f5edf5e62dd5562a58c6ec6fbbffb6c1579648124026841b360ba8a230500a9b"
Sep 10 21:22:24.270798 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.270737    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5edf5e62dd5562a58c6ec6fbbffb6c1579648124026841b360ba8a230500a9b"} err="failed to get container status \"f5edf5e62dd5562a58c6ec6fbbffb6c1579648124026841b360ba8a230500a9b\": rpc error: code = NotFound desc = could not find container \"f5edf5e62dd5562a58c6ec6fbbffb6c1579648124026841b360ba8a230500a9b\": container with ID starting with f5edf5e62dd5562a58c6ec6fbbffb6c1579648124026841b360ba8a230500a9b not found: ID does not exist"
Sep 10 21:22:24.299005 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.298958    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4663747e-50fd-498c-9d5c-681d59b52efd-plugins-config-volume\") pod \"4663747e-50fd-498c-9d5c-681d59b52efd\" (UID: \"4663747e-50fd-498c-9d5c-681d59b52efd\") "
Sep 10 21:22:24.299154 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.299043    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fcecd64-a5c2-476c-b688-92c7595f3964-plugins-config-volume\") pod \"6fcecd64-a5c2-476c-b688-92c7595f3964\" (UID: \"6fcecd64-a5c2-476c-b688-92c7595f3964\") "
Sep 10 21:22:24.299154 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.299099    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sskw\" (UniqueName: \"kubernetes.io/projected/6fcecd64-a5c2-476c-b688-92c7595f3964-kube-api-access-2sskw\") pod \"6fcecd64-a5c2-476c-b688-92c7595f3964\" (UID: \"6fcecd64-a5c2-476c-b688-92c7595f3964\") "
Sep 10 21:22:24.299154 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.299134    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcgbb\" (UniqueName: \"kubernetes.io/projected/4663747e-50fd-498c-9d5c-681d59b52efd-kube-api-access-tcgbb\") pod \"4663747e-50fd-498c-9d5c-681d59b52efd\" (UID: \"4663747e-50fd-498c-9d5c-681d59b52efd\") "
Sep 10 21:22:24.299411 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.299385    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4663747e-50fd-498c-9d5c-681d59b52efd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4663747e-50fd-498c-9d5c-681d59b52efd" (UID: "4663747e-50fd-498c-9d5c-681d59b52efd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:22:24.299486 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.299444    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fcecd64-a5c2-476c-b688-92c7595f3964-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6fcecd64-a5c2-476c-b688-92c7595f3964" (UID: "6fcecd64-a5c2-476c-b688-92c7595f3964"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:22:24.299532 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.299483    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4663747e-50fd-498c-9d5c-681d59b52efd-plugins-config-volume\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:22:24.301462 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.301428    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fcecd64-a5c2-476c-b688-92c7595f3964-kube-api-access-2sskw" (OuterVolumeSpecName: "kube-api-access-2sskw") pod "6fcecd64-a5c2-476c-b688-92c7595f3964" (UID: "6fcecd64-a5c2-476c-b688-92c7595f3964"). InnerVolumeSpecName "kube-api-access-2sskw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:22:24.301568 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.301483    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4663747e-50fd-498c-9d5c-681d59b52efd-kube-api-access-tcgbb" (OuterVolumeSpecName: "kube-api-access-tcgbb") pod "4663747e-50fd-498c-9d5c-681d59b52efd" (UID: "4663747e-50fd-498c-9d5c-681d59b52efd"). InnerVolumeSpecName "kube-api-access-tcgbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:22:24.400646 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.400556    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fcecd64-a5c2-476c-b688-92c7595f3964-plugins-config-volume\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:22:24.400646 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.400588    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sskw\" (UniqueName: \"kubernetes.io/projected/6fcecd64-a5c2-476c-b688-92c7595f3964-kube-api-access-2sskw\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:22:24.400646 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.400604    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcgbb\" (UniqueName: \"kubernetes.io/projected/4663747e-50fd-498c-9d5c-681d59b52efd-kube-api-access-tcgbb\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:22:24.592243 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.592215    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm"]
Sep 10 21:22:24.610767 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.610742    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d55386-d8d9dbff9-dq5hm"]
Sep 10 21:22:24.616407 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.616387    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp"]
Sep 10 21:22:24.626328 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:24.626308    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d55386-84df8dfddb-b5blp"]
Sep 10 21:22:25.439133 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:25.439102    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4663747e-50fd-498c-9d5c-681d59b52efd" path="/var/lib/kubelet/pods/4663747e-50fd-498c-9d5c-681d59b52efd/volumes"
Sep 10 21:22:25.439624 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:25.439599    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fcecd64-a5c2-476c-b688-92c7595f3964" path="/var/lib/kubelet/pods/6fcecd64-a5c2-476c-b688-92c7595f3964/volumes"
Sep 10 21:22:27.330850 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:27.330816    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:22:30.937967 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:30.937930    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"]
Sep 10 21:22:30.938390 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:30.938208    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8" podUID="079a8ef0-9f45-4e5b-bdbc-946e052e440d" containerName="istio-proxy" containerID="cri-o://048f1792f25045a8e789ebe45492ee45ba0f71a42fee35161e6dd70544c504d4" gracePeriod=30
Sep 10 21:22:36.184475 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.184442    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:22:36.288237 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.288211    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istio-envoy\") pod \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") "
Sep 10 21:22:36.288360 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.288271    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-workload-certs\") pod \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") "
Sep 10 21:22:36.288360 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.288298    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istiod-ca-cert\") pod \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") "
Sep 10 21:22:36.288463 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.288357    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5pn2\" (UniqueName: \"kubernetes.io/projected/079a8ef0-9f45-4e5b-bdbc-946e052e440d-kube-api-access-n5pn2\") pod \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") "
Sep 10 21:22:36.288463 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.288389    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istio-data\") pod \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") "
Sep 10 21:22:36.288463 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.288415    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istio-token\") pod \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") "
Sep 10 21:22:36.288630 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.288467    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istio-podinfo\") pod \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") "
Sep 10 21:22:36.288630 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.288496    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-credential-socket\") pod \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") "
Sep 10 21:22:36.288630 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.288528    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-workload-socket\") pod \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\" (UID: \"079a8ef0-9f45-4e5b-bdbc-946e052e440d\") "
Sep 10 21:22:36.288630 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.288550    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "079a8ef0-9f45-4e5b-bdbc-946e052e440d" (UID: "079a8ef0-9f45-4e5b-bdbc-946e052e440d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:22:36.288875 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.288674    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istio-data" (OuterVolumeSpecName: "istio-data") pod "079a8ef0-9f45-4e5b-bdbc-946e052e440d" (UID: "079a8ef0-9f45-4e5b-bdbc-946e052e440d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:22:36.288875 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.288749    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "079a8ef0-9f45-4e5b-bdbc-946e052e440d" (UID: "079a8ef0-9f45-4e5b-bdbc-946e052e440d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:22:36.289021 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.288869    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "079a8ef0-9f45-4e5b-bdbc-946e052e440d" (UID: "079a8ef0-9f45-4e5b-bdbc-946e052e440d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:22:36.289021 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.288926    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-workload-certs\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:22:36.289021 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.288952    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istiod-ca-cert\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:22:36.289021 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.288966    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istio-data\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:22:36.289193 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.289050    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "079a8ef0-9f45-4e5b-bdbc-946e052e440d" (UID: "079a8ef0-9f45-4e5b-bdbc-946e052e440d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:22:36.290331 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.290300    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "079a8ef0-9f45-4e5b-bdbc-946e052e440d" (UID: "079a8ef0-9f45-4e5b-bdbc-946e052e440d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:22:36.290826 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.290808    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/079a8ef0-9f45-4e5b-bdbc-946e052e440d-kube-api-access-n5pn2" (OuterVolumeSpecName: "kube-api-access-n5pn2") pod "079a8ef0-9f45-4e5b-bdbc-946e052e440d" (UID: "079a8ef0-9f45-4e5b-bdbc-946e052e440d"). InnerVolumeSpecName "kube-api-access-n5pn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:22:36.291294 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.291277    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "079a8ef0-9f45-4e5b-bdbc-946e052e440d" (UID: "079a8ef0-9f45-4e5b-bdbc-946e052e440d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:22:36.291294 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.291282    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istio-token" (OuterVolumeSpecName: "istio-token") pod "079a8ef0-9f45-4e5b-bdbc-946e052e440d" (UID: "079a8ef0-9f45-4e5b-bdbc-946e052e440d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:22:36.318871 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.318846    2572 generic.go:356] "Generic (PLEG): container finished" podID="079a8ef0-9f45-4e5b-bdbc-946e052e440d" containerID="048f1792f25045a8e789ebe45492ee45ba0f71a42fee35161e6dd70544c504d4" exitCode=0
Sep 10 21:22:36.318959 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.318892    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"
Sep 10 21:22:36.318959 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.318920    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8" event={"ID":"079a8ef0-9f45-4e5b-bdbc-946e052e440d","Type":"ContainerDied","Data":"048f1792f25045a8e789ebe45492ee45ba0f71a42fee35161e6dd70544c504d4"}
Sep 10 21:22:36.319053 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.318961    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8" event={"ID":"079a8ef0-9f45-4e5b-bdbc-946e052e440d","Type":"ContainerDied","Data":"0c4fcc0fb743cd15ea519eb4f323c3ae66216dbf38a3c151b79774d8c3d8ac11"}
Sep 10 21:22:36.319053 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.318995    2572 scope.go:117] "RemoveContainer" containerID="048f1792f25045a8e789ebe45492ee45ba0f71a42fee35161e6dd70544c504d4"
Sep 10 21:22:36.328431 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.328411    2572 scope.go:117] "RemoveContainer" containerID="263cca1650f174bef81d85f238350b4af927a1344133f6c369d9ff15ba604e4c"
Sep 10 21:22:36.336241 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.336223    2572 scope.go:117] "RemoveContainer" containerID="048f1792f25045a8e789ebe45492ee45ba0f71a42fee35161e6dd70544c504d4"
Sep 10 21:22:36.336513 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:22:36.336496    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"048f1792f25045a8e789ebe45492ee45ba0f71a42fee35161e6dd70544c504d4\": container with ID starting with 048f1792f25045a8e789ebe45492ee45ba0f71a42fee35161e6dd70544c504d4 not found: ID does not exist" containerID="048f1792f25045a8e789ebe45492ee45ba0f71a42fee35161e6dd70544c504d4"
Sep 10 21:22:36.336555 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.336521    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"048f1792f25045a8e789ebe45492ee45ba0f71a42fee35161e6dd70544c504d4"} err="failed to get container status \"048f1792f25045a8e789ebe45492ee45ba0f71a42fee35161e6dd70544c504d4\": rpc error: code = NotFound desc = could not find container \"048f1792f25045a8e789ebe45492ee45ba0f71a42fee35161e6dd70544c504d4\": container with ID starting with 048f1792f25045a8e789ebe45492ee45ba0f71a42fee35161e6dd70544c504d4 not found: ID does not exist"
Sep 10 21:22:36.336555 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.336537    2572 scope.go:117] "RemoveContainer" containerID="263cca1650f174bef81d85f238350b4af927a1344133f6c369d9ff15ba604e4c"
Sep 10 21:22:36.336796 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:22:36.336776    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"263cca1650f174bef81d85f238350b4af927a1344133f6c369d9ff15ba604e4c\": container with ID starting with 263cca1650f174bef81d85f238350b4af927a1344133f6c369d9ff15ba604e4c not found: ID does not exist" containerID="263cca1650f174bef81d85f238350b4af927a1344133f6c369d9ff15ba604e4c"
Sep 10 21:22:36.336850 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.336802    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"263cca1650f174bef81d85f238350b4af927a1344133f6c369d9ff15ba604e4c"} err="failed to get container status \"263cca1650f174bef81d85f238350b4af927a1344133f6c369d9ff15ba604e4c\": rpc error: code = NotFound desc = could not find container \"263cca1650f174bef81d85f238350b4af927a1344133f6c369d9ff15ba604e4c\": container with ID starting with 263cca1650f174bef81d85f238350b4af927a1344133f6c369d9ff15ba604e4c not found: ID does not exist"
Sep 10 21:22:36.342887 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.342850    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"]
Sep 10 21:22:36.349959 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.349938    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d55386-openshift-default-db4f7567b-l2jz8"]
Sep 10 21:22:36.389517 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.389490    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5pn2\" (UniqueName: \"kubernetes.io/projected/079a8ef0-9f45-4e5b-bdbc-946e052e440d-kube-api-access-n5pn2\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:22:36.389517 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.389515    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istio-token\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:22:36.389653 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.389530    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istio-podinfo\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:22:36.389653 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.389542    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-credential-socket\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:22:36.389653 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.389554    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-workload-socket\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:22:36.389653 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:36.389567    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079a8ef0-9f45-4e5b-bdbc-946e052e440d-istio-envoy\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:22:37.438540 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:37.438505    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="079a8ef0-9f45-4e5b-bdbc-946e052e440d" path="/var/lib/kubelet/pods/079a8ef0-9f45-4e5b-bdbc-946e052e440d/volumes"
Sep 10 21:22:41.944148 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:41.944106    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:22:42.227128 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:42.227048    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8"]
Sep 10 21:22:42.227309 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:42.227260    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8" podUID="6d19acfd-20c6-4d82-8b56-e84daef72d89" containerName="payload-processing" containerID="cri-o://d0743f5e23fb6844146d7bbaeb1b1880321e6ea4fca0a66b5ad0449ac6158fd7" gracePeriod=30
Sep 10 21:22:42.351897 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:42.351866    2572 generic.go:356] "Generic (PLEG): container finished" podID="6d19acfd-20c6-4d82-8b56-e84daef72d89" containerID="d0743f5e23fb6844146d7bbaeb1b1880321e6ea4fca0a66b5ad0449ac6158fd7" exitCode=0
Sep 10 21:22:42.352068 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:42.351934    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8" event={"ID":"6d19acfd-20c6-4d82-8b56-e84daef72d89","Type":"ContainerDied","Data":"d0743f5e23fb6844146d7bbaeb1b1880321e6ea4fca0a66b5ad0449ac6158fd7"}
Sep 10 21:22:42.488343 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:42.488290    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8"
Sep 10 21:22:42.537068 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:42.537042    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2rfw\" (UniqueName: \"kubernetes.io/projected/6d19acfd-20c6-4d82-8b56-e84daef72d89-kube-api-access-p2rfw\") pod \"6d19acfd-20c6-4d82-8b56-e84daef72d89\" (UID: \"6d19acfd-20c6-4d82-8b56-e84daef72d89\") "
Sep 10 21:22:42.537187 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:42.537094    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d19acfd-20c6-4d82-8b56-e84daef72d89-plugins-config-volume\") pod \"6d19acfd-20c6-4d82-8b56-e84daef72d89\" (UID: \"6d19acfd-20c6-4d82-8b56-e84daef72d89\") "
Sep 10 21:22:42.537482 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:42.537461    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d19acfd-20c6-4d82-8b56-e84daef72d89-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6d19acfd-20c6-4d82-8b56-e84daef72d89" (UID: "6d19acfd-20c6-4d82-8b56-e84daef72d89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:22:42.539294 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:42.539270    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d19acfd-20c6-4d82-8b56-e84daef72d89-kube-api-access-p2rfw" (OuterVolumeSpecName: "kube-api-access-p2rfw") pod "6d19acfd-20c6-4d82-8b56-e84daef72d89" (UID: "6d19acfd-20c6-4d82-8b56-e84daef72d89"). InnerVolumeSpecName "kube-api-access-p2rfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:22:42.638282 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:42.638261    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2rfw\" (UniqueName: \"kubernetes.io/projected/6d19acfd-20c6-4d82-8b56-e84daef72d89-kube-api-access-p2rfw\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:22:42.638282 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:42.638281    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d19acfd-20c6-4d82-8b56-e84daef72d89-plugins-config-volume\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:22:43.358734 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:43.358705    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8" event={"ID":"6d19acfd-20c6-4d82-8b56-e84daef72d89","Type":"ContainerDied","Data":"6cbe481c4f86ca8bee9d0a8979b580a804a4f70a350e24cfcf22ad3e155bfa6f"}
Sep 10 21:22:43.359181 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:43.358738    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8"
Sep 10 21:22:43.359181 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:43.358772    2572 scope.go:117] "RemoveContainer" containerID="d0743f5e23fb6844146d7bbaeb1b1880321e6ea4fca0a66b5ad0449ac6158fd7"
Sep 10 21:22:43.380209 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:43.380184    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8"]
Sep 10 21:22:43.384095 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:43.384076    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9cce25-746597d4b7-66tq8"]
Sep 10 21:22:43.438865 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:43.438844    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d19acfd-20c6-4d82-8b56-e84daef72d89" path="/var/lib/kubelet/pods/6d19acfd-20c6-4d82-8b56-e84daef72d89/volumes"
Sep 10 21:22:50.221069 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:50.221038    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-95d905-4qgjq"]
Sep 10 21:22:50.221451 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:50.221412    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d19acfd-20c6-4d82-8b56-e84daef72d89" containerName="payload-processing"
Sep 10 21:22:50.221451 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:50.221424    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d19acfd-20c6-4d82-8b56-e84daef72d89" containerName="payload-processing"
Sep 10 21:22:50.221451 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:50.221431    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="079a8ef0-9f45-4e5b-bdbc-946e052e440d" containerName="istio-proxy"
Sep 10 21:22:50.221451 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:50.221437    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="079a8ef0-9f45-4e5b-bdbc-946e052e440d" containerName="istio-proxy"
Sep 10 21:22:50.221451 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:50.221443    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fcecd64-a5c2-476c-b688-92c7595f3964" containerName="payload-pre-processing"
Sep 10 21:22:50.221451 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:50.221449    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fcecd64-a5c2-476c-b688-92c7595f3964" containerName="payload-pre-processing"
Sep 10 21:22:50.221640 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:50.221458    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4663747e-50fd-498c-9d5c-681d59b52efd" containerName="payload-processing"
Sep 10 21:22:50.221640 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:50.221464    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4663747e-50fd-498c-9d5c-681d59b52efd" containerName="payload-processing"
Sep 10 21:22:50.221640 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:50.221526    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fcecd64-a5c2-476c-b688-92c7595f3964" containerName="payload-pre-processing"
Sep 10 21:22:50.221640 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:50.221534    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d19acfd-20c6-4d82-8b56-e84daef72d89" containerName="payload-processing"
Sep 10 21:22:50.221640 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:50.221540    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="079a8ef0-9f45-4e5b-bdbc-946e052e440d" containerName="istio-proxy"
Sep 10 21:22:50.221640 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:50.221547    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4663747e-50fd-498c-9d5c-681d59b52efd" containerName="payload-processing"
Sep 10 21:22:50.221640 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:50.221551    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="079a8ef0-9f45-4e5b-bdbc-946e052e440d" containerName="istio-proxy"
Sep 10 21:22:50.225967 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:50.225948    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-95d905-4qgjq"
Sep 10 21:22:50.229214 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:50.229196    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:22:50.229302 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:50.229236    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5c7x\""
Sep 10 21:22:50.234343 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:50.234316    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-95d905-4qgjq"]
Sep 10 21:22:50.288997 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:50.288958    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9dea27a-92b3-4e72-9671-8ab0eb00caa7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-95d905-4qgjq\" (UID: \"f9dea27a-92b3-4e72-9671-8ab0eb00caa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-95d905-4qgjq"
Sep 10 21:22:50.389558 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:50.389524    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9dea27a-92b3-4e72-9671-8ab0eb00caa7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-95d905-4qgjq\" (UID: \"f9dea27a-92b3-4e72-9671-8ab0eb00caa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-95d905-4qgjq"
Sep 10 21:22:50.390177 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:50.390156    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9dea27a-92b3-4e72-9671-8ab0eb00caa7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-95d905-4qgjq\" (UID: \"f9dea27a-92b3-4e72-9671-8ab0eb00caa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-95d905-4qgjq"
Sep 10 21:22:50.536436 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:50.536406    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-95d905-4qgjq"
Sep 10 21:22:50.809117 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:50.809050    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-95d905-4qgjq"]
Sep 10 21:22:50.811571 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:22:50.811546    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9dea27a_92b3_4e72_9671_8ab0eb00caa7.slice/crio-af08ed09e33843111d4e9f6c10c3a158c7af26d24717849d6bea6b68c786bfca WatchSource:0}: Error finding container af08ed09e33843111d4e9f6c10c3a158c7af26d24717849d6bea6b68c786bfca: Status 404 returned error can't find the container with id af08ed09e33843111d4e9f6c10c3a158c7af26d24717849d6bea6b68c786bfca
Sep 10 21:22:51.399767 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:51.399738    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-95d905-4qgjq" event={"ID":"f9dea27a-92b3-4e72-9671-8ab0eb00caa7","Type":"ContainerStarted","Data":"74b0a187cce29587efe2e0a806559f731f769d0755d47ad47e1ae8a632699764"}
Sep 10 21:22:51.400102 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:51.399784    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-95d905-4qgjq" event={"ID":"f9dea27a-92b3-4e72-9671-8ab0eb00caa7","Type":"ContainerStarted","Data":"af08ed09e33843111d4e9f6c10c3a158c7af26d24717849d6bea6b68c786bfca"}
Sep 10 21:22:51.417944 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:51.417892    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-95d905-4qgjq" podStartSLOduration=1.4178721539999999 podStartE2EDuration="1.417872154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:22:51.415704532 +0000 UTC m=+1556.628169452" watchObservedRunningTime="2026-09-10 21:22:51.417872154 +0000 UTC m=+1556.630337071"
Sep 10 21:22:52.410614 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:52.410576    2572 generic.go:356] "Generic (PLEG): container finished" podID="f9dea27a-92b3-4e72-9671-8ab0eb00caa7" containerID="74b0a187cce29587efe2e0a806559f731f769d0755d47ad47e1ae8a632699764" exitCode=0
Sep 10 21:22:52.411033 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:52.410618    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-95d905-4qgjq" event={"ID":"f9dea27a-92b3-4e72-9671-8ab0eb00caa7","Type":"ContainerDied","Data":"74b0a187cce29587efe2e0a806559f731f769d0755d47ad47e1ae8a632699764"}
Sep 10 21:22:53.330343 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:53.330305    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:22:53.557023 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:53.556999    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-95d905-4qgjq"
Sep 10 21:22:53.613740 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:53.613672    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9dea27a-92b3-4e72-9671-8ab0eb00caa7-maas-api-service-ca\") pod \"f9dea27a-92b3-4e72-9671-8ab0eb00caa7\" (UID: \"f9dea27a-92b3-4e72-9671-8ab0eb00caa7\") "
Sep 10 21:22:53.614028 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:53.614006    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9dea27a-92b3-4e72-9671-8ab0eb00caa7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f9dea27a-92b3-4e72-9671-8ab0eb00caa7" (UID: "f9dea27a-92b3-4e72-9671-8ab0eb00caa7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:22:53.714364 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:53.714333    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9dea27a-92b3-4e72-9671-8ab0eb00caa7-maas-api-service-ca\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:22:54.422726 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:54.422697    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-95d905-4qgjq"
Sep 10 21:22:54.422883 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:54.422732    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-95d905-4qgjq" event={"ID":"f9dea27a-92b3-4e72-9671-8ab0eb00caa7","Type":"ContainerDied","Data":"af08ed09e33843111d4e9f6c10c3a158c7af26d24717849d6bea6b68c786bfca"}
Sep 10 21:22:54.422883 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:54.422791    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af08ed09e33843111d4e9f6c10c3a158c7af26d24717849d6bea6b68c786bfca"
Sep 10 21:22:59.800504 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:22:59.800471    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"]
Sep 10 21:23:00.946710 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:00.946678    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9"]
Sep 10 21:23:00.947138 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:00.946916    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9" podUID="bf70725b-8a40-4a17-aba0-4c844c611558" containerName="payload-processing" containerID="cri-o://f7ec62f628fe54a342ef20d5bc27622e17ef867a4fed7e2e1acd6e61675f4c98" gracePeriod=30
Sep 10 21:23:01.209000 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.208921    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9"
Sep 10 21:23:01.237365 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.237338    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-95d905-4qgjq"]
Sep 10 21:23:01.240049 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.240027    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-95d905-4qgjq"]
Sep 10 21:23:01.275282 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.275206    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf70725b-8a40-4a17-aba0-4c844c611558-plugins-config-volume\") pod \"bf70725b-8a40-4a17-aba0-4c844c611558\" (UID: \"bf70725b-8a40-4a17-aba0-4c844c611558\") "
Sep 10 21:23:01.275389 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.275314    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jl8xb\" (UniqueName: \"kubernetes.io/projected/bf70725b-8a40-4a17-aba0-4c844c611558-kube-api-access-jl8xb\") pod \"bf70725b-8a40-4a17-aba0-4c844c611558\" (UID: \"bf70725b-8a40-4a17-aba0-4c844c611558\") "
Sep 10 21:23:01.275612 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.275591    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf70725b-8a40-4a17-aba0-4c844c611558-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bf70725b-8a40-4a17-aba0-4c844c611558" (UID: "bf70725b-8a40-4a17-aba0-4c844c611558"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:23:01.277603 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.277581    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf70725b-8a40-4a17-aba0-4c844c611558-kube-api-access-jl8xb" (OuterVolumeSpecName: "kube-api-access-jl8xb") pod "bf70725b-8a40-4a17-aba0-4c844c611558" (UID: "bf70725b-8a40-4a17-aba0-4c844c611558"). InnerVolumeSpecName "kube-api-access-jl8xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:23:01.376333 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.376307    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl8xb\" (UniqueName: \"kubernetes.io/projected/bf70725b-8a40-4a17-aba0-4c844c611558-kube-api-access-jl8xb\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:23:01.376333 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.376330    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf70725b-8a40-4a17-aba0-4c844c611558-plugins-config-volume\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:23:01.438624 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.438594    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9dea27a-92b3-4e72-9671-8ab0eb00caa7" path="/var/lib/kubelet/pods/f9dea27a-92b3-4e72-9671-8ab0eb00caa7/volumes"
Sep 10 21:23:01.457840 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.457815    2572 generic.go:356] "Generic (PLEG): container finished" podID="bf70725b-8a40-4a17-aba0-4c844c611558" containerID="f7ec62f628fe54a342ef20d5bc27622e17ef867a4fed7e2e1acd6e61675f4c98" exitCode=0
Sep 10 21:23:01.457962 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.457864    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9"
Sep 10 21:23:01.457962 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.457911    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9" event={"ID":"bf70725b-8a40-4a17-aba0-4c844c611558","Type":"ContainerDied","Data":"f7ec62f628fe54a342ef20d5bc27622e17ef867a4fed7e2e1acd6e61675f4c98"}
Sep 10 21:23:01.458095 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.457961    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9" event={"ID":"bf70725b-8a40-4a17-aba0-4c844c611558","Type":"ContainerDied","Data":"e1cb8d644aacf34e7fcc93e2792d81fb8045cd9e3169775c91245411a6b9f07a"}
Sep 10 21:23:01.458095 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.458008    2572 scope.go:117] "RemoveContainer" containerID="f7ec62f628fe54a342ef20d5bc27622e17ef867a4fed7e2e1acd6e61675f4c98"
Sep 10 21:23:01.466618 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.466597    2572 scope.go:117] "RemoveContainer" containerID="f7ec62f628fe54a342ef20d5bc27622e17ef867a4fed7e2e1acd6e61675f4c98"
Sep 10 21:23:01.466907 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:23:01.466890    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7ec62f628fe54a342ef20d5bc27622e17ef867a4fed7e2e1acd6e61675f4c98\": container with ID starting with f7ec62f628fe54a342ef20d5bc27622e17ef867a4fed7e2e1acd6e61675f4c98 not found: ID does not exist" containerID="f7ec62f628fe54a342ef20d5bc27622e17ef867a4fed7e2e1acd6e61675f4c98"
Sep 10 21:23:01.466970 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.466914    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7ec62f628fe54a342ef20d5bc27622e17ef867a4fed7e2e1acd6e61675f4c98"} err="failed to get container status \"f7ec62f628fe54a342ef20d5bc27622e17ef867a4fed7e2e1acd6e61675f4c98\": rpc error: code = NotFound desc = could not find container \"f7ec62f628fe54a342ef20d5bc27622e17ef867a4fed7e2e1acd6e61675f4c98\": container with ID starting with f7ec62f628fe54a342ef20d5bc27622e17ef867a4fed7e2e1acd6e61675f4c98 not found: ID does not exist"
Sep 10 21:23:01.511410 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.511389    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9"]
Sep 10 21:23:01.532591 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.532568    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-95d905-65bc8df9b7-rhvn9"]
Sep 10 21:23:01.696383 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.696355    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"]
Sep 10 21:23:01.696733 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.696719    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf70725b-8a40-4a17-aba0-4c844c611558" containerName="payload-processing"
Sep 10 21:23:01.696784 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.696736    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf70725b-8a40-4a17-aba0-4c844c611558" containerName="payload-processing"
Sep 10 21:23:01.696784 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.696754    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="079a8ef0-9f45-4e5b-bdbc-946e052e440d" containerName="istio-proxy"
Sep 10 21:23:01.696784 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.696759    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="079a8ef0-9f45-4e5b-bdbc-946e052e440d" containerName="istio-proxy"
Sep 10 21:23:01.696784 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.696769    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9dea27a-92b3-4e72-9671-8ab0eb00caa7" containerName="revoke-keys"
Sep 10 21:23:01.696784 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.696774    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9dea27a-92b3-4e72-9671-8ab0eb00caa7" containerName="revoke-keys"
Sep 10 21:23:01.696947 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.696836    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9dea27a-92b3-4e72-9671-8ab0eb00caa7" containerName="revoke-keys"
Sep 10 21:23:01.696947 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.696845    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf70725b-8a40-4a17-aba0-4c844c611558" containerName="payload-processing"
Sep 10 21:23:01.701635 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.701619    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.705166 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.705143    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 21:23:01.706411 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.706392    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-24b606-openshift-default-dockercfg-w7wpz\""
Sep 10 21:23:01.748544 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.748492    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"]
Sep 10 21:23:01.779196 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.779166    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwqr2\" (UniqueName: \"kubernetes.io/projected/bad84f81-90e2-4743-8a48-5284f8aa5e9e-kube-api-access-mwqr2\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.779290 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.779209    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-credential-socket\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.779344 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.779305    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-workload-socket\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.779344 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.779331    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istio-envoy\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.779425 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.779352    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istio-data\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.779425 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.779379    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-workload-certs\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.779425 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.779412    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istio-podinfo\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.779540 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.779436    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istiod-ca-cert\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.779540 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.779466    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istio-token\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.880487 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.880459    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istio-token\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.880600 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.880505    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwqr2\" (UniqueName: \"kubernetes.io/projected/bad84f81-90e2-4743-8a48-5284f8aa5e9e-kube-api-access-mwqr2\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.880600 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.880542    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-credential-socket\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.880725 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.880615    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-workload-socket\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.880725 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.880643    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istio-envoy\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.880725 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.880667    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istio-data\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.880725 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.880693    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-workload-certs\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.880725 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.880718    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istio-podinfo\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.880972 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.880742    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istiod-ca-cert\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.880972 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.880932    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-credential-socket\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.881114 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.881073    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istio-data\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.881177 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.881111    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-workload-certs\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.881177 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.881148    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-workload-socket\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.881475 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.881455    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istiod-ca-cert\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.883011 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.882973    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istio-envoy\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.883234 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.883218    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istio-podinfo\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.899250 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.899221    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istio-token\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:01.901621 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:01.901600    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwqr2\" (UniqueName: \"kubernetes.io/projected/bad84f81-90e2-4743-8a48-5284f8aa5e9e-kube-api-access-mwqr2\") pod \"e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") " pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:02.011781 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:02.011726    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:02.301897 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:23:02.301865    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad84f81_90e2_4743_8a48_5284f8aa5e9e.slice/crio-12b022620970cd5977e63653523952585d2d04e99699038af4038d65cb33e53f WatchSource:0}: Error finding container 12b022620970cd5977e63653523952585d2d04e99699038af4038d65cb33e53f: Status 404 returned error can't find the container with id 12b022620970cd5977e63653523952585d2d04e99699038af4038d65cb33e53f
Sep 10 21:23:02.303700 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:02.303676    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"]
Sep 10 21:23:02.304096 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:02.304062    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:23:02.304194 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:02.304128    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:23:02.304194 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:02.304160    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:23:02.472267 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:02.472232    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c" event={"ID":"bad84f81-90e2-4743-8a48-5284f8aa5e9e","Type":"ContainerStarted","Data":"3e3e659ba4fdfe15686f0293c17ab2d4d8c6a42d925fdfc4d9d9ceb1657a76fe"}
Sep 10 21:23:02.472467 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:02.472320    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c" event={"ID":"bad84f81-90e2-4743-8a48-5284f8aa5e9e","Type":"ContainerStarted","Data":"12b022620970cd5977e63653523952585d2d04e99699038af4038d65cb33e53f"}
Sep 10 21:23:02.502636 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:02.502589    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c" podStartSLOduration=1.502575138 podStartE2EDuration="1.502575138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:23:02.501222795 +0000 UTC m=+1567.713687715" watchObservedRunningTime="2026-09-10 21:23:02.502575138 +0000 UTC m=+1567.715040056"
Sep 10 21:23:03.012220 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:03.012189    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:03.017290 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:03.017263    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:03.439035 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:03.438999    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf70725b-8a40-4a17-aba0-4c844c611558" path="/var/lib/kubelet/pods/bf70725b-8a40-4a17-aba0-4c844c611558/volumes"
Sep 10 21:23:03.478542 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:03.478520    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:03.479615 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:03.479592    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:10.667377 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:10.667342    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:23:14.824697 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:14.824652    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r" podUID="dea3f743-8c01-49e5-890e-2d65eba3efc2" containerName="main" containerID="cri-o://52537db3ab16e77a1964fe64f7842c37ae7d729b3d14fbebb6ccc916d3a85d40" gracePeriod=45
Sep 10 21:23:15.107829 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.107800    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:23:15.186235 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.186206    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28nbp\" (UniqueName: \"kubernetes.io/projected/dea3f743-8c01-49e5-890e-2d65eba3efc2-kube-api-access-28nbp\") pod \"dea3f743-8c01-49e5-890e-2d65eba3efc2\" (UID: \"dea3f743-8c01-49e5-890e-2d65eba3efc2\") "
Sep 10 21:23:15.186356 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.186246    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea3f743-8c01-49e5-890e-2d65eba3efc2-dshm\") pod \"dea3f743-8c01-49e5-890e-2d65eba3efc2\" (UID: \"dea3f743-8c01-49e5-890e-2d65eba3efc2\") "
Sep 10 21:23:15.186356 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.186263    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea3f743-8c01-49e5-890e-2d65eba3efc2-home\") pod \"dea3f743-8c01-49e5-890e-2d65eba3efc2\" (UID: \"dea3f743-8c01-49e5-890e-2d65eba3efc2\") "
Sep 10 21:23:15.186356 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.186295    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea3f743-8c01-49e5-890e-2d65eba3efc2-tmp-dir\") pod \"dea3f743-8c01-49e5-890e-2d65eba3efc2\" (UID: \"dea3f743-8c01-49e5-890e-2d65eba3efc2\") "
Sep 10 21:23:15.186356 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.186323    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea3f743-8c01-49e5-890e-2d65eba3efc2-tls-certs\") pod \"dea3f743-8c01-49e5-890e-2d65eba3efc2\" (UID: \"dea3f743-8c01-49e5-890e-2d65eba3efc2\") "
Sep 10 21:23:15.186356 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.186343    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea3f743-8c01-49e5-890e-2d65eba3efc2-model-cache\") pod \"dea3f743-8c01-49e5-890e-2d65eba3efc2\" (UID: \"dea3f743-8c01-49e5-890e-2d65eba3efc2\") "
Sep 10 21:23:15.186629 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.186567    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea3f743-8c01-49e5-890e-2d65eba3efc2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dea3f743-8c01-49e5-890e-2d65eba3efc2" (UID: "dea3f743-8c01-49e5-890e-2d65eba3efc2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:23:15.186689 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.186636    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea3f743-8c01-49e5-890e-2d65eba3efc2-home" (OuterVolumeSpecName: "home") pod "dea3f743-8c01-49e5-890e-2d65eba3efc2" (UID: "dea3f743-8c01-49e5-890e-2d65eba3efc2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:23:15.186728 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.186688    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea3f743-8c01-49e5-890e-2d65eba3efc2-model-cache" (OuterVolumeSpecName: "model-cache") pod "dea3f743-8c01-49e5-890e-2d65eba3efc2" (UID: "dea3f743-8c01-49e5-890e-2d65eba3efc2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:23:15.188585 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.188558    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea3f743-8c01-49e5-890e-2d65eba3efc2-dshm" (OuterVolumeSpecName: "dshm") pod "dea3f743-8c01-49e5-890e-2d65eba3efc2" (UID: "dea3f743-8c01-49e5-890e-2d65eba3efc2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:23:15.188706 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.188583    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dea3f743-8c01-49e5-890e-2d65eba3efc2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dea3f743-8c01-49e5-890e-2d65eba3efc2" (UID: "dea3f743-8c01-49e5-890e-2d65eba3efc2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:23:15.188706 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.188659    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea3f743-8c01-49e5-890e-2d65eba3efc2-kube-api-access-28nbp" (OuterVolumeSpecName: "kube-api-access-28nbp") pod "dea3f743-8c01-49e5-890e-2d65eba3efc2" (UID: "dea3f743-8c01-49e5-890e-2d65eba3efc2"). InnerVolumeSpecName "kube-api-access-28nbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:23:15.287879 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.287854    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28nbp\" (UniqueName: \"kubernetes.io/projected/dea3f743-8c01-49e5-890e-2d65eba3efc2-kube-api-access-28nbp\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:23:15.287879 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.287876    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea3f743-8c01-49e5-890e-2d65eba3efc2-dshm\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:23:15.287879 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.287885    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea3f743-8c01-49e5-890e-2d65eba3efc2-home\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:23:15.288101 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.287895    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea3f743-8c01-49e5-890e-2d65eba3efc2-tmp-dir\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:23:15.288101 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.287904    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea3f743-8c01-49e5-890e-2d65eba3efc2-tls-certs\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:23:15.288101 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.287911    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea3f743-8c01-49e5-890e-2d65eba3efc2-model-cache\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:23:15.547155 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.547120    2572 generic.go:356] "Generic (PLEG): container finished" podID="dea3f743-8c01-49e5-890e-2d65eba3efc2" containerID="52537db3ab16e77a1964fe64f7842c37ae7d729b3d14fbebb6ccc916d3a85d40" exitCode=0
Sep 10 21:23:15.547292 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.547168    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"
Sep 10 21:23:15.547292 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.547206    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r" event={"ID":"dea3f743-8c01-49e5-890e-2d65eba3efc2","Type":"ContainerDied","Data":"52537db3ab16e77a1964fe64f7842c37ae7d729b3d14fbebb6ccc916d3a85d40"}
Sep 10 21:23:15.547292 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.547251    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r" event={"ID":"dea3f743-8c01-49e5-890e-2d65eba3efc2","Type":"ContainerDied","Data":"1816df740198e29155e0ad3ad2f84eca6ab44907d6f4b5d820fb13a2280e3a19"}
Sep 10 21:23:15.547292 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.547272    2572 scope.go:117] "RemoveContainer" containerID="52537db3ab16e77a1964fe64f7842c37ae7d729b3d14fbebb6ccc916d3a85d40"
Sep 10 21:23:15.558033 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.558017    2572 scope.go:117] "RemoveContainer" containerID="52537db3ab16e77a1964fe64f7842c37ae7d729b3d14fbebb6ccc916d3a85d40"
Sep 10 21:23:15.558341 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:23:15.558319    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52537db3ab16e77a1964fe64f7842c37ae7d729b3d14fbebb6ccc916d3a85d40\": container with ID starting with 52537db3ab16e77a1964fe64f7842c37ae7d729b3d14fbebb6ccc916d3a85d40 not found: ID does not exist" containerID="52537db3ab16e77a1964fe64f7842c37ae7d729b3d14fbebb6ccc916d3a85d40"
Sep 10 21:23:15.558416 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.558353    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52537db3ab16e77a1964fe64f7842c37ae7d729b3d14fbebb6ccc916d3a85d40"} err="failed to get container status \"52537db3ab16e77a1964fe64f7842c37ae7d729b3d14fbebb6ccc916d3a85d40\": rpc error: code = NotFound desc = could not find container \"52537db3ab16e77a1964fe64f7842c37ae7d729b3d14fbebb6ccc916d3a85d40\": container with ID starting with 52537db3ab16e77a1964fe64f7842c37ae7d729b3d14fbebb6ccc916d3a85d40 not found: ID does not exist"
Sep 10 21:23:15.574478 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.574453    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"]
Sep 10 21:23:15.584227 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:15.584209    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-af7150/rate-test-model-af7150-kserve-b966b47ff-mwj2r"]
Sep 10 21:23:17.438089 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:17.438052    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dea3f743-8c01-49e5-890e-2d65eba3efc2" path="/var/lib/kubelet/pods/dea3f743-8c01-49e5-890e-2d65eba3efc2/volumes"
Sep 10 21:23:19.520135 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:19.520092    2572 patch_prober.go:28] interesting pod/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 21:23:19.520619 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:19.520166    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c" podUID="bad84f81-90e2-4743-8a48-5284f8aa5e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 21:23:20.574600 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:20.574570    2572 generic.go:356] "Generic (PLEG): container finished" podID="bad84f81-90e2-4743-8a48-5284f8aa5e9e" containerID="3e3e659ba4fdfe15686f0293c17ab2d4d8c6a42d925fdfc4d9d9ceb1657a76fe" exitCode=0
Sep 10 21:23:20.574961 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:20.574604    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c" event={"ID":"bad84f81-90e2-4743-8a48-5284f8aa5e9e","Type":"ContainerDied","Data":"3e3e659ba4fdfe15686f0293c17ab2d4d8c6a42d925fdfc4d9d9ceb1657a76fe"}
Sep 10 21:23:20.574961 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:20.574932    2572 scope.go:117] "RemoveContainer" containerID="3e3e659ba4fdfe15686f0293c17ab2d4d8c6a42d925fdfc4d9d9ceb1657a76fe"
Sep 10 21:23:20.576056 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:20.576023    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:23:20.576132 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:20.576116    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:23:20.576170 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:20.576159    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:23:21.581084 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:21.581044    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c" event={"ID":"bad84f81-90e2-4743-8a48-5284f8aa5e9e","Type":"ContainerStarted","Data":"d5695ca56ac9e4aa29154f210f7455a154b80dbae27232911acbe78d43cdb797"}
Sep 10 21:23:22.012578 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:22.012484    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:22.012908 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:22.012884    2572 patch_prober.go:28] interesting pod/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Sep 10 21:23:22.013107 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:22.013077    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c" podUID="bad84f81-90e2-4743-8a48-5284f8aa5e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Sep 10 21:23:23.012155 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:23.012122    2572 patch_prober.go:28] interesting pod/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Sep 10 21:23:23.012721 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:23.012689    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c" podUID="bad84f81-90e2-4743-8a48-5284f8aa5e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Sep 10 21:23:24.012401 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:24.012356    2572 patch_prober.go:28] interesting pod/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Sep 10 21:23:24.012888 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:24.012423    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c" podUID="bad84f81-90e2-4743-8a48-5284f8aa5e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Sep 10 21:23:25.012425 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:25.012390    2572 patch_prober.go:28] interesting pod/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Sep 10 21:23:25.013036 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:25.012973    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c" podUID="bad84f81-90e2-4743-8a48-5284f8aa5e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Sep 10 21:23:26.012364 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:26.012330    2572 patch_prober.go:28] interesting pod/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Sep 10 21:23:26.012688 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:26.012641    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c" podUID="bad84f81-90e2-4743-8a48-5284f8aa5e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Sep 10 21:23:27.015302 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:27.015273    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:27.015748 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:27.015529    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:27.016144 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:27.016126    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:23:41.336392 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:41.336362    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf"]
Sep 10 21:23:41.336765 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:41.336725    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dea3f743-8c01-49e5-890e-2d65eba3efc2" containerName="main"
Sep 10 21:23:41.336765 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:41.336736    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="dea3f743-8c01-49e5-890e-2d65eba3efc2" containerName="main"
Sep 10 21:23:41.336840 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:41.336798    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="dea3f743-8c01-49e5-890e-2d65eba3efc2" containerName="main"
Sep 10 21:23:41.339894 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:41.339877    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf"
Sep 10 21:23:41.347320 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:41.347302    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9f1286\""
Sep 10 21:23:41.347926 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:41.347910    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9f1286-dockercfg-dzwn7\""
Sep 10 21:23:41.399933 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:41.399905    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf"]
Sep 10 21:23:41.502106 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:41.502077    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ebf073-e170-442f-8be4-0f47b3cb2e42-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf\" (UID: \"71ebf073-e170-442f-8be4-0f47b3cb2e42\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf"
Sep 10 21:23:41.502237 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:41.502136    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4p7r\" (UniqueName: \"kubernetes.io/projected/71ebf073-e170-442f-8be4-0f47b3cb2e42-kube-api-access-q4p7r\") pod \"payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf\" (UID: \"71ebf073-e170-442f-8be4-0f47b3cb2e42\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf"
Sep 10 21:23:41.603243 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:41.603178    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ebf073-e170-442f-8be4-0f47b3cb2e42-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf\" (UID: \"71ebf073-e170-442f-8be4-0f47b3cb2e42\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf"
Sep 10 21:23:41.603243 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:41.603231    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4p7r\" (UniqueName: \"kubernetes.io/projected/71ebf073-e170-442f-8be4-0f47b3cb2e42-kube-api-access-q4p7r\") pod \"payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf\" (UID: \"71ebf073-e170-442f-8be4-0f47b3cb2e42\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf"
Sep 10 21:23:41.603841 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:41.603821    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ebf073-e170-442f-8be4-0f47b3cb2e42-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf\" (UID: \"71ebf073-e170-442f-8be4-0f47b3cb2e42\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf"
Sep 10 21:23:41.619958 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:41.619931    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4p7r\" (UniqueName: \"kubernetes.io/projected/71ebf073-e170-442f-8be4-0f47b3cb2e42-kube-api-access-q4p7r\") pod \"payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf\" (UID: \"71ebf073-e170-442f-8be4-0f47b3cb2e42\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf"
Sep 10 21:23:41.650418 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:41.650393    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf"
Sep 10 21:23:41.936222 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:41.936128    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf"]
Sep 10 21:23:41.939273 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:23:41.939244    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ebf073_e170_442f_8be4_0f47b3cb2e42.slice/crio-acc92ed8ce1209f7df536323efbd4c83196300b939e1cf4ab3c841a4a1837115 WatchSource:0}: Error finding container acc92ed8ce1209f7df536323efbd4c83196300b939e1cf4ab3c841a4a1837115: Status 404 returned error can't find the container with id acc92ed8ce1209f7df536323efbd4c83196300b939e1cf4ab3c841a4a1837115
Sep 10 21:23:42.689058 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:42.689026    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf" event={"ID":"71ebf073-e170-442f-8be4-0f47b3cb2e42","Type":"ContainerStarted","Data":"eed0680b2945bec4e70e009e006456bfc6a0d1c4de5ed08773f8e776e1751433"}
Sep 10 21:23:42.689474 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:42.689079    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf" event={"ID":"71ebf073-e170-442f-8be4-0f47b3cb2e42","Type":"ContainerStarted","Data":"acc92ed8ce1209f7df536323efbd4c83196300b939e1cf4ab3c841a4a1837115"}
Sep 10 21:23:42.689474 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:42.689103    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf"
Sep 10 21:23:42.720620 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:42.720575    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf" podStartSLOduration=1.720560476 podStartE2EDuration="1.720560476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:23:42.720179013 +0000 UTC m=+1607.932643932" watchObservedRunningTime="2026-09-10 21:23:42.720560476 +0000 UTC m=+1607.933025415"
Sep 10 21:23:46.649602 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:46.649565    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:23:53.697433 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:23:53.697340    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf"
Sep 10 21:24:32.619610 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.619575    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"]
Sep 10 21:24:32.623097 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.623073    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:24:32.628604 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.628580    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-24b606\"/\"kube-root-ca.crt\""
Sep 10 21:24:32.629614 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.629592    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-24b606\"/\"default-dockercfg-fbcz2\""
Sep 10 21:24:32.629742 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.629622    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-24b606\"/\"openshift-service-ca.crt\""
Sep 10 21:24:32.629742 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.629623    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-24b606\"/\"ipp-route-24b606-kserve-self-signed-certs\""
Sep 10 21:24:32.635533 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.635508    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"]
Sep 10 21:24:32.710445 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.710425    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-tmp-dir\") pod \"ipp-route-24b606-kserve-6c5f8c7cc9-xs457\" (UID: \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\") " pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:24:32.710552 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.710477    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-home\") pod \"ipp-route-24b606-kserve-6c5f8c7cc9-xs457\" (UID: \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\") " pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:24:32.710552 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.710525    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-model-cache\") pod \"ipp-route-24b606-kserve-6c5f8c7cc9-xs457\" (UID: \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\") " pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:24:32.710653 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.710565    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-dshm\") pod \"ipp-route-24b606-kserve-6c5f8c7cc9-xs457\" (UID: \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\") " pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:24:32.710653 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.710620    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-tls-certs\") pod \"ipp-route-24b606-kserve-6c5f8c7cc9-xs457\" (UID: \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\") " pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:24:32.710653 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.710641    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djbj2\" (UniqueName: \"kubernetes.io/projected/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-kube-api-access-djbj2\") pod \"ipp-route-24b606-kserve-6c5f8c7cc9-xs457\" (UID: \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\") " pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:24:32.811289 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.811262    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-model-cache\") pod \"ipp-route-24b606-kserve-6c5f8c7cc9-xs457\" (UID: \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\") " pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:24:32.811450 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.811298    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-dshm\") pod \"ipp-route-24b606-kserve-6c5f8c7cc9-xs457\" (UID: \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\") " pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:24:32.811450 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.811316    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-tls-certs\") pod \"ipp-route-24b606-kserve-6c5f8c7cc9-xs457\" (UID: \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\") " pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:24:32.811450 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.811333    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djbj2\" (UniqueName: \"kubernetes.io/projected/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-kube-api-access-djbj2\") pod \"ipp-route-24b606-kserve-6c5f8c7cc9-xs457\" (UID: \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\") " pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:24:32.811450 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.811369    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-tmp-dir\") pod \"ipp-route-24b606-kserve-6c5f8c7cc9-xs457\" (UID: \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\") " pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:24:32.811450 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.811423    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-home\") pod \"ipp-route-24b606-kserve-6c5f8c7cc9-xs457\" (UID: \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\") " pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:24:32.811735 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.811716    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-model-cache\") pod \"ipp-route-24b606-kserve-6c5f8c7cc9-xs457\" (UID: \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\") " pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:24:32.811822 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.811803    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-home\") pod \"ipp-route-24b606-kserve-6c5f8c7cc9-xs457\" (UID: \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\") " pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:24:32.811867 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.811841    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-tmp-dir\") pod \"ipp-route-24b606-kserve-6c5f8c7cc9-xs457\" (UID: \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\") " pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:24:32.813657 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.813634    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-dshm\") pod \"ipp-route-24b606-kserve-6c5f8c7cc9-xs457\" (UID: \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\") " pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:24:32.813881 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.813853    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-tls-certs\") pod \"ipp-route-24b606-kserve-6c5f8c7cc9-xs457\" (UID: \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\") " pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:24:32.820668 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.820645    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djbj2\" (UniqueName: \"kubernetes.io/projected/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-kube-api-access-djbj2\") pod \"ipp-route-24b606-kserve-6c5f8c7cc9-xs457\" (UID: \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\") " pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:24:32.935850 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:32.935787    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:24:33.213497 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:33.213473    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"]
Sep 10 21:24:33.216473 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:24:33.216447    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2b2bdb5_1498_4a0b_93ad_824ffdef01ef.slice/crio-3b7cf2aeb9a988fbe6005283cf13036581e7fe432c5bff14fce69a6b3a851b04 WatchSource:0}: Error finding container 3b7cf2aeb9a988fbe6005283cf13036581e7fe432c5bff14fce69a6b3a851b04: Status 404 returned error can't find the container with id 3b7cf2aeb9a988fbe6005283cf13036581e7fe432c5bff14fce69a6b3a851b04
Sep 10 21:24:33.953936 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:33.953888    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457" event={"ID":"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef","Type":"ContainerStarted","Data":"e7883b187a405792c6abb4d53dac1d69953610ac6e3b6f9803427390b5605d80"}
Sep 10 21:24:33.954428 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:33.953954    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457" event={"ID":"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef","Type":"ContainerStarted","Data":"3b7cf2aeb9a988fbe6005283cf13036581e7fe432c5bff14fce69a6b3a851b04"}
Sep 10 21:24:33.974298 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:33.974240    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457" podStartSLOduration=1.974223405 podStartE2EDuration="1.974223405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:24:33.972065327 +0000 UTC m=+1659.184530246" watchObservedRunningTime="2026-09-10 21:24:33.974223405 +0000 UTC m=+1659.186688386"
Sep 10 21:24:36.727774 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:36.727744    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:24:42.936844 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:42.936806    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:24:42.937409 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:42.936858    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:24:42.949596 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:42.949562    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:24:43.011561 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:43.011535    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:24:49.037771 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:49.037735    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:24:55.657284 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.657251    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"]
Sep 10 21:24:55.661039 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.661017    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.663669 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.663650    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-1ca93e-openshift-default-dockercfg-8j5lj\""
Sep 10 21:24:55.677371 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.677348    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"]
Sep 10 21:24:55.796836 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.796806    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-credential-socket\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.797024 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.796888    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istio-token\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.797024 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.796915    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istio-data\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.797024 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.796939    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-workload-socket\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.797024 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.796959    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-workload-certs\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.797237 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.797038    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istio-podinfo\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.797237 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.797061    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.797237 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.797104    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sqwc\" (UniqueName: \"kubernetes.io/projected/aa363798-f3d2-4cef-8dfb-af5e0043eee8-kube-api-access-8sqwc\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.797237 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.797179    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istio-envoy\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.898239 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.898215    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-credential-socket\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.898395 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.898271    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istio-token\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.898395 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.898297    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istio-data\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.898395 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.898326    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-workload-socket\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.898395 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.898353    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-workload-certs\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.898395 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.898380    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istio-podinfo\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.898655 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.898406    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.898655 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.898465    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sqwc\" (UniqueName: \"kubernetes.io/projected/aa363798-f3d2-4cef-8dfb-af5e0043eee8-kube-api-access-8sqwc\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.898655 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.898522    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istio-envoy\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.898811 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.898656    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-credential-socket\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.898811 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.898725    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istio-data\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.898811 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.898761    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-workload-socket\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.898950 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.898906    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-workload-certs\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.899292 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.899273    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.900719 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.900695    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istio-envoy\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.900924 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.900900    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istio-podinfo\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.908557 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.908507    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istio-token\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.908754 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.908718    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sqwc\" (UniqueName: \"kubernetes.io/projected/aa363798-f3d2-4cef-8dfb-af5e0043eee8-kube-api-access-8sqwc\") pod \"e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") " pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:55.970296 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:55.970268    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:56.247408 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:56.247382    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"]
Sep 10 21:24:56.248806 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:24:56.248775    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa363798_f3d2_4cef_8dfb_af5e0043eee8.slice/crio-bdfcf8e6f2fe391594d9c847924cfd3d46d3568534c6a84cfd6e89ed3b4be690 WatchSource:0}: Error finding container bdfcf8e6f2fe391594d9c847924cfd3d46d3568534c6a84cfd6e89ed3b4be690: Status 404 returned error can't find the container with id bdfcf8e6f2fe391594d9c847924cfd3d46d3568534c6a84cfd6e89ed3b4be690
Sep 10 21:24:56.251342 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:56.251308    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:24:56.251442 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:56.251390    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:24:56.251442 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:56.251434    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:24:57.075669 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:57.075637    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk" event={"ID":"aa363798-f3d2-4cef-8dfb-af5e0043eee8","Type":"ContainerStarted","Data":"f24461bf6874913fb3e9dd669d8f823bf4d47d6a243fc929799d9013b81d85b4"}
Sep 10 21:24:57.076079 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:57.075692    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk" event={"ID":"aa363798-f3d2-4cef-8dfb-af5e0043eee8","Type":"ContainerStarted","Data":"bdfcf8e6f2fe391594d9c847924cfd3d46d3568534c6a84cfd6e89ed3b4be690"}
Sep 10 21:24:57.100316 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:57.100267    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk" podStartSLOduration=2.10025315 podStartE2EDuration="2.10025315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:24:57.098332384 +0000 UTC m=+1682.310797317" watchObservedRunningTime="2026-09-10 21:24:57.10025315 +0000 UTC m=+1682.312718069"
Sep 10 21:24:57.971224 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:57.971190    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:57.977597 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:57.977566    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:58.081776 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:58.081746    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:24:58.082706 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:24:58.082683    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:25:00.865935 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:00.865904    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls"]
Sep 10 21:25:00.869316 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:00.869301    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls"
Sep 10 21:25:00.872074 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:00.872054    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-1ca93e-dockercfg-7pxwt\""
Sep 10 21:25:00.872196 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:00.872128    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-1ca93e\""
Sep 10 21:25:00.882784 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:00.882765    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls"]
Sep 10 21:25:01.041669 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:01.041641    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz9z7\" (UniqueName: \"kubernetes.io/projected/ab8e6952-6f9d-4c24-9328-1ce3d801ae14-kube-api-access-gz9z7\") pod \"payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls\" (UID: \"ab8e6952-6f9d-4c24-9328-1ce3d801ae14\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls"
Sep 10 21:25:01.041846 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:01.041755    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8e6952-6f9d-4c24-9328-1ce3d801ae14-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls\" (UID: \"ab8e6952-6f9d-4c24-9328-1ce3d801ae14\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls"
Sep 10 21:25:01.142622 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:01.142532    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz9z7\" (UniqueName: \"kubernetes.io/projected/ab8e6952-6f9d-4c24-9328-1ce3d801ae14-kube-api-access-gz9z7\") pod \"payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls\" (UID: \"ab8e6952-6f9d-4c24-9328-1ce3d801ae14\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls"
Sep 10 21:25:01.142622 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:01.142612    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8e6952-6f9d-4c24-9328-1ce3d801ae14-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls\" (UID: \"ab8e6952-6f9d-4c24-9328-1ce3d801ae14\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls"
Sep 10 21:25:01.143321 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:01.143302    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8e6952-6f9d-4c24-9328-1ce3d801ae14-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls\" (UID: \"ab8e6952-6f9d-4c24-9328-1ce3d801ae14\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls"
Sep 10 21:25:01.151822 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:01.151802    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz9z7\" (UniqueName: \"kubernetes.io/projected/ab8e6952-6f9d-4c24-9328-1ce3d801ae14-kube-api-access-gz9z7\") pod \"payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls\" (UID: \"ab8e6952-6f9d-4c24-9328-1ce3d801ae14\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls"
Sep 10 21:25:01.180193 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:01.180173    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls"
Sep 10 21:25:01.459237 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:01.459156    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls"]
Sep 10 21:25:01.461759 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:25:01.461732    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab8e6952_6f9d_4c24_9328_1ce3d801ae14.slice/crio-3ffacc112be2eef9575c118bacd166ebffecbb84ea3af4086354faff96599b14 WatchSource:0}: Error finding container 3ffacc112be2eef9575c118bacd166ebffecbb84ea3af4086354faff96599b14: Status 404 returned error can't find the container with id 3ffacc112be2eef9575c118bacd166ebffecbb84ea3af4086354faff96599b14
Sep 10 21:25:02.102085 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:02.102045    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls" event={"ID":"ab8e6952-6f9d-4c24-9328-1ce3d801ae14","Type":"ContainerStarted","Data":"707e60eb70a7467fd9263df62a2a5fa691c7c2712d07a76645ccf92feee96aa9"}
Sep 10 21:25:02.102555 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:02.102108    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls" event={"ID":"ab8e6952-6f9d-4c24-9328-1ce3d801ae14","Type":"ContainerStarted","Data":"3ffacc112be2eef9575c118bacd166ebffecbb84ea3af4086354faff96599b14"}
Sep 10 21:25:02.102555 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:02.102223    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls"
Sep 10 21:25:02.123165 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:02.123123    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls" podStartSLOduration=2.123109457 podStartE2EDuration="2.123109457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:25:02.121129028 +0000 UTC m=+1687.333593947" watchObservedRunningTime="2026-09-10 21:25:02.123109457 +0000 UTC m=+1687.335574419"
Sep 10 21:25:06.938071 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:06.938026    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:25:13.108826 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:13.108793    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls"
Sep 10 21:25:17.184141 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:17.184074    2572 generic.go:356] "Generic (PLEG): container finished" podID="aa363798-f3d2-4cef-8dfb-af5e0043eee8" containerID="f24461bf6874913fb3e9dd669d8f823bf4d47d6a243fc929799d9013b81d85b4" exitCode=0
Sep 10 21:25:17.184477 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:17.184150    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk" event={"ID":"aa363798-f3d2-4cef-8dfb-af5e0043eee8","Type":"ContainerDied","Data":"f24461bf6874913fb3e9dd669d8f823bf4d47d6a243fc929799d9013b81d85b4"}
Sep 10 21:25:17.184534 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:17.184519    2572 scope.go:117] "RemoveContainer" containerID="f24461bf6874913fb3e9dd669d8f823bf4d47d6a243fc929799d9013b81d85b4"
Sep 10 21:25:17.185711 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:17.185682    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:25:17.185775 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:17.185765    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:25:17.185812 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:17.185799    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 10 21:25:18.192554 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:18.192476    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk" event={"ID":"aa363798-f3d2-4cef-8dfb-af5e0043eee8","Type":"ContainerStarted","Data":"1f994826351e0313c330090783b69e63b7596e54efc84e62e12ee62dc024fdab"}
Sep 10 21:25:18.970723 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:18.970685    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:25:18.970906 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:18.970816    2572 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 10 21:25:18.970906 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:18.970864    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk" podUID="aa363798-f3d2-4cef-8dfb-af5e0043eee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 10 21:25:19.970784 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:19.970747    2572 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 10 21:25:19.971258 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:19.970802    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk" podUID="aa363798-f3d2-4cef-8dfb-af5e0043eee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 10 21:25:20.971303 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:20.971263    2572 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 10 21:25:20.971772 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:20.971337    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk" podUID="aa363798-f3d2-4cef-8dfb-af5e0043eee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 10 21:25:21.971335 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:21.971297    2572 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 10 21:25:21.971723 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:21.971355    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk" podUID="aa363798-f3d2-4cef-8dfb-af5e0043eee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 10 21:25:22.970923 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:22.970889    2572 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 10 21:25:22.971120 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:22.970937    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk" podUID="aa363798-f3d2-4cef-8dfb-af5e0043eee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 10 21:25:23.971461 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:23.971422    2572 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 10 21:25:23.971845 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:23.971473    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk" podUID="aa363798-f3d2-4cef-8dfb-af5e0043eee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 10 21:25:24.974576 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:24.974548    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:25:24.975075 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:24.974932    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:25:24.975500 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:24.975481    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:25:31.034216 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:31.034185    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls"]
Sep 10 21:25:31.034691 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:31.034451    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls" podUID="ab8e6952-6f9d-4c24-9328-1ce3d801ae14" containerName="payload-processing" containerID="cri-o://707e60eb70a7467fd9263df62a2a5fa691c7c2712d07a76645ccf92feee96aa9" gracePeriod=30
Sep 10 21:25:31.284591 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:31.284521    2572 generic.go:356] "Generic (PLEG): container finished" podID="ab8e6952-6f9d-4c24-9328-1ce3d801ae14" containerID="707e60eb70a7467fd9263df62a2a5fa691c7c2712d07a76645ccf92feee96aa9" exitCode=0
Sep 10 21:25:31.284697 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:31.284594    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls" event={"ID":"ab8e6952-6f9d-4c24-9328-1ce3d801ae14","Type":"ContainerDied","Data":"707e60eb70a7467fd9263df62a2a5fa691c7c2712d07a76645ccf92feee96aa9"}
Sep 10 21:25:31.284697 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:31.284644    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls" event={"ID":"ab8e6952-6f9d-4c24-9328-1ce3d801ae14","Type":"ContainerDied","Data":"3ffacc112be2eef9575c118bacd166ebffecbb84ea3af4086354faff96599b14"}
Sep 10 21:25:31.284697 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:31.284659    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ffacc112be2eef9575c118bacd166ebffecbb84ea3af4086354faff96599b14"
Sep 10 21:25:31.288513 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:31.288496    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls"
Sep 10 21:25:31.417175 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:31.417143    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8e6952-6f9d-4c24-9328-1ce3d801ae14-plugins-config-volume\") pod \"ab8e6952-6f9d-4c24-9328-1ce3d801ae14\" (UID: \"ab8e6952-6f9d-4c24-9328-1ce3d801ae14\") "
Sep 10 21:25:31.417321 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:31.417228    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz9z7\" (UniqueName: \"kubernetes.io/projected/ab8e6952-6f9d-4c24-9328-1ce3d801ae14-kube-api-access-gz9z7\") pod \"ab8e6952-6f9d-4c24-9328-1ce3d801ae14\" (UID: \"ab8e6952-6f9d-4c24-9328-1ce3d801ae14\") "
Sep 10 21:25:31.417516 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:31.417495    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab8e6952-6f9d-4c24-9328-1ce3d801ae14-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ab8e6952-6f9d-4c24-9328-1ce3d801ae14" (UID: "ab8e6952-6f9d-4c24-9328-1ce3d801ae14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:25:31.419475 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:31.419454    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab8e6952-6f9d-4c24-9328-1ce3d801ae14-kube-api-access-gz9z7" (OuterVolumeSpecName: "kube-api-access-gz9z7") pod "ab8e6952-6f9d-4c24-9328-1ce3d801ae14" (UID: "ab8e6952-6f9d-4c24-9328-1ce3d801ae14"). InnerVolumeSpecName "kube-api-access-gz9z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:25:31.517959 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:31.517931    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz9z7\" (UniqueName: \"kubernetes.io/projected/ab8e6952-6f9d-4c24-9328-1ce3d801ae14-kube-api-access-gz9z7\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:25:31.517959 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:31.517951    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8e6952-6f9d-4c24-9328-1ce3d801ae14-plugins-config-volume\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:25:32.290143 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:32.290116    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls"
Sep 10 21:25:32.308538 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:32.308506    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls"]
Sep 10 21:25:32.315293 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:32.315262    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1ca93e-79c54f84ff-xjlls"]
Sep 10 21:25:33.439737 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:33.439701    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab8e6952-6f9d-4c24-9328-1ce3d801ae14" path="/var/lib/kubelet/pods/ab8e6952-6f9d-4c24-9328-1ce3d801ae14/volumes"
Sep 10 21:25:35.328554 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:35.328500    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:25:38.090630 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:38.090592    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"]
Sep 10 21:25:38.091196 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:38.090842    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk" podUID="aa363798-f3d2-4cef-8dfb-af5e0043eee8" containerName="istio-proxy" containerID="cri-o://1f994826351e0313c330090783b69e63b7596e54efc84e62e12ee62dc024fdab" gracePeriod=30
Sep 10 21:25:39.976955 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:39.976921    2572 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 10 21:25:39.977420 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:39.977028    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk" podUID="aa363798-f3d2-4cef-8dfb-af5e0043eee8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 21:25:43.341274 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.341250    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:25:43.349892 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.349856    2572 generic.go:356] "Generic (PLEG): container finished" podID="aa363798-f3d2-4cef-8dfb-af5e0043eee8" containerID="1f994826351e0313c330090783b69e63b7596e54efc84e62e12ee62dc024fdab" exitCode=0
Sep 10 21:25:43.350014 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.349904    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"
Sep 10 21:25:43.350014 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.349905    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk" event={"ID":"aa363798-f3d2-4cef-8dfb-af5e0043eee8","Type":"ContainerDied","Data":"1f994826351e0313c330090783b69e63b7596e54efc84e62e12ee62dc024fdab"}
Sep 10 21:25:43.350014 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.349970    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk" event={"ID":"aa363798-f3d2-4cef-8dfb-af5e0043eee8","Type":"ContainerDied","Data":"bdfcf8e6f2fe391594d9c847924cfd3d46d3568534c6a84cfd6e89ed3b4be690"}
Sep 10 21:25:43.350014 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.350006    2572 scope.go:117] "RemoveContainer" containerID="1f994826351e0313c330090783b69e63b7596e54efc84e62e12ee62dc024fdab"
Sep 10 21:25:43.360579 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.360481    2572 scope.go:117] "RemoveContainer" containerID="f24461bf6874913fb3e9dd669d8f823bf4d47d6a243fc929799d9013b81d85b4"
Sep 10 21:25:43.370674 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.370657    2572 scope.go:117] "RemoveContainer" containerID="1f994826351e0313c330090783b69e63b7596e54efc84e62e12ee62dc024fdab"
Sep 10 21:25:43.370926 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:25:43.370907    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f994826351e0313c330090783b69e63b7596e54efc84e62e12ee62dc024fdab\": container with ID starting with 1f994826351e0313c330090783b69e63b7596e54efc84e62e12ee62dc024fdab not found: ID does not exist" containerID="1f994826351e0313c330090783b69e63b7596e54efc84e62e12ee62dc024fdab"
Sep 10 21:25:43.371014 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.370935    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f994826351e0313c330090783b69e63b7596e54efc84e62e12ee62dc024fdab"} err="failed to get container status \"1f994826351e0313c330090783b69e63b7596e54efc84e62e12ee62dc024fdab\": rpc error: code = NotFound desc = could not find container \"1f994826351e0313c330090783b69e63b7596e54efc84e62e12ee62dc024fdab\": container with ID starting with 1f994826351e0313c330090783b69e63b7596e54efc84e62e12ee62dc024fdab not found: ID does not exist"
Sep 10 21:25:43.371014 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.370953    2572 scope.go:117] "RemoveContainer" containerID="f24461bf6874913fb3e9dd669d8f823bf4d47d6a243fc929799d9013b81d85b4"
Sep 10 21:25:43.371238 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:25:43.371223    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f24461bf6874913fb3e9dd669d8f823bf4d47d6a243fc929799d9013b81d85b4\": container with ID starting with f24461bf6874913fb3e9dd669d8f823bf4d47d6a243fc929799d9013b81d85b4 not found: ID does not exist" containerID="f24461bf6874913fb3e9dd669d8f823bf4d47d6a243fc929799d9013b81d85b4"
Sep 10 21:25:43.371281 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.371243    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f24461bf6874913fb3e9dd669d8f823bf4d47d6a243fc929799d9013b81d85b4"} err="failed to get container status \"f24461bf6874913fb3e9dd669d8f823bf4d47d6a243fc929799d9013b81d85b4\": rpc error: code = NotFound desc = could not find container \"f24461bf6874913fb3e9dd669d8f823bf4d47d6a243fc929799d9013b81d85b4\": container with ID starting with f24461bf6874913fb3e9dd669d8f823bf4d47d6a243fc929799d9013b81d85b4 not found: ID does not exist"
Sep 10 21:25:43.418568 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.418545    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-workload-certs\") pod \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") "
Sep 10 21:25:43.418674 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.418596    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istio-token\") pod \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") "
Sep 10 21:25:43.418674 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.418612    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istio-podinfo\") pod \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") "
Sep 10 21:25:43.418674 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.418649    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istio-data\") pod \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") "
Sep 10 21:25:43.418846 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.418677    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istio-envoy\") pod \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") "
Sep 10 21:25:43.418846 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.418728    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-workload-socket\") pod \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") "
Sep 10 21:25:43.418846 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.418753    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-credential-socket\") pod \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") "
Sep 10 21:25:43.418846 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.418786    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istiod-ca-cert\") pod \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") "
Sep 10 21:25:43.418846 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.418807    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sqwc\" (UniqueName: \"kubernetes.io/projected/aa363798-f3d2-4cef-8dfb-af5e0043eee8-kube-api-access-8sqwc\") pod \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\" (UID: \"aa363798-f3d2-4cef-8dfb-af5e0043eee8\") "
Sep 10 21:25:43.418846 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.418819    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aa363798-f3d2-4cef-8dfb-af5e0043eee8" (UID: "aa363798-f3d2-4cef-8dfb-af5e0043eee8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:25:43.419189 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.418969    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aa363798-f3d2-4cef-8dfb-af5e0043eee8" (UID: "aa363798-f3d2-4cef-8dfb-af5e0043eee8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:25:43.419189 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.419030    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istio-data" (OuterVolumeSpecName: "istio-data") pod "aa363798-f3d2-4cef-8dfb-af5e0043eee8" (UID: "aa363798-f3d2-4cef-8dfb-af5e0043eee8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:25:43.419189 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.419076    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aa363798-f3d2-4cef-8dfb-af5e0043eee8" (UID: "aa363798-f3d2-4cef-8dfb-af5e0043eee8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:25:43.419531 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.419244    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aa363798-f3d2-4cef-8dfb-af5e0043eee8" (UID: "aa363798-f3d2-4cef-8dfb-af5e0043eee8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:25:43.419531 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.419270    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-workload-certs\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:25:43.419531 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.419290    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istio-data\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:25:43.419531 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.419301    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-workload-socket\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:25:43.419531 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.419317    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-credential-socket\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:25:43.420999 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.420955    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istio-token" (OuterVolumeSpecName: "istio-token") pod "aa363798-f3d2-4cef-8dfb-af5e0043eee8" (UID: "aa363798-f3d2-4cef-8dfb-af5e0043eee8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:25:43.421177 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.421152    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aa363798-f3d2-4cef-8dfb-af5e0043eee8" (UID: "aa363798-f3d2-4cef-8dfb-af5e0043eee8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:25:43.421177 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.421169    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa363798-f3d2-4cef-8dfb-af5e0043eee8-kube-api-access-8sqwc" (OuterVolumeSpecName: "kube-api-access-8sqwc") pod "aa363798-f3d2-4cef-8dfb-af5e0043eee8" (UID: "aa363798-f3d2-4cef-8dfb-af5e0043eee8"). InnerVolumeSpecName "kube-api-access-8sqwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:25:43.421277 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.421236    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aa363798-f3d2-4cef-8dfb-af5e0043eee8" (UID: "aa363798-f3d2-4cef-8dfb-af5e0043eee8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:25:43.520058 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.520037    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istio-envoy\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:25:43.520058 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.520055    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istiod-ca-cert\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:25:43.520183 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.520064    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sqwc\" (UniqueName: \"kubernetes.io/projected/aa363798-f3d2-4cef-8dfb-af5e0043eee8-kube-api-access-8sqwc\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:25:43.520183 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.520072    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istio-token\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:25:43.520183 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.520080    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa363798-f3d2-4cef-8dfb-af5e0043eee8-istio-podinfo\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:25:43.665782 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.665725    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"]
Sep 10 21:25:43.674691 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:43.669010    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1ca93e-openshift-default-8d595976-29kkk"]
Sep 10 21:25:45.438172 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:45.438139    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa363798-f3d2-4cef-8dfb-af5e0043eee8" path="/var/lib/kubelet/pods/aa363798-f3d2-4cef-8dfb-af5e0043eee8/volumes"
Sep 10 21:25:54.835621 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:25:54.835583    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:26:03.098942 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:03.098893    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"]
Sep 10 21:26:07.335897 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:07.335861    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:26:18.118688 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.118632    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457" podUID="f2b2bdb5-1498-4a0b-93ad-824ffdef01ef" containerName="main" containerID="cri-o://e7883b187a405792c6abb4d53dac1d69953610ac6e3b6f9803427390b5605d80" gracePeriod=45
Sep 10 21:26:18.376799 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.376736    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:26:18.501862 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.501828    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-home\") pod \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\" (UID: \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\") "
Sep 10 21:26:18.501862 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.501862    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djbj2\" (UniqueName: \"kubernetes.io/projected/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-kube-api-access-djbj2\") pod \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\" (UID: \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\") "
Sep 10 21:26:18.502122 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.501893    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-tmp-dir\") pod \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\" (UID: \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\") "
Sep 10 21:26:18.502122 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.501961    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-model-cache\") pod \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\" (UID: \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\") "
Sep 10 21:26:18.502122 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.502011    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-dshm\") pod \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\" (UID: \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\") "
Sep 10 21:26:18.502122 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.502062    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-tls-certs\") pod \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\" (UID: \"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef\") "
Sep 10 21:26:18.502342 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.502151    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-home" (OuterVolumeSpecName: "home") pod "f2b2bdb5-1498-4a0b-93ad-824ffdef01ef" (UID: "f2b2bdb5-1498-4a0b-93ad-824ffdef01ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:26:18.502342 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.502233    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2b2bdb5-1498-4a0b-93ad-824ffdef01ef" (UID: "f2b2bdb5-1498-4a0b-93ad-824ffdef01ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:26:18.502342 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.502251    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2b2bdb5-1498-4a0b-93ad-824ffdef01ef" (UID: "f2b2bdb5-1498-4a0b-93ad-824ffdef01ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:26:18.502492 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.502415    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-home\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:26:18.502492 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.502465    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-tmp-dir\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:26:18.502492 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.502482    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-model-cache\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:26:18.504318 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.504297    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2b2bdb5-1498-4a0b-93ad-824ffdef01ef" (UID: "f2b2bdb5-1498-4a0b-93ad-824ffdef01ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:26:18.504409 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.504338    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-dshm" (OuterVolumeSpecName: "dshm") pod "f2b2bdb5-1498-4a0b-93ad-824ffdef01ef" (UID: "f2b2bdb5-1498-4a0b-93ad-824ffdef01ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:26:18.504464 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.504415    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-kube-api-access-djbj2" (OuterVolumeSpecName: "kube-api-access-djbj2") pod "f2b2bdb5-1498-4a0b-93ad-824ffdef01ef" (UID: "f2b2bdb5-1498-4a0b-93ad-824ffdef01ef"). InnerVolumeSpecName "kube-api-access-djbj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:26:18.525406 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.525377    2572 generic.go:356] "Generic (PLEG): container finished" podID="f2b2bdb5-1498-4a0b-93ad-824ffdef01ef" containerID="e7883b187a405792c6abb4d53dac1d69953610ac6e3b6f9803427390b5605d80" exitCode=0
Sep 10 21:26:18.525521 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.525417    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"
Sep 10 21:26:18.525521 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.525460    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457" event={"ID":"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef","Type":"ContainerDied","Data":"e7883b187a405792c6abb4d53dac1d69953610ac6e3b6f9803427390b5605d80"}
Sep 10 21:26:18.525521 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.525505    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457" event={"ID":"f2b2bdb5-1498-4a0b-93ad-824ffdef01ef","Type":"ContainerDied","Data":"3b7cf2aeb9a988fbe6005283cf13036581e7fe432c5bff14fce69a6b3a851b04"}
Sep 10 21:26:18.525681 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.525526    2572 scope.go:117] "RemoveContainer" containerID="e7883b187a405792c6abb4d53dac1d69953610ac6e3b6f9803427390b5605d80"
Sep 10 21:26:18.534147 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.534128    2572 scope.go:117] "RemoveContainer" containerID="e7883b187a405792c6abb4d53dac1d69953610ac6e3b6f9803427390b5605d80"
Sep 10 21:26:18.534392 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:26:18.534376    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7883b187a405792c6abb4d53dac1d69953610ac6e3b6f9803427390b5605d80\": container with ID starting with e7883b187a405792c6abb4d53dac1d69953610ac6e3b6f9803427390b5605d80 not found: ID does not exist" containerID="e7883b187a405792c6abb4d53dac1d69953610ac6e3b6f9803427390b5605d80"
Sep 10 21:26:18.534448 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.534399    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7883b187a405792c6abb4d53dac1d69953610ac6e3b6f9803427390b5605d80"} err="failed to get container status \"e7883b187a405792c6abb4d53dac1d69953610ac6e3b6f9803427390b5605d80\": rpc error: code = NotFound desc = could not find container \"e7883b187a405792c6abb4d53dac1d69953610ac6e3b6f9803427390b5605d80\": container with ID starting with e7883b187a405792c6abb4d53dac1d69953610ac6e3b6f9803427390b5605d80 not found: ID does not exist"
Sep 10 21:26:18.547131 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.547110    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"]
Sep 10 21:26:18.552430 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.552409    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-24b606/ipp-route-24b606-kserve-6c5f8c7cc9-xs457"]
Sep 10 21:26:18.603251 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.603225    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-tls-certs\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:26:18.603251 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.603247    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djbj2\" (UniqueName: \"kubernetes.io/projected/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-kube-api-access-djbj2\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:26:18.603376 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:18.603263    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef-dshm\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:26:19.439105 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:19.439060    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2b2bdb5-1498-4a0b-93ad-824ffdef01ef" path="/var/lib/kubelet/pods/f2b2bdb5-1498-4a0b-93ad-824ffdef01ef/volumes"
Sep 10 21:26:48.493554 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:48.493454    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"]
Sep 10 21:26:48.494171 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:48.493693    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c" podUID="bad84f81-90e2-4743-8a48-5284f8aa5e9e" containerName="istio-proxy" containerID="cri-o://d5695ca56ac9e4aa29154f210f7455a154b80dbae27232911acbe78d43cdb797" gracePeriod=30
Sep 10 21:26:53.703311 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.703245    2572 generic.go:356] "Generic (PLEG): container finished" podID="bad84f81-90e2-4743-8a48-5284f8aa5e9e" containerID="d5695ca56ac9e4aa29154f210f7455a154b80dbae27232911acbe78d43cdb797" exitCode=0
Sep 10 21:26:53.703311 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.703286    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c" event={"ID":"bad84f81-90e2-4743-8a48-5284f8aa5e9e","Type":"ContainerDied","Data":"d5695ca56ac9e4aa29154f210f7455a154b80dbae27232911acbe78d43cdb797"}
Sep 10 21:26:53.703706 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.703339    2572 scope.go:117] "RemoveContainer" containerID="3e3e659ba4fdfe15686f0293c17ab2d4d8c6a42d925fdfc4d9d9ceb1657a76fe"
Sep 10 21:26:53.750639 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.750619    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:26:53.885445 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.885415    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istiod-ca-cert\") pod \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") "
Sep 10 21:26:53.885552 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.885457    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-credential-socket\") pod \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") "
Sep 10 21:26:53.885552 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.885489    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-workload-socket\") pod \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") "
Sep 10 21:26:53.885552 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.885522    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istio-envoy\") pod \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") "
Sep 10 21:26:53.885552 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.885544    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istio-data\") pod \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") "
Sep 10 21:26:53.885750 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.885562    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-workload-certs\") pod \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") "
Sep 10 21:26:53.885750 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.885594    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istio-podinfo\") pod \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") "
Sep 10 21:26:53.885750 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.885630    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istio-token\") pod \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") "
Sep 10 21:26:53.885750 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.885670    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwqr2\" (UniqueName: \"kubernetes.io/projected/bad84f81-90e2-4743-8a48-5284f8aa5e9e-kube-api-access-mwqr2\") pod \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\" (UID: \"bad84f81-90e2-4743-8a48-5284f8aa5e9e\") "
Sep 10 21:26:53.885958 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.885775    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bad84f81-90e2-4743-8a48-5284f8aa5e9e" (UID: "bad84f81-90e2-4743-8a48-5284f8aa5e9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:26:53.885958 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.885797    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bad84f81-90e2-4743-8a48-5284f8aa5e9e" (UID: "bad84f81-90e2-4743-8a48-5284f8aa5e9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:26:53.885958 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.885925    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istiod-ca-cert\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:26:53.885958 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.885944    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-workload-socket\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:26:53.885958 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.885940    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bad84f81-90e2-4743-8a48-5284f8aa5e9e" (UID: "bad84f81-90e2-4743-8a48-5284f8aa5e9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:26:53.886187 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.886006    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istio-data" (OuterVolumeSpecName: "istio-data") pod "bad84f81-90e2-4743-8a48-5284f8aa5e9e" (UID: "bad84f81-90e2-4743-8a48-5284f8aa5e9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:26:53.886187 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.886042    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bad84f81-90e2-4743-8a48-5284f8aa5e9e" (UID: "bad84f81-90e2-4743-8a48-5284f8aa5e9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:26:53.888042 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.888013    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bad84f81-90e2-4743-8a48-5284f8aa5e9e" (UID: "bad84f81-90e2-4743-8a48-5284f8aa5e9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:26:53.888155 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.888106    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istio-token" (OuterVolumeSpecName: "istio-token") pod "bad84f81-90e2-4743-8a48-5284f8aa5e9e" (UID: "bad84f81-90e2-4743-8a48-5284f8aa5e9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:26:53.888155 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.888117    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bad84f81-90e2-4743-8a48-5284f8aa5e9e" (UID: "bad84f81-90e2-4743-8a48-5284f8aa5e9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:26:53.888272 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.888179    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad84f81-90e2-4743-8a48-5284f8aa5e9e-kube-api-access-mwqr2" (OuterVolumeSpecName: "kube-api-access-mwqr2") pod "bad84f81-90e2-4743-8a48-5284f8aa5e9e" (UID: "bad84f81-90e2-4743-8a48-5284f8aa5e9e"). InnerVolumeSpecName "kube-api-access-mwqr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:26:53.987335 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.987283    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istio-envoy\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:26:53.987335 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.987302    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istio-data\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:26:53.987335 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.987311    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-workload-certs\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:26:53.987335 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.987319    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istio-podinfo\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:26:53.987335 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.987327    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad84f81-90e2-4743-8a48-5284f8aa5e9e-istio-token\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:26:53.987549 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.987335    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwqr2\" (UniqueName: \"kubernetes.io/projected/bad84f81-90e2-4743-8a48-5284f8aa5e9e-kube-api-access-mwqr2\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:26:53.987549 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:53.987351    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad84f81-90e2-4743-8a48-5284f8aa5e9e-credential-socket\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:26:54.710813 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:54.710772    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c" event={"ID":"bad84f81-90e2-4743-8a48-5284f8aa5e9e","Type":"ContainerDied","Data":"12b022620970cd5977e63653523952585d2d04e99699038af4038d65cb33e53f"}
Sep 10 21:26:54.710813 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:54.710791    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"
Sep 10 21:26:54.711426 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:54.710847    2572 scope.go:117] "RemoveContainer" containerID="d5695ca56ac9e4aa29154f210f7455a154b80dbae27232911acbe78d43cdb797"
Sep 10 21:26:54.733360 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:54.733332    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"]
Sep 10 21:26:54.738123 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:54.738098    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-24b606-openshift-default-8bd8dd95d-ltf8c"]
Sep 10 21:26:55.437883 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:26:55.437849    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bad84f81-90e2-4743-8a48-5284f8aa5e9e" path="/var/lib/kubelet/pods/bad84f81-90e2-4743-8a48-5284f8aa5e9e/volumes"
Sep 10 21:27:00.053138 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:00.053102    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf"]
Sep 10 21:27:00.053545 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:00.053304    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf" podUID="71ebf073-e170-442f-8be4-0f47b3cb2e42" containerName="payload-pre-processing" containerID="cri-o://eed0680b2945bec4e70e009e006456bfc6a0d1c4de5ed08773f8e776e1751433" gracePeriod=30
Sep 10 21:27:00.308946 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:00.308886    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf"
Sep 10 21:27:00.436327 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:00.436299    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ebf073-e170-442f-8be4-0f47b3cb2e42-plugins-config-volume\") pod \"71ebf073-e170-442f-8be4-0f47b3cb2e42\" (UID: \"71ebf073-e170-442f-8be4-0f47b3cb2e42\") "
Sep 10 21:27:00.436457 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:00.436359    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4p7r\" (UniqueName: \"kubernetes.io/projected/71ebf073-e170-442f-8be4-0f47b3cb2e42-kube-api-access-q4p7r\") pod \"71ebf073-e170-442f-8be4-0f47b3cb2e42\" (UID: \"71ebf073-e170-442f-8be4-0f47b3cb2e42\") "
Sep 10 21:27:00.436653 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:00.436629    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ebf073-e170-442f-8be4-0f47b3cb2e42-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "71ebf073-e170-442f-8be4-0f47b3cb2e42" (UID: "71ebf073-e170-442f-8be4-0f47b3cb2e42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:27:00.438559 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:00.438533    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ebf073-e170-442f-8be4-0f47b3cb2e42-kube-api-access-q4p7r" (OuterVolumeSpecName: "kube-api-access-q4p7r") pod "71ebf073-e170-442f-8be4-0f47b3cb2e42" (UID: "71ebf073-e170-442f-8be4-0f47b3cb2e42"). InnerVolumeSpecName "kube-api-access-q4p7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:27:00.537474 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:00.537441    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4p7r\" (UniqueName: \"kubernetes.io/projected/71ebf073-e170-442f-8be4-0f47b3cb2e42-kube-api-access-q4p7r\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:27:00.537474 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:00.537470    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ebf073-e170-442f-8be4-0f47b3cb2e42-plugins-config-volume\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:27:00.740887 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:00.740822    2572 generic.go:356] "Generic (PLEG): container finished" podID="71ebf073-e170-442f-8be4-0f47b3cb2e42" containerID="eed0680b2945bec4e70e009e006456bfc6a0d1c4de5ed08773f8e776e1751433" exitCode=0
Sep 10 21:27:00.740887 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:00.740865    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf"
Sep 10 21:27:00.741065 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:00.740891    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf" event={"ID":"71ebf073-e170-442f-8be4-0f47b3cb2e42","Type":"ContainerDied","Data":"eed0680b2945bec4e70e009e006456bfc6a0d1c4de5ed08773f8e776e1751433"}
Sep 10 21:27:00.741065 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:00.740932    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf" event={"ID":"71ebf073-e170-442f-8be4-0f47b3cb2e42","Type":"ContainerDied","Data":"acc92ed8ce1209f7df536323efbd4c83196300b939e1cf4ab3c841a4a1837115"}
Sep 10 21:27:00.741065 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:00.740954    2572 scope.go:117] "RemoveContainer" containerID="eed0680b2945bec4e70e009e006456bfc6a0d1c4de5ed08773f8e776e1751433"
Sep 10 21:27:00.759709 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:00.759683    2572 scope.go:117] "RemoveContainer" containerID="eed0680b2945bec4e70e009e006456bfc6a0d1c4de5ed08773f8e776e1751433"
Sep 10 21:27:00.760253 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:27:00.760228    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed0680b2945bec4e70e009e006456bfc6a0d1c4de5ed08773f8e776e1751433\": container with ID starting with eed0680b2945bec4e70e009e006456bfc6a0d1c4de5ed08773f8e776e1751433 not found: ID does not exist" containerID="eed0680b2945bec4e70e009e006456bfc6a0d1c4de5ed08773f8e776e1751433"
Sep 10 21:27:00.760344 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:00.760264    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed0680b2945bec4e70e009e006456bfc6a0d1c4de5ed08773f8e776e1751433"} err="failed to get container status \"eed0680b2945bec4e70e009e006456bfc6a0d1c4de5ed08773f8e776e1751433\": rpc error: code = NotFound desc = could not find container \"eed0680b2945bec4e70e009e006456bfc6a0d1c4de5ed08773f8e776e1751433\": container with ID starting with eed0680b2945bec4e70e009e006456bfc6a0d1c4de5ed08773f8e776e1751433 not found: ID does not exist"
Sep 10 21:27:00.773965 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:00.773939    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf"]
Sep 10 21:27:00.779618 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:00.779598    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9f1286-6dd9cc558b-h6srf"]
Sep 10 21:27:01.437418 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:01.437392    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71ebf073-e170-442f-8be4-0f47b3cb2e42" path="/var/lib/kubelet/pods/71ebf073-e170-442f-8be4-0f47b3cb2e42/volumes"
Sep 10 21:27:03.723576 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:03.723542    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:27:18.991080 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:18.991045    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd"]
Sep 10 21:27:18.991630 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:18.991245    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd" podUID="a87ef12f-1556-4db9-9c07-5c655e6dbd80" containerName="payload-processing" containerID="cri-o://b827cf4a110abb139d3bf533e75f5970645e24c466e861b7b6d91f3bcc097d89" gracePeriod=30
Sep 10 21:27:18.999341 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:18.999315    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw"]
Sep 10 21:27:18.999522 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:18.999500    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw" podUID="2c984132-b24c-46f5-8808-4557e7cd8bc7" containerName="payload-pre-processing" containerID="cri-o://5e97d3f7e1e7544d6a18f5ae82f688c337e3aad49775b0a88a0ddfa7166eba06" gracePeriod=30
Sep 10 21:27:19.265490 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.265462    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd"
Sep 10 21:27:19.268051 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.268033    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw"
Sep 10 21:27:19.273717 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.273697    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crm9b\" (UniqueName: \"kubernetes.io/projected/a87ef12f-1556-4db9-9c07-5c655e6dbd80-kube-api-access-crm9b\") pod \"a87ef12f-1556-4db9-9c07-5c655e6dbd80\" (UID: \"a87ef12f-1556-4db9-9c07-5c655e6dbd80\") "
Sep 10 21:27:19.273786 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.273744    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bj8v5\" (UniqueName: \"kubernetes.io/projected/2c984132-b24c-46f5-8808-4557e7cd8bc7-kube-api-access-bj8v5\") pod \"2c984132-b24c-46f5-8808-4557e7cd8bc7\" (UID: \"2c984132-b24c-46f5-8808-4557e7cd8bc7\") "
Sep 10 21:27:19.273829 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.273819    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87ef12f-1556-4db9-9c07-5c655e6dbd80-plugins-config-volume\") pod \"a87ef12f-1556-4db9-9c07-5c655e6dbd80\" (UID: \"a87ef12f-1556-4db9-9c07-5c655e6dbd80\") "
Sep 10 21:27:19.273865 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.273856    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c984132-b24c-46f5-8808-4557e7cd8bc7-plugins-config-volume\") pod \"2c984132-b24c-46f5-8808-4557e7cd8bc7\" (UID: \"2c984132-b24c-46f5-8808-4557e7cd8bc7\") "
Sep 10 21:27:19.274256 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.274227    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a87ef12f-1556-4db9-9c07-5c655e6dbd80-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a87ef12f-1556-4db9-9c07-5c655e6dbd80" (UID: "a87ef12f-1556-4db9-9c07-5c655e6dbd80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:27:19.274349 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.274261    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c984132-b24c-46f5-8808-4557e7cd8bc7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c984132-b24c-46f5-8808-4557e7cd8bc7" (UID: "2c984132-b24c-46f5-8808-4557e7cd8bc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:27:19.276475 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.276450    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a87ef12f-1556-4db9-9c07-5c655e6dbd80-kube-api-access-crm9b" (OuterVolumeSpecName: "kube-api-access-crm9b") pod "a87ef12f-1556-4db9-9c07-5c655e6dbd80" (UID: "a87ef12f-1556-4db9-9c07-5c655e6dbd80"). InnerVolumeSpecName "kube-api-access-crm9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:27:19.276475 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.276462    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c984132-b24c-46f5-8808-4557e7cd8bc7-kube-api-access-bj8v5" (OuterVolumeSpecName: "kube-api-access-bj8v5") pod "2c984132-b24c-46f5-8808-4557e7cd8bc7" (UID: "2c984132-b24c-46f5-8808-4557e7cd8bc7"). InnerVolumeSpecName "kube-api-access-bj8v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:27:19.374878 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.374853    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crm9b\" (UniqueName: \"kubernetes.io/projected/a87ef12f-1556-4db9-9c07-5c655e6dbd80-kube-api-access-crm9b\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:27:19.374878 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.374876    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj8v5\" (UniqueName: \"kubernetes.io/projected/2c984132-b24c-46f5-8808-4557e7cd8bc7-kube-api-access-bj8v5\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:27:19.375082 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.374885    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87ef12f-1556-4db9-9c07-5c655e6dbd80-plugins-config-volume\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:27:19.375082 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.374895    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c984132-b24c-46f5-8808-4557e7cd8bc7-plugins-config-volume\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:27:19.843454 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.843415    2572 generic.go:356] "Generic (PLEG): container finished" podID="2c984132-b24c-46f5-8808-4557e7cd8bc7" containerID="5e97d3f7e1e7544d6a18f5ae82f688c337e3aad49775b0a88a0ddfa7166eba06" exitCode=0
Sep 10 21:27:19.843629 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.843516    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw"
Sep 10 21:27:19.843629 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.843521    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw" event={"ID":"2c984132-b24c-46f5-8808-4557e7cd8bc7","Type":"ContainerDied","Data":"5e97d3f7e1e7544d6a18f5ae82f688c337e3aad49775b0a88a0ddfa7166eba06"}
Sep 10 21:27:19.843810 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.843714    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw" event={"ID":"2c984132-b24c-46f5-8808-4557e7cd8bc7","Type":"ContainerDied","Data":"cbcc983d622b864b8b6d26bfbdd54a43421562df7059b58e367dbf83d330c2af"}
Sep 10 21:27:19.843810 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.843745    2572 scope.go:117] "RemoveContainer" containerID="5e97d3f7e1e7544d6a18f5ae82f688c337e3aad49775b0a88a0ddfa7166eba06"
Sep 10 21:27:19.845214 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.845189    2572 generic.go:356] "Generic (PLEG): container finished" podID="a87ef12f-1556-4db9-9c07-5c655e6dbd80" containerID="b827cf4a110abb139d3bf533e75f5970645e24c466e861b7b6d91f3bcc097d89" exitCode=0
Sep 10 21:27:19.845358 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.845227    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd" event={"ID":"a87ef12f-1556-4db9-9c07-5c655e6dbd80","Type":"ContainerDied","Data":"b827cf4a110abb139d3bf533e75f5970645e24c466e861b7b6d91f3bcc097d89"}
Sep 10 21:27:19.845358 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.845237    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd"
Sep 10 21:27:19.845358 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.845263    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd" event={"ID":"a87ef12f-1556-4db9-9c07-5c655e6dbd80","Type":"ContainerDied","Data":"b1edd721abeb751077547d5b007fabf7bd0e1b8c285f2b43963a6c44dbbb6427"}
Sep 10 21:27:19.853268 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.853252    2572 scope.go:117] "RemoveContainer" containerID="5e97d3f7e1e7544d6a18f5ae82f688c337e3aad49775b0a88a0ddfa7166eba06"
Sep 10 21:27:19.853519 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:27:19.853500    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e97d3f7e1e7544d6a18f5ae82f688c337e3aad49775b0a88a0ddfa7166eba06\": container with ID starting with 5e97d3f7e1e7544d6a18f5ae82f688c337e3aad49775b0a88a0ddfa7166eba06 not found: ID does not exist" containerID="5e97d3f7e1e7544d6a18f5ae82f688c337e3aad49775b0a88a0ddfa7166eba06"
Sep 10 21:27:19.853601 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.853528    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e97d3f7e1e7544d6a18f5ae82f688c337e3aad49775b0a88a0ddfa7166eba06"} err="failed to get container status \"5e97d3f7e1e7544d6a18f5ae82f688c337e3aad49775b0a88a0ddfa7166eba06\": rpc error: code = NotFound desc = could not find container \"5e97d3f7e1e7544d6a18f5ae82f688c337e3aad49775b0a88a0ddfa7166eba06\": container with ID starting with 5e97d3f7e1e7544d6a18f5ae82f688c337e3aad49775b0a88a0ddfa7166eba06 not found: ID does not exist"
Sep 10 21:27:19.853601 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.853551    2572 scope.go:117] "RemoveContainer" containerID="b827cf4a110abb139d3bf533e75f5970645e24c466e861b7b6d91f3bcc097d89"
Sep 10 21:27:19.861000 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.860959    2572 scope.go:117] "RemoveContainer" containerID="b827cf4a110abb139d3bf533e75f5970645e24c466e861b7b6d91f3bcc097d89"
Sep 10 21:27:19.861258 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:27:19.861238    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b827cf4a110abb139d3bf533e75f5970645e24c466e861b7b6d91f3bcc097d89\": container with ID starting with b827cf4a110abb139d3bf533e75f5970645e24c466e861b7b6d91f3bcc097d89 not found: ID does not exist" containerID="b827cf4a110abb139d3bf533e75f5970645e24c466e861b7b6d91f3bcc097d89"
Sep 10 21:27:19.861323 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.861268    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b827cf4a110abb139d3bf533e75f5970645e24c466e861b7b6d91f3bcc097d89"} err="failed to get container status \"b827cf4a110abb139d3bf533e75f5970645e24c466e861b7b6d91f3bcc097d89\": rpc error: code = NotFound desc = could not find container \"b827cf4a110abb139d3bf533e75f5970645e24c466e861b7b6d91f3bcc097d89\": container with ID starting with b827cf4a110abb139d3bf533e75f5970645e24c466e861b7b6d91f3bcc097d89 not found: ID does not exist"
Sep 10 21:27:19.871625 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.871602    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw"]
Sep 10 21:27:19.882871 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.882849    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-af7150-78c6789dc8-6v7nw"]
Sep 10 21:27:19.913327 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.913308    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd"]
Sep 10 21:27:19.915567 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:19.915548    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-af7150-6fb7bff97f-6jdgd"]
Sep 10 21:27:21.438376 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:21.438338    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c984132-b24c-46f5-8808-4557e7cd8bc7" path="/var/lib/kubelet/pods/2c984132-b24c-46f5-8808-4557e7cd8bc7/volumes"
Sep 10 21:27:21.438785 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:21.438704    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a87ef12f-1556-4db9-9c07-5c655e6dbd80" path="/var/lib/kubelet/pods/a87ef12f-1556-4db9-9c07-5c655e6dbd80/volumes"
Sep 10 21:27:22.334679 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:22.334645    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:27:28.110173 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110137    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7810bd-5b6c7"]
Sep 10 21:27:28.110670 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110503    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa363798-f3d2-4cef-8dfb-af5e0043eee8" containerName="istio-proxy"
Sep 10 21:27:28.110670 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110518    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa363798-f3d2-4cef-8dfb-af5e0043eee8" containerName="istio-proxy"
Sep 10 21:27:28.110670 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110534    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa363798-f3d2-4cef-8dfb-af5e0043eee8" containerName="istio-proxy"
Sep 10 21:27:28.110670 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110540    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa363798-f3d2-4cef-8dfb-af5e0043eee8" containerName="istio-proxy"
Sep 10 21:27:28.110670 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110549    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87ef12f-1556-4db9-9c07-5c655e6dbd80" containerName="payload-processing"
Sep 10 21:27:28.110670 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110555    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87ef12f-1556-4db9-9c07-5c655e6dbd80" containerName="payload-processing"
Sep 10 21:27:28.110670 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110564    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bad84f81-90e2-4743-8a48-5284f8aa5e9e" containerName="istio-proxy"
Sep 10 21:27:28.110670 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110570    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bad84f81-90e2-4743-8a48-5284f8aa5e9e" containerName="istio-proxy"
Sep 10 21:27:28.110670 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110575    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b2bdb5-1498-4a0b-93ad-824ffdef01ef" containerName="main"
Sep 10 21:27:28.110670 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110581    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b2bdb5-1498-4a0b-93ad-824ffdef01ef" containerName="main"
Sep 10 21:27:28.110670 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110589    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab8e6952-6f9d-4c24-9328-1ce3d801ae14" containerName="payload-processing"
Sep 10 21:27:28.110670 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110595    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab8e6952-6f9d-4c24-9328-1ce3d801ae14" containerName="payload-processing"
Sep 10 21:27:28.110670 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110602    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bad84f81-90e2-4743-8a48-5284f8aa5e9e" containerName="istio-proxy"
Sep 10 21:27:28.110670 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110607    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bad84f81-90e2-4743-8a48-5284f8aa5e9e" containerName="istio-proxy"
Sep 10 21:27:28.110670 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110621    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c984132-b24c-46f5-8808-4557e7cd8bc7" containerName="payload-pre-processing"
Sep 10 21:27:28.110670 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110625    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c984132-b24c-46f5-8808-4557e7cd8bc7" containerName="payload-pre-processing"
Sep 10 21:27:28.110670 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110631    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71ebf073-e170-442f-8be4-0f47b3cb2e42" containerName="payload-pre-processing"
Sep 10 21:27:28.110670 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110636    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="71ebf073-e170-442f-8be4-0f47b3cb2e42" containerName="payload-pre-processing"
Sep 10 21:27:28.111266 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110690    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c984132-b24c-46f5-8808-4557e7cd8bc7" containerName="payload-pre-processing"
Sep 10 21:27:28.111266 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110701    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2b2bdb5-1498-4a0b-93ad-824ffdef01ef" containerName="main"
Sep 10 21:27:28.111266 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110707    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa363798-f3d2-4cef-8dfb-af5e0043eee8" containerName="istio-proxy"
Sep 10 21:27:28.111266 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110715    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87ef12f-1556-4db9-9c07-5c655e6dbd80" containerName="payload-processing"
Sep 10 21:27:28.111266 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110721    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bad84f81-90e2-4743-8a48-5284f8aa5e9e" containerName="istio-proxy"
Sep 10 21:27:28.111266 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110726    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa363798-f3d2-4cef-8dfb-af5e0043eee8" containerName="istio-proxy"
Sep 10 21:27:28.111266 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110733    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bad84f81-90e2-4743-8a48-5284f8aa5e9e" containerName="istio-proxy"
Sep 10 21:27:28.111266 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110741    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="71ebf073-e170-442f-8be4-0f47b3cb2e42" containerName="payload-pre-processing"
Sep 10 21:27:28.111266 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.110748    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab8e6952-6f9d-4c24-9328-1ce3d801ae14" containerName="payload-processing"
Sep 10 21:27:28.113714 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.113695    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7810bd-5b6c7"
Sep 10 21:27:28.116515 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.116491    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5c7x\""
Sep 10 21:27:28.117465 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.117447    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:27:28.128178 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.128147    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7810bd-5b6c7"]
Sep 10 21:27:28.139823 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.139798    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eaf689d-06a9-4ee1-aa60-4f11185b2b98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7810bd-5b6c7\" (UID: \"1eaf689d-06a9-4ee1-aa60-4f11185b2b98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7810bd-5b6c7"
Sep 10 21:27:28.240617 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.240590    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eaf689d-06a9-4ee1-aa60-4f11185b2b98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7810bd-5b6c7\" (UID: \"1eaf689d-06a9-4ee1-aa60-4f11185b2b98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7810bd-5b6c7"
Sep 10 21:27:28.241204 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.241183    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eaf689d-06a9-4ee1-aa60-4f11185b2b98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7810bd-5b6c7\" (UID: \"1eaf689d-06a9-4ee1-aa60-4f11185b2b98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7810bd-5b6c7"
Sep 10 21:27:28.424920 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.424863    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7810bd-5b6c7"
Sep 10 21:27:28.718735 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.718706    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7810bd-5b6c7"]
Sep 10 21:27:28.719728 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:27:28.719691    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eaf689d_06a9_4ee1_aa60_4f11185b2b98.slice/crio-793e139ad0c6ea27220e973391bed5190a29ef1162c56c3c7a31cb0db811a541 WatchSource:0}: Error finding container 793e139ad0c6ea27220e973391bed5190a29ef1162c56c3c7a31cb0db811a541: Status 404 returned error can't find the container with id 793e139ad0c6ea27220e973391bed5190a29ef1162c56c3c7a31cb0db811a541
Sep 10 21:27:28.889037 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:28.889013    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7810bd-5b6c7" event={"ID":"1eaf689d-06a9-4ee1-aa60-4f11185b2b98","Type":"ContainerStarted","Data":"793e139ad0c6ea27220e973391bed5190a29ef1162c56c3c7a31cb0db811a541"}
Sep 10 21:27:29.894751 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:29.894714    2572 generic.go:356] "Generic (PLEG): container finished" podID="1eaf689d-06a9-4ee1-aa60-4f11185b2b98" containerID="37d2135f61a3cb641c62af32840fb18e36aa53db1387a2d890b34dfe2e615d0b" exitCode=0
Sep 10 21:27:29.895162 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:29.894803    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7810bd-5b6c7" event={"ID":"1eaf689d-06a9-4ee1-aa60-4f11185b2b98","Type":"ContainerDied","Data":"37d2135f61a3cb641c62af32840fb18e36aa53db1387a2d890b34dfe2e615d0b"}
Sep 10 21:27:31.029776 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:31.029754    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7810bd-5b6c7"
Sep 10 21:27:31.064626 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:31.064601    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eaf689d-06a9-4ee1-aa60-4f11185b2b98-maas-api-service-ca\") pod \"1eaf689d-06a9-4ee1-aa60-4f11185b2b98\" (UID: \"1eaf689d-06a9-4ee1-aa60-4f11185b2b98\") "
Sep 10 21:27:31.064908 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:31.064887    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eaf689d-06a9-4ee1-aa60-4f11185b2b98-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1eaf689d-06a9-4ee1-aa60-4f11185b2b98" (UID: "1eaf689d-06a9-4ee1-aa60-4f11185b2b98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:27:31.165828 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:31.165776    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eaf689d-06a9-4ee1-aa60-4f11185b2b98-maas-api-service-ca\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:27:31.905340 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:31.905311    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7810bd-5b6c7"
Sep 10 21:27:31.905534 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:31.905308    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7810bd-5b6c7" event={"ID":"1eaf689d-06a9-4ee1-aa60-4f11185b2b98","Type":"ContainerDied","Data":"793e139ad0c6ea27220e973391bed5190a29ef1162c56c3c7a31cb0db811a541"}
Sep 10 21:27:31.905534 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:31.905483    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="793e139ad0c6ea27220e973391bed5190a29ef1162c56c3c7a31cb0db811a541"
Sep 10 21:27:39.310063 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.310028    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f"]
Sep 10 21:27:39.310475 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.310263    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f" podUID="9e97ece3-54ca-4af5-baa3-7a1472975414" containerName="payload-processing" containerID="cri-o://604c6e202566496e6240cad3776b6aec7c916e939ecd207673e8f83e7345e5c4" gracePeriod=30
Sep 10 21:27:39.328524 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.328498    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd"]
Sep 10 21:27:39.328694 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.328674    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd" podUID="766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c" containerName="payload-pre-processing" containerID="cri-o://1d225bf84adac9024fdcc477dca362a467fa0423d6cf46db326c1989c6940cd0" gracePeriod=30
Sep 10 21:27:39.596195 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.596170    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f"
Sep 10 21:27:39.599503 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.599483    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd"
Sep 10 21:27:39.628997 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.628962    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c-plugins-config-volume\") pod \"766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c\" (UID: \"766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c\") "
Sep 10 21:27:39.629088 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.629017    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e97ece3-54ca-4af5-baa3-7a1472975414-plugins-config-volume\") pod \"9e97ece3-54ca-4af5-baa3-7a1472975414\" (UID: \"9e97ece3-54ca-4af5-baa3-7a1472975414\") "
Sep 10 21:27:39.629088 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.629066    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sx27\" (UniqueName: \"kubernetes.io/projected/9e97ece3-54ca-4af5-baa3-7a1472975414-kube-api-access-5sx27\") pod \"9e97ece3-54ca-4af5-baa3-7a1472975414\" (UID: \"9e97ece3-54ca-4af5-baa3-7a1472975414\") "
Sep 10 21:27:39.629173 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.629093    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzxf8\" (UniqueName: \"kubernetes.io/projected/766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c-kube-api-access-nzxf8\") pod \"766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c\" (UID: \"766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c\") "
Sep 10 21:27:39.629352 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.629330    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c" (UID: "766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:27:39.629435 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.629411    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e97ece3-54ca-4af5-baa3-7a1472975414-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9e97ece3-54ca-4af5-baa3-7a1472975414" (UID: "9e97ece3-54ca-4af5-baa3-7a1472975414"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:27:39.631207 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.631185    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e97ece3-54ca-4af5-baa3-7a1472975414-kube-api-access-5sx27" (OuterVolumeSpecName: "kube-api-access-5sx27") pod "9e97ece3-54ca-4af5-baa3-7a1472975414" (UID: "9e97ece3-54ca-4af5-baa3-7a1472975414"). InnerVolumeSpecName "kube-api-access-5sx27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:27:39.631294 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.631221    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c-kube-api-access-nzxf8" (OuterVolumeSpecName: "kube-api-access-nzxf8") pod "766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c" (UID: "766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c"). InnerVolumeSpecName "kube-api-access-nzxf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:27:39.714695 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.714666    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7810bd-5b6c7"]
Sep 10 21:27:39.716023 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.716001    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7810bd-5b6c7"]
Sep 10 21:27:39.730227 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.730203    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c-plugins-config-volume\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:27:39.730227 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.730225    2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e97ece3-54ca-4af5-baa3-7a1472975414-plugins-config-volume\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:27:39.730331 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.730235    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sx27\" (UniqueName: \"kubernetes.io/projected/9e97ece3-54ca-4af5-baa3-7a1472975414-kube-api-access-5sx27\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:27:39.730331 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.730243    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzxf8\" (UniqueName: \"kubernetes.io/projected/766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c-kube-api-access-nzxf8\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:27:39.946599 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.946515    2572 generic.go:356] "Generic (PLEG): container finished" podID="766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c" containerID="1d225bf84adac9024fdcc477dca362a467fa0423d6cf46db326c1989c6940cd0" exitCode=0
Sep 10 21:27:39.946599 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.946581    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd"
Sep 10 21:27:39.946811 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.946600    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd" event={"ID":"766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c","Type":"ContainerDied","Data":"1d225bf84adac9024fdcc477dca362a467fa0423d6cf46db326c1989c6940cd0"}
Sep 10 21:27:39.946811 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.946688    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd" event={"ID":"766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c","Type":"ContainerDied","Data":"98e5ad485d4426a9143177dbbeed2a114a6dcdd4afceaddeb7572fb8b10c5787"}
Sep 10 21:27:39.946811 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.946721    2572 scope.go:117] "RemoveContainer" containerID="1d225bf84adac9024fdcc477dca362a467fa0423d6cf46db326c1989c6940cd0"
Sep 10 21:27:39.948206 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.948179    2572 generic.go:356] "Generic (PLEG): container finished" podID="9e97ece3-54ca-4af5-baa3-7a1472975414" containerID="604c6e202566496e6240cad3776b6aec7c916e939ecd207673e8f83e7345e5c4" exitCode=0
Sep 10 21:27:39.948312 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.948237    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f"
Sep 10 21:27:39.948312 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.948267    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f" event={"ID":"9e97ece3-54ca-4af5-baa3-7a1472975414","Type":"ContainerDied","Data":"604c6e202566496e6240cad3776b6aec7c916e939ecd207673e8f83e7345e5c4"}
Sep 10 21:27:39.948506 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.948418    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f" event={"ID":"9e97ece3-54ca-4af5-baa3-7a1472975414","Type":"ContainerDied","Data":"d5de4c86e6dcfd00fb33d073b4b668475dc7e3df4bf11bc2703dcc9f5a1d229f"}
Sep 10 21:27:39.957148 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.957125    2572 scope.go:117] "RemoveContainer" containerID="1d225bf84adac9024fdcc477dca362a467fa0423d6cf46db326c1989c6940cd0"
Sep 10 21:27:39.957455 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:27:39.957434    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d225bf84adac9024fdcc477dca362a467fa0423d6cf46db326c1989c6940cd0\": container with ID starting with 1d225bf84adac9024fdcc477dca362a467fa0423d6cf46db326c1989c6940cd0 not found: ID does not exist" containerID="1d225bf84adac9024fdcc477dca362a467fa0423d6cf46db326c1989c6940cd0"
Sep 10 21:27:39.957518 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.957465    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d225bf84adac9024fdcc477dca362a467fa0423d6cf46db326c1989c6940cd0"} err="failed to get container status \"1d225bf84adac9024fdcc477dca362a467fa0423d6cf46db326c1989c6940cd0\": rpc error: code = NotFound desc = could not find container \"1d225bf84adac9024fdcc477dca362a467fa0423d6cf46db326c1989c6940cd0\": container with ID starting with 1d225bf84adac9024fdcc477dca362a467fa0423d6cf46db326c1989c6940cd0 not found: ID does not exist"
Sep 10 21:27:39.957518 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.957487    2572 scope.go:117] "RemoveContainer" containerID="604c6e202566496e6240cad3776b6aec7c916e939ecd207673e8f83e7345e5c4"
Sep 10 21:27:39.966206 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.966191    2572 scope.go:117] "RemoveContainer" containerID="604c6e202566496e6240cad3776b6aec7c916e939ecd207673e8f83e7345e5c4"
Sep 10 21:27:39.966495 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:27:39.966477    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"604c6e202566496e6240cad3776b6aec7c916e939ecd207673e8f83e7345e5c4\": container with ID starting with 604c6e202566496e6240cad3776b6aec7c916e939ecd207673e8f83e7345e5c4 not found: ID does not exist" containerID="604c6e202566496e6240cad3776b6aec7c916e939ecd207673e8f83e7345e5c4"
Sep 10 21:27:39.966544 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.966502    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"604c6e202566496e6240cad3776b6aec7c916e939ecd207673e8f83e7345e5c4"} err="failed to get container status \"604c6e202566496e6240cad3776b6aec7c916e939ecd207673e8f83e7345e5c4\": rpc error: code = NotFound desc = could not find container \"604c6e202566496e6240cad3776b6aec7c916e939ecd207673e8f83e7345e5c4\": container with ID starting with 604c6e202566496e6240cad3776b6aec7c916e939ecd207673e8f83e7345e5c4 not found: ID does not exist"
Sep 10 21:27:39.977346 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.977324    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd"]
Sep 10 21:27:39.983884 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:39.983860    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7810bd-5bd6b98f98-gg2jd"]
Sep 10 21:27:40.010332 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:40.010309    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f"]
Sep 10 21:27:40.021074 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:40.021049    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7810bd-6989987d77-2lg9f"]
Sep 10 21:27:41.438504 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:41.438466    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1eaf689d-06a9-4ee1-aa60-4f11185b2b98" path="/var/lib/kubelet/pods/1eaf689d-06a9-4ee1-aa60-4f11185b2b98/volumes"
Sep 10 21:27:41.438909 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:41.438853    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c" path="/var/lib/kubelet/pods/766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c/volumes"
Sep 10 21:27:41.439251 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:41.439233    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e97ece3-54ca-4af5-baa3-7a1472975414" path="/var/lib/kubelet/pods/9e97ece3-54ca-4af5-baa3-7a1472975414/volumes"
Sep 10 21:27:42.092325 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:42.092295    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:27:54.168528 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:27:54.168493    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:28:55.791058 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:28:55.791030    2572 scope.go:117] "RemoveContainer" containerID="74b0a187cce29587efe2e0a806559f731f769d0755d47ad47e1ae8a632699764"
Sep 10 21:29:17.323008 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:29:17.322959    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:29:42.827475 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:29:42.827442    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:29:50.331293 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:29:50.331192    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:29:57.421760 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:29:57.421717    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:30:00.153466 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:00.153434    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817930-zx466"]
Sep 10 21:30:00.153829 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:00.153737    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c" containerName="payload-pre-processing"
Sep 10 21:30:00.153829 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:00.153748    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c" containerName="payload-pre-processing"
Sep 10 21:30:00.153829 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:00.153760    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e97ece3-54ca-4af5-baa3-7a1472975414" containerName="payload-processing"
Sep 10 21:30:00.153829 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:00.153766    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e97ece3-54ca-4af5-baa3-7a1472975414" containerName="payload-processing"
Sep 10 21:30:00.153829 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:00.153784    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eaf689d-06a9-4ee1-aa60-4f11185b2b98" containerName="revoke-keys"
Sep 10 21:30:00.153829 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:00.153790    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eaf689d-06a9-4ee1-aa60-4f11185b2b98" containerName="revoke-keys"
Sep 10 21:30:00.154065 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:00.153844    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e97ece3-54ca-4af5-baa3-7a1472975414" containerName="payload-processing"
Sep 10 21:30:00.154065 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:00.153852    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eaf689d-06a9-4ee1-aa60-4f11185b2b98" containerName="revoke-keys"
Sep 10 21:30:00.154065 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:00.153859    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="766dfa93-eeda-4c2f-8b32-52d8f5dd0b9c" containerName="payload-pre-processing"
Sep 10 21:30:00.156718 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:00.156702    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-zx466"
Sep 10 21:30:00.160413 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:00.160391    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5c7x\""
Sep 10 21:30:00.165949 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:00.165933    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-zx466"
Sep 10 21:30:00.173911 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:00.173892    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817930-zx466"]
Sep 10 21:30:00.437449 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:00.437405    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817930-zx466"]
Sep 10 21:30:00.439722 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:30:00.439695    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef56890_a7b4_4bf2_bd07_63b243137d3e.slice/crio-3bdb1df7d16e0fbfd9fbd214d11471ecc08e24565feff64976c78b9b8182bca8 WatchSource:0}: Error finding container 3bdb1df7d16e0fbfd9fbd214d11471ecc08e24565feff64976c78b9b8182bca8: Status 404 returned error can't find the container with id 3bdb1df7d16e0fbfd9fbd214d11471ecc08e24565feff64976c78b9b8182bca8
Sep 10 21:30:00.597156 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:00.597122    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-zx466" event={"ID":"9ef56890-a7b4-4bf2-bd07-63b243137d3e","Type":"ContainerStarted","Data":"3bdb1df7d16e0fbfd9fbd214d11471ecc08e24565feff64976c78b9b8182bca8"}
Sep 10 21:30:01.603901 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:01.603869    2572 generic.go:356] "Generic (PLEG): container finished" podID="9ef56890-a7b4-4bf2-bd07-63b243137d3e" containerID="99e31624a95a459abee203170c81b7148e51992a522873d54d848c2b3fa5784f" exitCode=0
Sep 10 21:30:01.604274 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:01.603919    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-zx466" event={"ID":"9ef56890-a7b4-4bf2-bd07-63b243137d3e","Type":"ContainerDied","Data":"99e31624a95a459abee203170c81b7148e51992a522873d54d848c2b3fa5784f"}
Sep 10 21:30:02.731399 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:02.731373    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-zx466"
Sep 10 21:30:03.229083 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:03.229051    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:30:03.615108 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:03.615083    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-zx466"
Sep 10 21:30:03.615266 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:03.615108    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-zx466" event={"ID":"9ef56890-a7b4-4bf2-bd07-63b243137d3e","Type":"ContainerDied","Data":"3bdb1df7d16e0fbfd9fbd214d11471ecc08e24565feff64976c78b9b8182bca8"}
Sep 10 21:30:03.615266 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:03.615150    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bdb1df7d16e0fbfd9fbd214d11471ecc08e24565feff64976c78b9b8182bca8"
Sep 10 21:30:18.018471 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:18.018435    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:30:25.767876 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:30:25.767845    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:31:52.420997 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:31:52.420909    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:31:55.825768 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:31:55.825736    2572 scope.go:117] "RemoveContainer" containerID="707e60eb70a7467fd9263df62a2a5fa691c7c2712d07a76645ccf92feee96aa9"
Sep 10 21:32:00.324932 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:00.324892    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:32:06.332007 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:06.331953    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:32:08.792855 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.792819    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"]
Sep 10 21:32:08.793258 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.793188    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ef56890-a7b4-4bf2-bd07-63b243137d3e" containerName="cleanup"
Sep 10 21:32:08.793258 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.793200    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ef56890-a7b4-4bf2-bd07-63b243137d3e" containerName="cleanup"
Sep 10 21:32:08.793332 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.793260    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ef56890-a7b4-4bf2-bd07-63b243137d3e" containerName="cleanup"
Sep 10 21:32:08.796202 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.796185    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:08.798517 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.798502    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-39afae6d-kserve-self-signed-certs\""
Sep 10 21:32:08.805738 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.805715    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"]
Sep 10 21:32:08.866737 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.866709    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aaef37e-37c4-49fd-979e-2c3ffbc88915-model-cache\") pod \"e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn\" (UID: \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\") " pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:08.866857 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.866762    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aaef37e-37c4-49fd-979e-2c3ffbc88915-home\") pod \"e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn\" (UID: \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\") " pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:08.866928 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.866894    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aaef37e-37c4-49fd-979e-2c3ffbc88915-tmp-dir\") pod \"e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn\" (UID: \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\") " pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:08.867006 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.866947    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aaef37e-37c4-49fd-979e-2c3ffbc88915-dshm\") pod \"e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn\" (UID: \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\") " pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:08.867062 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.867007    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lfld\" (UniqueName: \"kubernetes.io/projected/2aaef37e-37c4-49fd-979e-2c3ffbc88915-kube-api-access-6lfld\") pod \"e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn\" (UID: \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\") " pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:08.867062 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.867038    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aaef37e-37c4-49fd-979e-2c3ffbc88915-tls-certs\") pod \"e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn\" (UID: \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\") " pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:08.967468 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.967444    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aaef37e-37c4-49fd-979e-2c3ffbc88915-dshm\") pod \"e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn\" (UID: \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\") " pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:08.967574 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.967482    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lfld\" (UniqueName: \"kubernetes.io/projected/2aaef37e-37c4-49fd-979e-2c3ffbc88915-kube-api-access-6lfld\") pod \"e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn\" (UID: \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\") " pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:08.967574 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.967499    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aaef37e-37c4-49fd-979e-2c3ffbc88915-tls-certs\") pod \"e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn\" (UID: \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\") " pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:08.967574 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.967520    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aaef37e-37c4-49fd-979e-2c3ffbc88915-model-cache\") pod \"e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn\" (UID: \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\") " pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:08.967574 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.967545    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aaef37e-37c4-49fd-979e-2c3ffbc88915-home\") pod \"e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn\" (UID: \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\") " pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:08.967853 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.967599    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aaef37e-37c4-49fd-979e-2c3ffbc88915-tmp-dir\") pod \"e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn\" (UID: \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\") " pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:08.967913 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.967871    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aaef37e-37c4-49fd-979e-2c3ffbc88915-tmp-dir\") pod \"e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn\" (UID: \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\") " pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:08.968017 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.967977    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aaef37e-37c4-49fd-979e-2c3ffbc88915-model-cache\") pod \"e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn\" (UID: \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\") " pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:08.968074 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.968043    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aaef37e-37c4-49fd-979e-2c3ffbc88915-home\") pod \"e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn\" (UID: \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\") " pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:08.969853 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.969833    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aaef37e-37c4-49fd-979e-2c3ffbc88915-dshm\") pod \"e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn\" (UID: \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\") " pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:08.970081 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.970061    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aaef37e-37c4-49fd-979e-2c3ffbc88915-tls-certs\") pod \"e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn\" (UID: \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\") " pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:08.977300 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:08.977279    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lfld\" (UniqueName: \"kubernetes.io/projected/2aaef37e-37c4-49fd-979e-2c3ffbc88915-kube-api-access-6lfld\") pod \"e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn\" (UID: \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\") " pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:09.105602 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:09.105535    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:09.378115 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:09.378089    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"]
Sep 10 21:32:09.380800 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:32:09.380774    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2aaef37e_37c4_49fd_979e_2c3ffbc88915.slice/crio-128600b96d28e5ae109e17afee3674a270a168f95ada6166829c6762e1bbe2d7 WatchSource:0}: Error finding container 128600b96d28e5ae109e17afee3674a270a168f95ada6166829c6762e1bbe2d7: Status 404 returned error can't find the container with id 128600b96d28e5ae109e17afee3674a270a168f95ada6166829c6762e1bbe2d7
Sep 10 21:32:10.210298 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:10.210262    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn" event={"ID":"2aaef37e-37c4-49fd-979e-2c3ffbc88915","Type":"ContainerStarted","Data":"bb13b50c4475d5f64fd7450ad437500ba1456e91bedcada1e1585d848801e418"}
Sep 10 21:32:10.210663 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:10.210307    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn" event={"ID":"2aaef37e-37c4-49fd-979e-2c3ffbc88915","Type":"ContainerStarted","Data":"128600b96d28e5ae109e17afee3674a270a168f95ada6166829c6762e1bbe2d7"}
Sep 10 21:32:10.230660 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:10.230611    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn" podStartSLOduration=2.230595472 podStartE2EDuration="2.230595472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:32:10.2287554 +0000 UTC m=+2115.441220318" watchObservedRunningTime="2026-09-10 21:32:10.230595472 +0000 UTC m=+2115.443060390"
Sep 10 21:32:12.649096 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:12.649059    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:32:15.831214 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:15.831181    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"]
Sep 10 21:32:15.834521 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:15.834501    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:15.839863 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:15.839841    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-39afae6d-kserve-self-signed-certs\""
Sep 10 21:32:15.871081 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:15.871060    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"]
Sep 10 21:32:15.929190 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:15.929164    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-model-cache\") pod \"e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5\" (UID: \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\") " pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:15.929316 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:15.929200    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-home\") pod \"e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5\" (UID: \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\") " pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:15.929316 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:15.929225    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-dshm\") pod \"e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5\" (UID: \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\") " pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:15.929316 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:15.929250    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-tls-certs\") pod \"e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5\" (UID: \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\") " pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:15.929316 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:15.929271    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-tmp-dir\") pod \"e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5\" (UID: \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\") " pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:15.929478 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:15.929324    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jnqb\" (UniqueName: \"kubernetes.io/projected/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-kube-api-access-6jnqb\") pod \"e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5\" (UID: \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\") " pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:16.029791 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:16.029768    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jnqb\" (UniqueName: \"kubernetes.io/projected/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-kube-api-access-6jnqb\") pod \"e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5\" (UID: \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\") " pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:16.029934 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:16.029809    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-model-cache\") pod \"e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5\" (UID: \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\") " pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:16.029934 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:16.029839    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-home\") pod \"e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5\" (UID: \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\") " pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:16.029934 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:16.029873    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-dshm\") pod \"e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5\" (UID: \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\") " pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:16.029934 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:16.029889    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-tls-certs\") pod \"e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5\" (UID: \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\") " pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:16.029934 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:16.029908    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-tmp-dir\") pod \"e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5\" (UID: \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\") " pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:16.030289 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:16.030271    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-model-cache\") pod \"e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5\" (UID: \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\") " pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:16.030352 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:16.030291    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-tmp-dir\") pod \"e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5\" (UID: \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\") " pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:16.030352 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:16.030329    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-home\") pod \"e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5\" (UID: \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\") " pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:16.032396 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:16.032375    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-dshm\") pod \"e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5\" (UID: \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\") " pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:16.032538 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:16.032521    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-tls-certs\") pod \"e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5\" (UID: \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\") " pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:16.038310 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:16.038280    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jnqb\" (UniqueName: \"kubernetes.io/projected/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-kube-api-access-6jnqb\") pod \"e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5\" (UID: \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\") " pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:16.144308 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:16.144245    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:16.417010 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:16.416927    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"]
Sep 10 21:32:16.423177 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:32:16.423130    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb8206e9_9356_4b4a_8e8d_c6b18b08069f.slice/crio-ade5248f0ab1d1c0a8736df1241af68c4e39c990ef50a448f421688e0084ba95 WatchSource:0}: Error finding container ade5248f0ab1d1c0a8736df1241af68c4e39c990ef50a448f421688e0084ba95: Status 404 returned error can't find the container with id ade5248f0ab1d1c0a8736df1241af68c4e39c990ef50a448f421688e0084ba95
Sep 10 21:32:17.245850 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:17.245818    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5" event={"ID":"fb8206e9-9356-4b4a-8e8d-c6b18b08069f","Type":"ContainerStarted","Data":"e94dae4554c3a2570ffedc988a759a4095eb3bb3ab41db927cf71e321e0b9de3"}
Sep 10 21:32:17.245850 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:17.245857    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5" event={"ID":"fb8206e9-9356-4b4a-8e8d-c6b18b08069f","Type":"ContainerStarted","Data":"ade5248f0ab1d1c0a8736df1241af68c4e39c990ef50a448f421688e0084ba95"}
Sep 10 21:32:17.264083 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:17.264039    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5" podStartSLOduration=2.264023764 podStartE2EDuration="2.264023764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:32:17.262897531 +0000 UTC m=+2122.475362451" watchObservedRunningTime="2026-09-10 21:32:17.264023764 +0000 UTC m=+2122.476488677"
Sep 10 21:32:19.105838 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:19.105805    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:19.106319 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:19.105855    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:19.118442 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:19.118420    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:19.266927 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:19.266900    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:23.921616 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:23.921586    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:32:26.144653 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:26.144618    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:26.144653 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:26.144659    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:26.156710 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:26.156687    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:26.301389 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:26.301365    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:32.554448 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:32.554404    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"]
Sep 10 21:32:34.272709 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:34.272664    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"]
Sep 10 21:32:34.673000 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:34.672952    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:32:43.527795 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:43.527760    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:32:47.580059 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:47.579926    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn" podUID="2aaef37e-37c4-49fd-979e-2c3ffbc88915" containerName="main" containerID="cri-o://bb13b50c4475d5f64fd7450ad437500ba1456e91bedcada1e1585d848801e418" gracePeriod=45
Sep 10 21:32:47.839014 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:47.838944    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:47.867232 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:47.867207    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aaef37e-37c4-49fd-979e-2c3ffbc88915-home\") pod \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\" (UID: \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\") "
Sep 10 21:32:47.867376 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:47.867246    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aaef37e-37c4-49fd-979e-2c3ffbc88915-tmp-dir\") pod \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\" (UID: \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\") "
Sep 10 21:32:47.867376 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:47.867308    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lfld\" (UniqueName: \"kubernetes.io/projected/2aaef37e-37c4-49fd-979e-2c3ffbc88915-kube-api-access-6lfld\") pod \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\" (UID: \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\") "
Sep 10 21:32:47.867376 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:47.867332    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aaef37e-37c4-49fd-979e-2c3ffbc88915-dshm\") pod \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\" (UID: \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\") "
Sep 10 21:32:47.867544 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:47.867426    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aaef37e-37c4-49fd-979e-2c3ffbc88915-model-cache\") pod \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\" (UID: \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\") "
Sep 10 21:32:47.867544 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:47.867454    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aaef37e-37c4-49fd-979e-2c3ffbc88915-tls-certs\") pod \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\" (UID: \"2aaef37e-37c4-49fd-979e-2c3ffbc88915\") "
Sep 10 21:32:47.867544 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:47.867463    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aaef37e-37c4-49fd-979e-2c3ffbc88915-home" (OuterVolumeSpecName: "home") pod "2aaef37e-37c4-49fd-979e-2c3ffbc88915" (UID: "2aaef37e-37c4-49fd-979e-2c3ffbc88915"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:32:47.867705 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:47.867552    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aaef37e-37c4-49fd-979e-2c3ffbc88915-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2aaef37e-37c4-49fd-979e-2c3ffbc88915" (UID: "2aaef37e-37c4-49fd-979e-2c3ffbc88915"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:32:47.867705 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:47.867642    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aaef37e-37c4-49fd-979e-2c3ffbc88915-model-cache" (OuterVolumeSpecName: "model-cache") pod "2aaef37e-37c4-49fd-979e-2c3ffbc88915" (UID: "2aaef37e-37c4-49fd-979e-2c3ffbc88915"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:32:47.867817 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:47.867758    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aaef37e-37c4-49fd-979e-2c3ffbc88915-home\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:32:47.867817 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:47.867772    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aaef37e-37c4-49fd-979e-2c3ffbc88915-tmp-dir\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:32:47.867817 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:47.867785    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aaef37e-37c4-49fd-979e-2c3ffbc88915-model-cache\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:32:47.869501 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:47.869479    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aaef37e-37c4-49fd-979e-2c3ffbc88915-dshm" (OuterVolumeSpecName: "dshm") pod "2aaef37e-37c4-49fd-979e-2c3ffbc88915" (UID: "2aaef37e-37c4-49fd-979e-2c3ffbc88915"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:32:47.869501 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:47.869481    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aaef37e-37c4-49fd-979e-2c3ffbc88915-kube-api-access-6lfld" (OuterVolumeSpecName: "kube-api-access-6lfld") pod "2aaef37e-37c4-49fd-979e-2c3ffbc88915" (UID: "2aaef37e-37c4-49fd-979e-2c3ffbc88915"). InnerVolumeSpecName "kube-api-access-6lfld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:32:47.869962 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:47.869944    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aaef37e-37c4-49fd-979e-2c3ffbc88915-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2aaef37e-37c4-49fd-979e-2c3ffbc88915" (UID: "2aaef37e-37c4-49fd-979e-2c3ffbc88915"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:32:47.968703 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:47.968680    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lfld\" (UniqueName: \"kubernetes.io/projected/2aaef37e-37c4-49fd-979e-2c3ffbc88915-kube-api-access-6lfld\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:32:47.968703 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:47.968701    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aaef37e-37c4-49fd-979e-2c3ffbc88915-dshm\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:32:47.968833 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:47.968711    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aaef37e-37c4-49fd-979e-2c3ffbc88915-tls-certs\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:32:48.398182 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:48.398151    2572 generic.go:356] "Generic (PLEG): container finished" podID="2aaef37e-37c4-49fd-979e-2c3ffbc88915" containerID="bb13b50c4475d5f64fd7450ad437500ba1456e91bedcada1e1585d848801e418" exitCode=0
Sep 10 21:32:48.398308 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:48.398201    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"
Sep 10 21:32:48.398308 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:48.398215    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn" event={"ID":"2aaef37e-37c4-49fd-979e-2c3ffbc88915","Type":"ContainerDied","Data":"bb13b50c4475d5f64fd7450ad437500ba1456e91bedcada1e1585d848801e418"}
Sep 10 21:32:48.398308 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:48.398258    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn" event={"ID":"2aaef37e-37c4-49fd-979e-2c3ffbc88915","Type":"ContainerDied","Data":"128600b96d28e5ae109e17afee3674a270a168f95ada6166829c6762e1bbe2d7"}
Sep 10 21:32:48.398308 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:48.398276    2572 scope.go:117] "RemoveContainer" containerID="bb13b50c4475d5f64fd7450ad437500ba1456e91bedcada1e1585d848801e418"
Sep 10 21:32:48.408055 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:48.408031    2572 scope.go:117] "RemoveContainer" containerID="bb13b50c4475d5f64fd7450ad437500ba1456e91bedcada1e1585d848801e418"
Sep 10 21:32:48.408420 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:32:48.408393    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb13b50c4475d5f64fd7450ad437500ba1456e91bedcada1e1585d848801e418\": container with ID starting with bb13b50c4475d5f64fd7450ad437500ba1456e91bedcada1e1585d848801e418 not found: ID does not exist" containerID="bb13b50c4475d5f64fd7450ad437500ba1456e91bedcada1e1585d848801e418"
Sep 10 21:32:48.408535 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:48.408424    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb13b50c4475d5f64fd7450ad437500ba1456e91bedcada1e1585d848801e418"} err="failed to get container status \"bb13b50c4475d5f64fd7450ad437500ba1456e91bedcada1e1585d848801e418\": rpc error: code = NotFound desc = could not find container \"bb13b50c4475d5f64fd7450ad437500ba1456e91bedcada1e1585d848801e418\": container with ID starting with bb13b50c4475d5f64fd7450ad437500ba1456e91bedcada1e1585d848801e418 not found: ID does not exist"
Sep 10 21:32:48.422200 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:48.422176    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"]
Sep 10 21:32:48.427846 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:48.427824    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-39afae6d-kserve-7ccfb98c95-sx9zn"]
Sep 10 21:32:49.292762 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:49.292718    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5" podUID="fb8206e9-9356-4b4a-8e8d-c6b18b08069f" containerName="main" containerID="cri-o://e94dae4554c3a2570ffedc988a759a4095eb3bb3ab41db927cf71e321e0b9de3" gracePeriod=45
Sep 10 21:32:49.437683 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:49.437646    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2aaef37e-37c4-49fd-979e-2c3ffbc88915" path="/var/lib/kubelet/pods/2aaef37e-37c4-49fd-979e-2c3ffbc88915/volumes"
Sep 10 21:32:49.544199 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:49.544144    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:49.582388 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:49.582358    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-model-cache\") pod \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\" (UID: \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\") "
Sep 10 21:32:49.582534 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:49.582398    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-tmp-dir\") pod \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\" (UID: \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\") "
Sep 10 21:32:49.582534 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:49.582426    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-home\") pod \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\" (UID: \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\") "
Sep 10 21:32:49.582534 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:49.582454    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jnqb\" (UniqueName: \"kubernetes.io/projected/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-kube-api-access-6jnqb\") pod \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\" (UID: \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\") "
Sep 10 21:32:49.582534 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:49.582480    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-dshm\") pod \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\" (UID: \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\") "
Sep 10 21:32:49.582534 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:49.582505    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-tls-certs\") pod \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\" (UID: \"fb8206e9-9356-4b4a-8e8d-c6b18b08069f\") "
Sep 10 21:32:49.582800 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:49.582606    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-model-cache" (OuterVolumeSpecName: "model-cache") pod "fb8206e9-9356-4b4a-8e8d-c6b18b08069f" (UID: "fb8206e9-9356-4b4a-8e8d-c6b18b08069f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:32:49.582800 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:49.582634    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fb8206e9-9356-4b4a-8e8d-c6b18b08069f" (UID: "fb8206e9-9356-4b4a-8e8d-c6b18b08069f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:32:49.582800 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:49.582698    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-home" (OuterVolumeSpecName: "home") pod "fb8206e9-9356-4b4a-8e8d-c6b18b08069f" (UID: "fb8206e9-9356-4b4a-8e8d-c6b18b08069f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:32:49.582800 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:49.582787    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-model-cache\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:32:49.582971 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:49.582804    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-tmp-dir\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:32:49.582971 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:49.582817    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-home\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:32:49.584676 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:49.584654    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-dshm" (OuterVolumeSpecName: "dshm") pod "fb8206e9-9356-4b4a-8e8d-c6b18b08069f" (UID: "fb8206e9-9356-4b4a-8e8d-c6b18b08069f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:32:49.584676 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:49.584666    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-kube-api-access-6jnqb" (OuterVolumeSpecName: "kube-api-access-6jnqb") pod "fb8206e9-9356-4b4a-8e8d-c6b18b08069f" (UID: "fb8206e9-9356-4b4a-8e8d-c6b18b08069f"). InnerVolumeSpecName "kube-api-access-6jnqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:32:49.585085 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:49.585069    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb8206e9-9356-4b4a-8e8d-c6b18b08069f" (UID: "fb8206e9-9356-4b4a-8e8d-c6b18b08069f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:32:49.683179 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:49.683155    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jnqb\" (UniqueName: \"kubernetes.io/projected/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-kube-api-access-6jnqb\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:32:49.683179 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:49.683178    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-dshm\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:32:49.683318 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:49.683191    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb8206e9-9356-4b4a-8e8d-c6b18b08069f-tls-certs\") on node \"ip-10-0-138-232.ec2.internal\" DevicePath \"\""
Sep 10 21:32:50.410199 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:50.410155    2572 generic.go:356] "Generic (PLEG): container finished" podID="fb8206e9-9356-4b4a-8e8d-c6b18b08069f" containerID="e94dae4554c3a2570ffedc988a759a4095eb3bb3ab41db927cf71e321e0b9de3" exitCode=0
Sep 10 21:32:50.410199 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:50.410200    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"
Sep 10 21:32:50.410690 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:50.410234    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5" event={"ID":"fb8206e9-9356-4b4a-8e8d-c6b18b08069f","Type":"ContainerDied","Data":"e94dae4554c3a2570ffedc988a759a4095eb3bb3ab41db927cf71e321e0b9de3"}
Sep 10 21:32:50.410690 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:50.410282    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5" event={"ID":"fb8206e9-9356-4b4a-8e8d-c6b18b08069f","Type":"ContainerDied","Data":"ade5248f0ab1d1c0a8736df1241af68c4e39c990ef50a448f421688e0084ba95"}
Sep 10 21:32:50.410690 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:50.410299    2572 scope.go:117] "RemoveContainer" containerID="e94dae4554c3a2570ffedc988a759a4095eb3bb3ab41db927cf71e321e0b9de3"
Sep 10 21:32:50.419210 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:50.419193    2572 scope.go:117] "RemoveContainer" containerID="e94dae4554c3a2570ffedc988a759a4095eb3bb3ab41db927cf71e321e0b9de3"
Sep 10 21:32:50.419493 ip-10-0-138-232 kubenswrapper[2572]: E0910 21:32:50.419474    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94dae4554c3a2570ffedc988a759a4095eb3bb3ab41db927cf71e321e0b9de3\": container with ID starting with e94dae4554c3a2570ffedc988a759a4095eb3bb3ab41db927cf71e321e0b9de3 not found: ID does not exist" containerID="e94dae4554c3a2570ffedc988a759a4095eb3bb3ab41db927cf71e321e0b9de3"
Sep 10 21:32:50.419534 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:50.419503    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94dae4554c3a2570ffedc988a759a4095eb3bb3ab41db927cf71e321e0b9de3"} err="failed to get container status \"e94dae4554c3a2570ffedc988a759a4095eb3bb3ab41db927cf71e321e0b9de3\": rpc error: code = NotFound desc = could not find container \"e94dae4554c3a2570ffedc988a759a4095eb3bb3ab41db927cf71e321e0b9de3\": container with ID starting with e94dae4554c3a2570ffedc988a759a4095eb3bb3ab41db927cf71e321e0b9de3 not found: ID does not exist"
Sep 10 21:32:50.430911 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:50.430890    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"]
Sep 10 21:32:50.437585 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:50.437567    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-39afae6d-kserve-794b89bf7f-v8vd5"]
Sep 10 21:32:50.934731 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:50.934699    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:32:51.437797 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:51.437762    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb8206e9-9356-4b4a-8e8d-c6b18b08069f" path="/var/lib/kubelet/pods/fb8206e9-9356-4b4a-8e8d-c6b18b08069f/volumes"
Sep 10 21:32:57.834709 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:32:57.834674    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:33:05.729224 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:05.729191    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:33:06.044647 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.044621    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-84ff5476d-jptcr"]
Sep 10 21:33:06.044938 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.044927    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aaef37e-37c4-49fd-979e-2c3ffbc88915" containerName="main"
Sep 10 21:33:06.045001 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.044940    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aaef37e-37c4-49fd-979e-2c3ffbc88915" containerName="main"
Sep 10 21:33:06.045001 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.044957    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb8206e9-9356-4b4a-8e8d-c6b18b08069f" containerName="main"
Sep 10 21:33:06.045001 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.044963    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb8206e9-9356-4b4a-8e8d-c6b18b08069f" containerName="main"
Sep 10 21:33:06.045099 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.045049    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb8206e9-9356-4b4a-8e8d-c6b18b08069f" containerName="main"
Sep 10 21:33:06.045099 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.045057    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aaef37e-37c4-49fd-979e-2c3ffbc88915" containerName="main"
Sep 10 21:33:06.047545 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.047526    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84ff5476d-jptcr"
Sep 10 21:33:06.051048 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.051022    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 21:33:06.051048 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.051046    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 21:33:06.051234 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.051025    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9ckhj\""
Sep 10 21:33:06.051234 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.051025    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:33:06.051234 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.051032    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 21:33:06.059081 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.059051    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84ff5476d-jptcr"]
Sep 10 21:33:06.100804 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.100781    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d541f7c3-4ba3-4893-9151-caa9a6703e6d-openshift-service-ca\") pod \"maas-api-84ff5476d-jptcr\" (UID: \"d541f7c3-4ba3-4893-9151-caa9a6703e6d\") " pod="odh-ai-gateway-infra/maas-api-84ff5476d-jptcr"
Sep 10 21:33:06.100923 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.100825    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d541f7c3-4ba3-4893-9151-caa9a6703e6d-maas-api-tls\") pod \"maas-api-84ff5476d-jptcr\" (UID: \"d541f7c3-4ba3-4893-9151-caa9a6703e6d\") " pod="odh-ai-gateway-infra/maas-api-84ff5476d-jptcr"
Sep 10 21:33:06.100923 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.100846    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dnjw\" (UniqueName: \"kubernetes.io/projected/d541f7c3-4ba3-4893-9151-caa9a6703e6d-kube-api-access-6dnjw\") pod \"maas-api-84ff5476d-jptcr\" (UID: \"d541f7c3-4ba3-4893-9151-caa9a6703e6d\") " pod="odh-ai-gateway-infra/maas-api-84ff5476d-jptcr"
Sep 10 21:33:06.101035 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.100953    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d541f7c3-4ba3-4893-9151-caa9a6703e6d-metrics-certs\") pod \"maas-api-84ff5476d-jptcr\" (UID: \"d541f7c3-4ba3-4893-9151-caa9a6703e6d\") " pod="odh-ai-gateway-infra/maas-api-84ff5476d-jptcr"
Sep 10 21:33:06.202269 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.202244    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dnjw\" (UniqueName: \"kubernetes.io/projected/d541f7c3-4ba3-4893-9151-caa9a6703e6d-kube-api-access-6dnjw\") pod \"maas-api-84ff5476d-jptcr\" (UID: \"d541f7c3-4ba3-4893-9151-caa9a6703e6d\") " pod="odh-ai-gateway-infra/maas-api-84ff5476d-jptcr"
Sep 10 21:33:06.202365 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.202287    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d541f7c3-4ba3-4893-9151-caa9a6703e6d-metrics-certs\") pod \"maas-api-84ff5476d-jptcr\" (UID: \"d541f7c3-4ba3-4893-9151-caa9a6703e6d\") " pod="odh-ai-gateway-infra/maas-api-84ff5476d-jptcr"
Sep 10 21:33:06.202365 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.202314    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d541f7c3-4ba3-4893-9151-caa9a6703e6d-openshift-service-ca\") pod \"maas-api-84ff5476d-jptcr\" (UID: \"d541f7c3-4ba3-4893-9151-caa9a6703e6d\") " pod="odh-ai-gateway-infra/maas-api-84ff5476d-jptcr"
Sep 10 21:33:06.202365 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.202341    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d541f7c3-4ba3-4893-9151-caa9a6703e6d-maas-api-tls\") pod \"maas-api-84ff5476d-jptcr\" (UID: \"d541f7c3-4ba3-4893-9151-caa9a6703e6d\") " pod="odh-ai-gateway-infra/maas-api-84ff5476d-jptcr"
Sep 10 21:33:06.202947 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.202923    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d541f7c3-4ba3-4893-9151-caa9a6703e6d-openshift-service-ca\") pod \"maas-api-84ff5476d-jptcr\" (UID: \"d541f7c3-4ba3-4893-9151-caa9a6703e6d\") " pod="odh-ai-gateway-infra/maas-api-84ff5476d-jptcr"
Sep 10 21:33:06.204776 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.204759    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d541f7c3-4ba3-4893-9151-caa9a6703e6d-maas-api-tls\") pod \"maas-api-84ff5476d-jptcr\" (UID: \"d541f7c3-4ba3-4893-9151-caa9a6703e6d\") " pod="odh-ai-gateway-infra/maas-api-84ff5476d-jptcr"
Sep 10 21:33:06.204834 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.204809    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d541f7c3-4ba3-4893-9151-caa9a6703e6d-metrics-certs\") pod \"maas-api-84ff5476d-jptcr\" (UID: \"d541f7c3-4ba3-4893-9151-caa9a6703e6d\") " pod="odh-ai-gateway-infra/maas-api-84ff5476d-jptcr"
Sep 10 21:33:06.234908 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.234889    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dnjw\" (UniqueName: \"kubernetes.io/projected/d541f7c3-4ba3-4893-9151-caa9a6703e6d-kube-api-access-6dnjw\") pod \"maas-api-84ff5476d-jptcr\" (UID: \"d541f7c3-4ba3-4893-9151-caa9a6703e6d\") " pod="odh-ai-gateway-infra/maas-api-84ff5476d-jptcr"
Sep 10 21:33:06.357849 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.357794    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84ff5476d-jptcr"
Sep 10 21:33:06.631831 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.631756    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84ff5476d-jptcr"]
Sep 10 21:33:06.634788 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:33:06.634765    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd541f7c3_4ba3_4893_9151_caa9a6703e6d.slice/crio-044ade2cfb349557def90c0d345aae5507e929a0f4fc298655c9ad6caee5f3af WatchSource:0}: Error finding container 044ade2cfb349557def90c0d345aae5507e929a0f4fc298655c9ad6caee5f3af: Status 404 returned error can't find the container with id 044ade2cfb349557def90c0d345aae5507e929a0f4fc298655c9ad6caee5f3af
Sep 10 21:33:06.635992 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:06.635954    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:33:07.497745 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:07.497707    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84ff5476d-jptcr" event={"ID":"d541f7c3-4ba3-4893-9151-caa9a6703e6d","Type":"ContainerStarted","Data":"044ade2cfb349557def90c0d345aae5507e929a0f4fc298655c9ad6caee5f3af"}
Sep 10 21:33:09.509501 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:09.509467    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84ff5476d-jptcr" event={"ID":"d541f7c3-4ba3-4893-9151-caa9a6703e6d","Type":"ContainerStarted","Data":"de99e9f8af7eafaeebe0bbe84a1cc1f62909a46506bb7ffd0b406eec2df25f2d"}
Sep 10 21:33:09.509864 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:09.509576    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-84ff5476d-jptcr"
Sep 10 21:33:09.529914 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:09.529850    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-84ff5476d-jptcr" podStartSLOduration=1.259204525 podStartE2EDuration="3.529834401s" totalImagesPullingTime="2.270629876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:33:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:33:09.527847166 +0000 UTC m=+2174.740312078" watchObservedRunningTime="2026-09-10 21:33:09.529834401 +0000 UTC m=+2174.742299344"
Sep 10 21:33:12.817256 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:12.817223    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:33:15.519915 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:15.519890    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-84ff5476d-jptcr"
Sep 10 21:33:18.533637 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:18.533599    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:33:25.626865 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:25.626833    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n4h5b"]
Sep 10 21:33:28.300783 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:28.300754    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fd9f86dc6-2vsr9_5f9fd438-9816-4e56-8cec-eb342ed06382/authorino/0.log"
Sep 10 21:33:34.645273 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:34.645243    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-769bdd6b4-mfblr_0556d7c8-57d9-4775-8532-1e260ea10416/copy-manifests/0.log"
Sep 10 21:33:34.652397 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:34.652376    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-769bdd6b4-mfblr_0556d7c8-57d9-4775-8532-1e260ea10416/manager/0.log"
Sep 10 21:33:35.249966 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:35.249941    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-84ff5476d-jptcr_d541f7c3-4ba3-4893-9151-caa9a6703e6d/maas-api/0.log"
Sep 10 21:33:35.364157 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:35.364133    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817915-tvvlp_1ddd7181-205c-465e-a978-3fa74b7b2c78/cleanup/0.log"
Sep 10 21:33:35.476643 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:35.476615    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817930-zx466_9ef56890-a7b4-4bf2-bd07-63b243137d3e/cleanup/0.log"
Sep 10 21:33:36.392944 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:36.392903    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fd9f86dc6-2vsr9_5f9fd438-9816-4e56-8cec-eb342ed06382/authorino/0.log"
Sep 10 21:33:36.739245 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:36.739167    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-9v9sk_58f8cb50-eaad-47f3-878c-2930d88a8d06/kuadrant-console-plugin/0.log"
Sep 10 21:33:36.994361 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:36.994283    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-n4h5b_4c3ebeca-4c69-4c1a-8f5e-6939411484a0/limitador/0.log"
Sep 10 21:33:37.830338 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:37.830306    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-98c4dbc44-qkwf9_b5e5879b-760c-457b-88d4-6a1734f02c17/kube-auth-proxy/0.log"
Sep 10 21:33:39.397925 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:39.397892    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-xjgmn_ad3cef32-8927-4b38-b735-3e48eefc2955/main/0.log"
Sep 10 21:33:45.542234 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:45.542200    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2hcpt/must-gather-5p5bg"]
Sep 10 21:33:45.547183 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:45.547164    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2hcpt/must-gather-5p5bg"
Sep 10 21:33:45.549434 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:45.549412    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2hcpt\"/\"openshift-service-ca.crt\""
Sep 10 21:33:45.549547 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:45.549435    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2hcpt\"/\"kube-root-ca.crt\""
Sep 10 21:33:45.550241 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:45.550226    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2hcpt\"/\"default-dockercfg-w8z8c\""
Sep 10 21:33:45.554697 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:45.554676    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2hcpt/must-gather-5p5bg"]
Sep 10 21:33:45.717514 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:45.717490    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d43deff3-d354-4474-a1b7-5b556830ac2e-must-gather-output\") pod \"must-gather-5p5bg\" (UID: \"d43deff3-d354-4474-a1b7-5b556830ac2e\") " pod="openshift-must-gather-2hcpt/must-gather-5p5bg"
Sep 10 21:33:45.717657 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:45.717544    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7vj7\" (UniqueName: \"kubernetes.io/projected/d43deff3-d354-4474-a1b7-5b556830ac2e-kube-api-access-f7vj7\") pod \"must-gather-5p5bg\" (UID: \"d43deff3-d354-4474-a1b7-5b556830ac2e\") " pod="openshift-must-gather-2hcpt/must-gather-5p5bg"
Sep 10 21:33:45.818664 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:45.818602    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7vj7\" (UniqueName: \"kubernetes.io/projected/d43deff3-d354-4474-a1b7-5b556830ac2e-kube-api-access-f7vj7\") pod \"must-gather-5p5bg\" (UID: \"d43deff3-d354-4474-a1b7-5b556830ac2e\") " pod="openshift-must-gather-2hcpt/must-gather-5p5bg"
Sep 10 21:33:45.818664 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:45.818658    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d43deff3-d354-4474-a1b7-5b556830ac2e-must-gather-output\") pod \"must-gather-5p5bg\" (UID: \"d43deff3-d354-4474-a1b7-5b556830ac2e\") " pod="openshift-must-gather-2hcpt/must-gather-5p5bg"
Sep 10 21:33:45.818950 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:45.818933    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d43deff3-d354-4474-a1b7-5b556830ac2e-must-gather-output\") pod \"must-gather-5p5bg\" (UID: \"d43deff3-d354-4474-a1b7-5b556830ac2e\") " pod="openshift-must-gather-2hcpt/must-gather-5p5bg"
Sep 10 21:33:45.826828 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:45.826803    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7vj7\" (UniqueName: \"kubernetes.io/projected/d43deff3-d354-4474-a1b7-5b556830ac2e-kube-api-access-f7vj7\") pod \"must-gather-5p5bg\" (UID: \"d43deff3-d354-4474-a1b7-5b556830ac2e\") " pod="openshift-must-gather-2hcpt/must-gather-5p5bg"
Sep 10 21:33:45.857122 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:45.857103    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2hcpt/must-gather-5p5bg"
Sep 10 21:33:46.126601 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:46.126580    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2hcpt/must-gather-5p5bg"]
Sep 10 21:33:46.128795 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:33:46.128763    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd43deff3_d354_4474_a1b7_5b556830ac2e.slice/crio-188e42121fe154ad6bfd26b944b3f32886578bddc65f734abffeb236ac4456f0 WatchSource:0}: Error finding container 188e42121fe154ad6bfd26b944b3f32886578bddc65f734abffeb236ac4456f0: Status 404 returned error can't find the container with id 188e42121fe154ad6bfd26b944b3f32886578bddc65f734abffeb236ac4456f0
Sep 10 21:33:46.688901 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:46.688865    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hcpt/must-gather-5p5bg" event={"ID":"d43deff3-d354-4474-a1b7-5b556830ac2e","Type":"ContainerStarted","Data":"188e42121fe154ad6bfd26b944b3f32886578bddc65f734abffeb236ac4456f0"}
Sep 10 21:33:47.698343 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:47.698251    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hcpt/must-gather-5p5bg" event={"ID":"d43deff3-d354-4474-a1b7-5b556830ac2e","Type":"ContainerStarted","Data":"b27f9787d18412f609b0680e2498983a197a1fa819244132766f47db905e2e45"}
Sep 10 21:33:47.698343 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:47.698305    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hcpt/must-gather-5p5bg" event={"ID":"d43deff3-d354-4474-a1b7-5b556830ac2e","Type":"ContainerStarted","Data":"dfc90f3c053e4c44f4a4c84b7fd89a2907b4e737d4cbb3ddcdf0cc0d6a6ca6cc"}
Sep 10 21:33:47.717803 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:47.717754    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2hcpt/must-gather-5p5bg" podStartSLOduration=2.050670823 podStartE2EDuration="2.717740847s" totalImagesPullingTime="667.070024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:33:47.714927675 +0000 UTC m=+2212.927392594" watchObservedRunningTime="2026-09-10 21:33:47.717740847 +0000 UTC m=+2212.930205765"
Sep 10 21:33:48.483422 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:48.483389    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sm62m_f720dd2d-8c6a-4365-986a-68bbf39853cb/global-pull-secret-syncer/0.log"
Sep 10 21:33:48.577587 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:48.577559    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jfw2v_218e6e53-1d6e-4c3a-adf3-4e9c2d9f9afa/konnectivity-agent/0.log"
Sep 10 21:33:48.681923 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:48.681890    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-232.ec2.internal_cfb73c088fb2994699656a729ac976af/haproxy/0.log"
Sep 10 21:33:52.754848 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:52.754817    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fd9f86dc6-2vsr9_5f9fd438-9816-4e56-8cec-eb342ed06382/authorino/0.log"
Sep 10 21:33:52.869900 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:52.869859    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-9v9sk_58f8cb50-eaad-47f3-878c-2930d88a8d06/kuadrant-console-plugin/0.log"
Sep 10 21:33:52.993498 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:52.993466    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-n4h5b_4c3ebeca-4c69-4c1a-8f5e-6939411484a0/limitador/0.log"
Sep 10 21:33:54.573856 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:54.573674    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3/alertmanager/0.log"
Sep 10 21:33:54.598443 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:54.598410    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3/config-reloader/0.log"
Sep 10 21:33:54.625847 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:54.625815    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3/kube-rbac-proxy-web/0.log"
Sep 10 21:33:54.652098 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:54.651934    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3/kube-rbac-proxy/0.log"
Sep 10 21:33:54.674757 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:54.674713    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3/kube-rbac-proxy-metric/0.log"
Sep 10 21:33:54.697813 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:54.697789    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3/prom-label-proxy/0.log"
Sep 10 21:33:54.722577 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:54.722551    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eeeefc09-7bf6-4e2f-8b6e-b6daa0d412c3/init-config-reloader/0.log"
Sep 10 21:33:54.883556 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:54.883474    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-pd69z_a7abb943-7fef-4fcd-9454-728be3c193ae/monitoring-plugin/0.log"
Sep 10 21:33:55.006806 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:55.006779    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j78zp_eae9e839-dfdb-41b5-8657-77744647bb9a/node-exporter/0.log"
Sep 10 21:33:55.030190 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:55.030149    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j78zp_eae9e839-dfdb-41b5-8657-77744647bb9a/kube-rbac-proxy/0.log"
Sep 10 21:33:55.051481 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:55.051457    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j78zp_eae9e839-dfdb-41b5-8657-77744647bb9a/init-textfile/0.log"
Sep 10 21:33:55.165078 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:55.164996    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-qhhck_7eac4c07-da69-470e-9285-1dec8f7ae8a9/kube-rbac-proxy-main/0.log"
Sep 10 21:33:55.190443 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:55.190415    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-qhhck_7eac4c07-da69-470e-9285-1dec8f7ae8a9/kube-rbac-proxy-self/0.log"
Sep 10 21:33:55.216156 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:55.215960    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-qhhck_7eac4c07-da69-470e-9285-1dec8f7ae8a9/openshift-state-metrics/0.log"
Sep 10 21:33:55.561511 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:55.561475    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7d786f5d47-9ph4f_a9145cc2-5aef-4ab8-af32-675e626ed7c4/telemeter-client/0.log"
Sep 10 21:33:55.600692 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:55.600640    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7d786f5d47-9ph4f_a9145cc2-5aef-4ab8-af32-675e626ed7c4/reload/0.log"
Sep 10 21:33:55.667527 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:55.667496    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7d786f5d47-9ph4f_a9145cc2-5aef-4ab8-af32-675e626ed7c4/kube-rbac-proxy/0.log"
Sep 10 21:33:55.862361 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:55.862282    2572 scope.go:117] "RemoveContainer" containerID="37d2135f61a3cb641c62af32840fb18e36aa53db1387a2d890b34dfe2e615d0b"
Sep 10 21:33:58.587513 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:58.587476    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-f5qj7_f0cb65f4-5aa1-41bd-9584-e3d440e4cb79/download-server/0.log"
Sep 10 21:33:58.748426 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:58.748390    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s"]
Sep 10 21:33:58.752581 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:58.752559    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s"
Sep 10 21:33:58.762225 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:58.762201    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s"]
Sep 10 21:33:58.853239 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:58.853142    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/45cde866-1818-4fa3-8d6c-27fcb5dcb6a1-proc\") pod \"perf-node-gather-daemonset-z8g2s\" (UID: \"45cde866-1818-4fa3-8d6c-27fcb5dcb6a1\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s"
Sep 10 21:33:58.853239 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:58.853224    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/45cde866-1818-4fa3-8d6c-27fcb5dcb6a1-podres\") pod \"perf-node-gather-daemonset-z8g2s\" (UID: \"45cde866-1818-4fa3-8d6c-27fcb5dcb6a1\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s"
Sep 10 21:33:58.853490 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:58.853292    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45cde866-1818-4fa3-8d6c-27fcb5dcb6a1-sys\") pod \"perf-node-gather-daemonset-z8g2s\" (UID: \"45cde866-1818-4fa3-8d6c-27fcb5dcb6a1\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s"
Sep 10 21:33:58.853490 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:58.853323    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/45cde866-1818-4fa3-8d6c-27fcb5dcb6a1-lib-modules\") pod \"perf-node-gather-daemonset-z8g2s\" (UID: \"45cde866-1818-4fa3-8d6c-27fcb5dcb6a1\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s"
Sep 10 21:33:58.853490 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:58.853356    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcg88\" (UniqueName: \"kubernetes.io/projected/45cde866-1818-4fa3-8d6c-27fcb5dcb6a1-kube-api-access-jcg88\") pod \"perf-node-gather-daemonset-z8g2s\" (UID: \"45cde866-1818-4fa3-8d6c-27fcb5dcb6a1\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s"
Sep 10 21:33:58.955174 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:58.954228    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/45cde866-1818-4fa3-8d6c-27fcb5dcb6a1-proc\") pod \"perf-node-gather-daemonset-z8g2s\" (UID: \"45cde866-1818-4fa3-8d6c-27fcb5dcb6a1\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s"
Sep 10 21:33:58.955174 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:58.954313    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/45cde866-1818-4fa3-8d6c-27fcb5dcb6a1-podres\") pod \"perf-node-gather-daemonset-z8g2s\" (UID: \"45cde866-1818-4fa3-8d6c-27fcb5dcb6a1\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s"
Sep 10 21:33:58.955174 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:58.954386    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45cde866-1818-4fa3-8d6c-27fcb5dcb6a1-sys\") pod \"perf-node-gather-daemonset-z8g2s\" (UID: \"45cde866-1818-4fa3-8d6c-27fcb5dcb6a1\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s"
Sep 10 21:33:58.955174 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:58.954419    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/45cde866-1818-4fa3-8d6c-27fcb5dcb6a1-lib-modules\") pod \"perf-node-gather-daemonset-z8g2s\" (UID: \"45cde866-1818-4fa3-8d6c-27fcb5dcb6a1\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s"
Sep 10 21:33:58.955174 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:58.954452    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcg88\" (UniqueName: \"kubernetes.io/projected/45cde866-1818-4fa3-8d6c-27fcb5dcb6a1-kube-api-access-jcg88\") pod \"perf-node-gather-daemonset-z8g2s\" (UID: \"45cde866-1818-4fa3-8d6c-27fcb5dcb6a1\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s"
Sep 10 21:33:58.955174 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:58.954850    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/45cde866-1818-4fa3-8d6c-27fcb5dcb6a1-proc\") pod \"perf-node-gather-daemonset-z8g2s\" (UID: \"45cde866-1818-4fa3-8d6c-27fcb5dcb6a1\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s"
Sep 10 21:33:58.955174 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:58.954973    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/45cde866-1818-4fa3-8d6c-27fcb5dcb6a1-podres\") pod \"perf-node-gather-daemonset-z8g2s\" (UID: \"45cde866-1818-4fa3-8d6c-27fcb5dcb6a1\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s"
Sep 10 21:33:58.955174 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:58.955049    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45cde866-1818-4fa3-8d6c-27fcb5dcb6a1-sys\") pod \"perf-node-gather-daemonset-z8g2s\" (UID: \"45cde866-1818-4fa3-8d6c-27fcb5dcb6a1\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s"
Sep 10 21:33:58.955174 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:58.955137    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/45cde866-1818-4fa3-8d6c-27fcb5dcb6a1-lib-modules\") pod \"perf-node-gather-daemonset-z8g2s\" (UID: \"45cde866-1818-4fa3-8d6c-27fcb5dcb6a1\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s"
Sep 10 21:33:58.963811 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:58.963782    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcg88\" (UniqueName: \"kubernetes.io/projected/45cde866-1818-4fa3-8d6c-27fcb5dcb6a1-kube-api-access-jcg88\") pod \"perf-node-gather-daemonset-z8g2s\" (UID: \"45cde866-1818-4fa3-8d6c-27fcb5dcb6a1\") " pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s"
Sep 10 21:33:59.065264 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:59.065228    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s"
Sep 10 21:33:59.324085 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:59.324049    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-j49cs_408e4fb2-52fb-4a31-92d8-9f9aeaeacf6d/volume-data-source-validator/0.log"
Sep 10 21:33:59.364658 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:59.364583    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s"]
Sep 10 21:33:59.368244 ip-10-0-138-232 kubenswrapper[2572]: W0910 21:33:59.368211    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod45cde866_1818_4fa3_8d6c_27fcb5dcb6a1.slice/crio-c3f20e3a24c90613096c3860605a3ec430bd02e9b71ed3553681bcf1ef3cb2df WatchSource:0}: Error finding container c3f20e3a24c90613096c3860605a3ec430bd02e9b71ed3553681bcf1ef3cb2df: Status 404 returned error can't find the container with id c3f20e3a24c90613096c3860605a3ec430bd02e9b71ed3553681bcf1ef3cb2df
Sep 10 21:33:59.806803 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:59.806280    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s" event={"ID":"45cde866-1818-4fa3-8d6c-27fcb5dcb6a1","Type":"ContainerStarted","Data":"4fb495e294c97048c802f78a06d0dae96b2f05852a471bd4767dafd9c5ba36ce"}
Sep 10 21:33:59.806803 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:59.806348    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s" event={"ID":"45cde866-1818-4fa3-8d6c-27fcb5dcb6a1","Type":"ContainerStarted","Data":"c3f20e3a24c90613096c3860605a3ec430bd02e9b71ed3553681bcf1ef3cb2df"}
Sep 10 21:33:59.806803 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:59.806668    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s"
Sep 10 21:33:59.825653 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:33:59.825604    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s" podStartSLOduration=1.825581356 podStartE2EDuration="1.825581356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 21:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:33:59.825219421 +0000 UTC m=+2225.037684340" watchObservedRunningTime="2026-09-10 21:33:59.825581356 +0000 UTC m=+2225.038046278"
Sep 10 21:34:00.588075 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:00.588050    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lwhlk_7afb1a12-ba19-4437-8444-efd2b3a5de37/dns/0.log"
Sep 10 21:34:00.613261 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:00.613221    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lwhlk_7afb1a12-ba19-4437-8444-efd2b3a5de37/kube-rbac-proxy/0.log"
Sep 10 21:34:00.736565 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:00.736539    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v86hl_33e1772c-e184-4b7d-89a8-05da5870b62c/dns-node-resolver/0.log"
Sep 10 21:34:01.285385 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:01.285329    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5776c87579-ddfvx_16069e45-d22c-453e-96b2-34941e28b05c/registry/0.log"
Sep 10 21:34:01.306261 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:01.306239    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7jnhf_be238cba-4fb7-4176-9a0a-cd247ac53feb/node-ca/0.log"
Sep 10 21:34:02.562697 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:02.562670    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-98c4dbc44-qkwf9_b5e5879b-760c-457b-88d4-6a1734f02c17/kube-auth-proxy/0.log"
Sep 10 21:34:03.588295 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:03.588266    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n29fp_80ad8dbd-fba2-4011-befe-6a1b7aa0c6b0/serve-healthcheck-canary/0.log"
Sep 10 21:34:04.161253 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:04.161224    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f74sn_a20b64fb-c649-43e7-ba2c-3238ed96520c/kube-rbac-proxy/0.log"
Sep 10 21:34:04.183339 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:04.183309    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f74sn_a20b64fb-c649-43e7-ba2c-3238ed96520c/exporter/0.log"
Sep 10 21:34:04.207043 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:04.207019    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f74sn_a20b64fb-c649-43e7-ba2c-3238ed96520c/extractor/0.log"
Sep 10 21:34:05.824325 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:05.824295    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2hcpt/perf-node-gather-daemonset-z8g2s"
Sep 10 21:34:06.803129 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:06.803093    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-769bdd6b4-mfblr_0556d7c8-57d9-4775-8532-1e260ea10416/manager/0.log"
Sep 10 21:34:06.825948 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:06.825920    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-769bdd6b4-mfblr_0556d7c8-57d9-4775-8532-1e260ea10416/copy-manifests/0.log"
Sep 10 21:34:08.326302 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:08.326272    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-799dfcfb97-kdnsc_8f7828f0-e34a-46df-aaff-59857058ebc3/manager/0.log"
Sep 10 21:34:09.061777 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:09.061749    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-98c4dbc44-qkwf9_b5e5879b-760c-457b-88d4-6a1734f02c17/kube-auth-proxy/0.log"
Sep 10 21:34:10.880217 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:10.880192    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-98c4dbc44-qkwf9_b5e5879b-760c-457b-88d4-6a1734f02c17/kube-auth-proxy/0.log"
Sep 10 21:34:13.572954 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:13.572925    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-r72gv_cb644a90-87a0-4942-8ec6-53fd20873973/migrator/0.log"
Sep 10 21:34:13.596495 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:13.596464    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-r72gv_cb644a90-87a0-4942-8ec6-53fd20873973/graceful-termination/0.log"
Sep 10 21:34:15.303822 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:15.303747    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hwvmr_97856763-2459-4493-9a2e-f34819281bc6/kube-multus-additional-cni-plugins/0.log"
Sep 10 21:34:15.325845 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:15.325798    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hwvmr_97856763-2459-4493-9a2e-f34819281bc6/egress-router-binary-copy/0.log"
Sep 10 21:34:15.354027 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:15.354007    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hwvmr_97856763-2459-4493-9a2e-f34819281bc6/cni-plugins/0.log"
Sep 10 21:34:15.378753 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:15.378735    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hwvmr_97856763-2459-4493-9a2e-f34819281bc6/bond-cni-plugin/0.log"
Sep 10 21:34:15.400215 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:15.400189    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hwvmr_97856763-2459-4493-9a2e-f34819281bc6/routeoverride-cni/0.log"
Sep 10 21:34:15.420604 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:15.420586    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hwvmr_97856763-2459-4493-9a2e-f34819281bc6/whereabouts-cni-bincopy/0.log"
Sep 10 21:34:15.442315 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:15.442294    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hwvmr_97856763-2459-4493-9a2e-f34819281bc6/whereabouts-cni/0.log"
Sep 10 21:34:15.531033 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:15.531008    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-p2jdt_8873b843-8d37-4612-93c0-57eedc38ea18/kube-multus/0.log"
Sep 10 21:34:15.636087 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:15.635973    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p9wtc_27f82b26-5156-4cbe-ac0c-23a386fa331b/network-metrics-daemon/0.log"
Sep 10 21:34:15.656742 ip-10-0-138-232 kubenswrapper[2572]: I0910 21:34:15.656708    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p9wtc_27f82b26-5156-4cbe-ac0c-23a386fa331b/kube-rbac-proxy/0.log"