Sep 09 20:09:11.688442 ip-10-0-134-84 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 20:09:11.688454 ip-10-0-134-84 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 20:09:11.688463 ip-10-0-134-84 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 20:09:11.688665 ip-10-0-134-84 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 20:09:21.782170 ip-10-0-134-84 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 20:09:21.782187 ip-10-0-134-84 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 2fc3a94d708f4fe0b1aa10154a175053 --
Sep 09 20:10:57.902749 ip-10-0-134-84 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 20:10:58.286520 ip-10-0-134-84 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:10:58.286520 ip-10-0-134-84 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 20:10:58.286520 ip-10-0-134-84 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:10:58.286520 ip-10-0-134-84 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 20:10:58.286520 ip-10-0-134-84 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:10:58.289276 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.289188    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 20:10:58.293395 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293380    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:10:58.293434 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293396    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:10:58.293434 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293401    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:10:58.293434 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293404    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:10:58.293434 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293407    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:10:58.293434 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293410    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:10:58.293434 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293413    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:10:58.293434 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293416    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:10:58.293434 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293419    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:10:58.293434 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293421    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:10:58.293434 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293424    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:10:58.293434 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293427    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:10:58.293434 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293429    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:10:58.293434 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293432    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:10:58.293434 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293434    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:10:58.293434 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293437    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:10:58.293434 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293440    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:10:58.293434 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293443    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:10:58.293852 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293446    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:10:58.293852 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293449    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:10:58.293852 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293451    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:10:58.293852 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293468    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:10:58.293852 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293476    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:10:58.293852 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293479    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:10:58.293852 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293482    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:10:58.293852 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293484    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:10:58.293852 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293487    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:10:58.293852 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293489    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:10:58.293852 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293492    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:10:58.293852 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293494    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:10:58.293852 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293497    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:10:58.293852 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293500    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:10:58.293852 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293502    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:10:58.293852 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293505    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:10:58.293852 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293507    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:10:58.293852 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293509    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:10:58.293852 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293512    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:10:58.293852 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293514    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:10:58.294360 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293517    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:10:58.294360 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293520    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:10:58.294360 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293522    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:10:58.294360 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293525    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:10:58.294360 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293528    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:10:58.294360 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293530    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:10:58.294360 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293533    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:10:58.294360 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293535    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:10:58.294360 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293538    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:10:58.294360 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293540    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:10:58.294360 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293543    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:10:58.294360 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293545    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:10:58.294360 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293548    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:10:58.294360 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293552    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:10:58.294360 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293554    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:10:58.294360 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293557    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:10:58.294360 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293560    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:10:58.294360 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293562    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:10:58.294360 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293565    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:10:58.294360 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293568    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:10:58.294864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293570    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:10:58.294864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293573    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:10:58.294864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293575    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:10:58.294864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293578    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:10:58.294864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293580    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:10:58.294864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293583    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:10:58.294864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293585    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:10:58.294864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293587    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:10:58.294864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293590    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:10:58.294864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293592    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:10:58.294864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293595    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:10:58.294864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293597    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:10:58.294864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293600    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:10:58.294864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293602    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:10:58.294864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293606    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:10:58.294864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293608    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:10:58.294864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293611    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:10:58.294864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293614    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:10:58.294864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293618    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:10:58.294864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293621    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:10:58.295337 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293625    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:10:58.295337 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293628    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:10:58.295337 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293630    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:10:58.295337 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293633    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:10:58.295337 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293636    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:10:58.295337 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293640    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:10:58.295337 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293643    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:10:58.295337 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293646    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:10:58.295337 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293648    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:10:58.295337 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293653    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:10:58.295337 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293657    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:10:58.295337 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293660    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:10:58.295337 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293662    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:10:58.295337 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293666    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:10:58.295337 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293670    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:10:58.295337 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293673    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:10:58.295337 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293676    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:10:58.295337 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293679    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:10:58.295337 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293682    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:10:58.295821 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293685    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:10:58.295821 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.293687    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:10:58.295821 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294359    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:10:58.295821 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294365    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:10:58.295821 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294368    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:10:58.295821 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294371    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:10:58.295821 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294374    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:10:58.295821 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294378    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:10:58.295821 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294381    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:10:58.295821 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294384    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:10:58.295821 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294387    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:10:58.295821 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294389    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:10:58.295821 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294392    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:10:58.295821 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294394    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:10:58.295821 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294397    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:10:58.295821 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294400    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:10:58.295821 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294402    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:10:58.295821 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294405    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:10:58.295821 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294410    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:10:58.296303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294413    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:10:58.296303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294416    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:10:58.296303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294419    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:10:58.296303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294422    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:10:58.296303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294426    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:10:58.296303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294430    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:10:58.296303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294433    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:10:58.296303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294436    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:10:58.296303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294438    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:10:58.296303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294441    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:10:58.296303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294443    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:10:58.296303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294446    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:10:58.296303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294448    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:10:58.296303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294451    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:10:58.296303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294467    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:10:58.296303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294471    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:10:58.296303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294473    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:10:58.296303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294476    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:10:58.296303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294478    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:10:58.296303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294481    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:10:58.296805 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294484    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:10:58.296805 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294487    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:10:58.296805 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294490    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:10:58.296805 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294493    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:10:58.296805 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294495    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:10:58.296805 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294498    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:10:58.296805 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294500    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:10:58.296805 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294503    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:10:58.296805 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294511    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:10:58.296805 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294514    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:10:58.296805 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294517    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:10:58.296805 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294520    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:10:58.296805 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294522    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:10:58.296805 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294525    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:10:58.296805 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294527    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:10:58.296805 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294530    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:10:58.296805 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294533    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:10:58.296805 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294535    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:10:58.296805 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294538    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:10:58.297303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294540    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:10:58.297303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294543    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:10:58.297303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294546    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:10:58.297303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294548    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:10:58.297303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294551    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:10:58.297303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294553    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:10:58.297303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294556    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:10:58.297303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294559    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:10:58.297303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294561    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:10:58.297303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294564    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:10:58.297303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294566    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:10:58.297303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294569    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:10:58.297303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294571    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:10:58.297303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294574    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:10:58.297303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294577    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:10:58.297303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294582    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:10:58.297303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294586    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:10:58.297303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294590    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:10:58.297303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294593    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:10:58.297303 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294596    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:10:58.297807 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294598    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:10:58.297807 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294601    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:10:58.297807 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294604    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:10:58.297807 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294607    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:10:58.297807 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294610    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:10:58.297807 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294612    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:10:58.297807 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294615    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:10:58.297807 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294617    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:10:58.297807 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294620    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:10:58.297807 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294622    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:10:58.297807 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294625    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:10:58.297807 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294627    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:10:58.297807 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294630    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:10:58.297807 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294632    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:10:58.297807 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294635    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:10:58.297807 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294637    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:10:58.297807 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294640    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:10:58.297807 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294642    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:10:58.297807 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294645    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:10:58.297807 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294647    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:10:58.298368 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294650    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:10:58.298368 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294652    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:10:58.298368 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.294655    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:10:58.298368 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295039    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:10:58.298368 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295045    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:10:58.298368 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295048    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:10:58.298368 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295051    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:10:58.298368 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295056    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:10:58.298368 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295059    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:10:58.298368 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295063    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:10:58.298368 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295065    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:10:58.298368 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295068    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:10:58.298368 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295070    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:10:58.298368 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295073    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:10:58.298368 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295076    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:10:58.298368 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295078    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:10:58.298368 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295081    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:10:58.298368 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295083    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:10:58.298368 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295086    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:10:58.298368 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295089    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:10:58.298864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295091    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:10:58.298864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295094    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:10:58.298864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295096    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:10:58.298864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295098    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:10:58.298864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295101    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:10:58.298864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295104    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:10:58.298864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295107    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:10:58.298864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295109    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:10:58.298864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295112    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:10:58.298864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295114    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:10:58.298864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295117    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:10:58.298864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295120    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:10:58.298864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295122    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:10:58.298864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295125    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:10:58.298864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295127    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:10:58.298864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295130    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:10:58.298864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295133    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:10:58.298864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295135    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:10:58.298864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295138    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:10:58.298864 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295141    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:10:58.299358 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295144    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:10:58.299358 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295147    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:10:58.299358 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295150    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:10:58.299358 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295152    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:10:58.299358 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295155    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:10:58.299358 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295158    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:10:58.299358 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295160    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:10:58.299358 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295162    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:10:58.299358 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295167    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:10:58.299358 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295170    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:10:58.299358 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295174    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:10:58.299358 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295178    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:10:58.299358 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295183    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:10:58.299358 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295186    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:10:58.299358 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295189    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:10:58.299358 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295191    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:10:58.299358 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295194    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:10:58.299358 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295196    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:10:58.299358 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295199    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:10:58.299834 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295202    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:10:58.299834 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295204    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:10:58.299834 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295207    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:10:58.299834 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295209    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:10:58.299834 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295211    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:10:58.299834 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295214    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:10:58.299834 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295217    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:10:58.299834 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295219    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:10:58.299834 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295222    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:10:58.299834 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295224    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:10:58.299834 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295227    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:10:58.299834 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295230    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:10:58.299834 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295233    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:10:58.299834 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295236    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:10:58.299834 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295239    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:10:58.299834 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295243    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:10:58.299834 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295247    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:10:58.299834 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295250    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:10:58.299834 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295253    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:10:58.300336 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295256    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:10:58.300336 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295258    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:10:58.300336 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295263    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:10:58.300336 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295265    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:10:58.300336 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295268    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:10:58.300336 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295270    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:10:58.300336 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295273    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:10:58.300336 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295275    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:10:58.300336 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295278    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:10:58.300336 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295280    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:10:58.300336 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295283    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:10:58.300336 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295285    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:10:58.300336 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295288    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:10:58.300336 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295290    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:10:58.300336 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295293    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:10:58.300336 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295295    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:10:58.300336 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295298    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:10:58.300336 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295301    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:10:58.300336 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295303    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:10:58.300336 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295306    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:10:58.300832 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295308    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:10:58.300832 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295311    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:10:58.300832 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295313    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:10:58.300832 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.295317    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:10:58.300832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295389    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 20:10:58.300832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295421    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 20:10:58.300832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295427    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 20:10:58.300832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295431    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 20:10:58.300832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295436    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 20:10:58.300832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295439    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 20:10:58.300832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295444    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 20:10:58.300832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295448    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 20:10:58.300832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295451    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 20:10:58.300832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295472    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 20:10:58.300832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295480    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 20:10:58.300832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295485    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 20:10:58.300832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295491    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 20:10:58.300832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295494    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 09 20:10:58.300832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295497    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 20:10:58.300832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295500    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 09 20:10:58.300832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295503    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 20:10:58.300832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295506    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 20:10:58.300832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295511    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 09 20:10:58.300832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295514    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 20:10:58.301406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295517    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 20:10:58.301406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295520    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 20:10:58.301406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295524    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 20:10:58.301406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295528    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 20:10:58.301406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295531    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 20:10:58.301406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295534    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 20:10:58.301406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295538    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 20:10:58.301406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295541    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 20:10:58.301406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295544    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 20:10:58.301406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295547    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 20:10:58.301406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295550    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 20:10:58.301406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295555    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 20:10:58.301406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295559    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 20:10:58.301406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295562    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 20:10:58.301406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295565    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 20:10:58.301406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295567    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 20:10:58.301406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295571    2581 flags.go:64] FLAG: --enable-server="true"
Sep 09 20:10:58.301406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295574    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 20:10:58.301406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295578    2581 flags.go:64] FLAG: --event-burst="100"
Sep 09 20:10:58.301406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295582    2581 flags.go:64] FLAG: --event-qps="50"
Sep 09 20:10:58.301406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295585    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 20:10:58.301406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295588    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 20:10:58.301406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295592    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 09 20:10:58.301406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295595    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 20:10:58.301985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295598    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 20:10:58.301985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295601    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 20:10:58.301985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295604    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 09 20:10:58.301985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295607    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 20:10:58.301985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295610    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 20:10:58.301985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295613    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 20:10:58.301985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295616    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 20:10:58.301985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295618    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 20:10:58.301985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295621    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 20:10:58.301985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295624    2581 flags.go:64] FLAG: --feature-gates=""
Sep 09 20:10:58.301985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295628    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 20:10:58.301985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295631    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 20:10:58.301985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295634    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 20:10:58.301985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295637    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 20:10:58.301985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295640    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 20:10:58.301985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295643    2581 flags.go:64] FLAG: --help="false"
Sep 09 20:10:58.301985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295646    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-134-84.ec2.internal"
Sep 09 20:10:58.301985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295649    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 20:10:58.301985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295652    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 20:10:58.301985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295656    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 20:10:58.301985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295659    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 20:10:58.301985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295662    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 20:10:58.301985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295665    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 20:10:58.301985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295668    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295670    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295673    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295676    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295679    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295682    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295685    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295689    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295692    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295695    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295698    2581 flags.go:64] FLAG: --lock-file=""
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295700    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295703    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295706    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295711    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295714    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295717    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295720    2581 flags.go:64] FLAG: --logging-format="text"
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295723    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295726    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295729    2581 flags.go:64] FLAG: --manifest-url=""
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295731    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295736    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295739    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295743    2581 flags.go:64] FLAG: --max-pods="110"
Sep 09 20:10:58.302673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295746    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 20:10:58.303290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295749    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 20:10:58.303290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295752    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 20:10:58.303290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295756    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 20:10:58.303290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295759    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 20:10:58.303290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295762    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 20:10:58.303290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295764    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 20:10:58.303290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295771    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 20:10:58.303290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295774    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 20:10:58.303290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295777    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 20:10:58.303290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295780    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 09 20:10:58.303290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295784    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 20:10:58.303290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295789    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 20:10:58.303290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295792    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 20:10:58.303290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295796    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 20:10:58.303290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295799    2581 flags.go:64] FLAG: --port="10250"
Sep 09 20:10:58.303290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295802    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 20:10:58.303290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295805    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-027c1fa0758e30f39"
Sep 09 20:10:58.303290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295808    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 09 20:10:58.303290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295811    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 20:10:58.303290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295814    2581 flags.go:64] FLAG: --register-node="true"
Sep 09 20:10:58.303290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295817    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 09 20:10:58.303290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295821    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 09 20:10:58.303290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295824    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 09 20:10:58.303290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295827    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295830    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295834    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295837    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295840    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295844    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295846    2581 flags.go:64] FLAG: --runonce="false"
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295849    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295852    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295855    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295858    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295861    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295865    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295868    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295872    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295875    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295877    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295880    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295883    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295886    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295890    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295893    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295899    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295902    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295904    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 20:10:58.303885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295909    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 09 20:10:58.304505 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295917    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 20:10:58.304505 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295919    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 20:10:58.304505 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295922    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 20:10:58.304505 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295925    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 20:10:58.304505 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295928    2581 flags.go:64] FLAG: --v="2"
Sep 09 20:10:58.304505 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295937    2581 flags.go:64] FLAG: --version="false"
Sep 09 20:10:58.304505 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295941    2581 flags.go:64] FLAG: --vmodule=""
Sep 09 20:10:58.304505 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295945    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 20:10:58.304505 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.295948    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 20:10:58.304505 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296069    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:10:58.304505 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296073    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:10:58.304505 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296076    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:10:58.304505 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296079    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:10:58.304505 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296083    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:10:58.304505 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296087    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:10:58.304505 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296090    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:10:58.304505 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296092    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:10:58.304505 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296095    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:10:58.304505 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296100    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:10:58.304505 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296103    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:10:58.304505 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296106    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:10:58.305020 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296110    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:10:58.305020 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296113    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:10:58.305020 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296116    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:10:58.305020 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296119    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:10:58.305020 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296122    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:10:58.305020 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296125    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:10:58.305020 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296128    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:10:58.305020 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296131    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:10:58.305020 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296134    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:10:58.305020 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296136    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:10:58.305020 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296139    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:10:58.305020 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296142    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:10:58.305020 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296145    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:10:58.305020 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296148    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:10:58.305020 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296150    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:10:58.305020 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296153    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:10:58.305020 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296156    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:10:58.305020 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296158    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:10:58.305020 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296161    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:10:58.305492 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296163    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:10:58.305492 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296166    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:10:58.305492 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296168    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:10:58.305492 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296171    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:10:58.305492 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296173    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:10:58.305492 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296176    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:10:58.305492 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296179    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:10:58.305492 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296181    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:10:58.305492 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296184    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:10:58.305492 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296186    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:10:58.305492 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296190    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:10:58.305492 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296192    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:10:58.305492 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296195    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:10:58.305492 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296198    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:10:58.305492 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296200    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:10:58.305492 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296203    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:10:58.305492 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296205    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:10:58.305492 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296208    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:10:58.305492 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296210    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:10:58.305492 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296213    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:10:58.305976 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296216    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:10:58.305976 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296219    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:10:58.305976 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296221    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:10:58.305976 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296224    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:10:58.305976 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296227    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:10:58.305976 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296229    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:10:58.305976 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296231    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:10:58.305976 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296234    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:10:58.305976 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296236    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:10:58.305976 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296239    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:10:58.305976 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296241    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:10:58.305976 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296244    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:10:58.305976 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296246    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:10:58.305976 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296249    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:10:58.305976 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296251    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:10:58.305976 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296254    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:10:58.305976 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296256    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:10:58.305976 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296259    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:10:58.305976 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296262    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:10:58.305976 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296264    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:10:58.306508 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296267    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:10:58.306508 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296269    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:10:58.306508 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296273    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:10:58.306508 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296275    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:10:58.306508 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296278    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:10:58.306508 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296280    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:10:58.306508 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296283    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:10:58.306508 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296285    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:10:58.306508 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296288    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:10:58.306508 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296290    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:10:58.306508 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296293    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:10:58.306508 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296296    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:10:58.306508 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296298    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:10:58.306508 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296301    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:10:58.306508 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296304    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:10:58.306508 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296308    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:10:58.306508 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296311    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:10:58.306508 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296314    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:10:58.306508 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296317    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:10:58.306508 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296321    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:10:58.307001 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296323    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:10:58.307001 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296326    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:10:58.307001 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296329    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:10:58.307001 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296331    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:10:58.307001 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296334    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:10:58.307001 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296336    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:10:58.307001 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296339    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:10:58.307001 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.296341    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:10:58.307001 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.296346    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:10:58.307001 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.304126    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 20:10:58.307001 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.304141    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 20:10:58.307001 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.304160    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 20:10:58.307001 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.304166    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 20:10:58.307001 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304235    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:10:58.307001 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304240    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:10:58.307369 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304244    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:10:58.307369 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304247    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:10:58.307369 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304250    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:10:58.307369 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304253    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:10:58.307369 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304256    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:10:58.307369 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304259    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:10:58.307369 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304262    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:10:58.307369 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304265    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:10:58.307369 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304267    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:10:58.307369 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304270    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:10:58.307369 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304278    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:10:58.307369 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304282    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:10:58.307369 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304286    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:10:58.307369 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304289    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:10:58.307369 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304292    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:10:58.307369 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304296    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:10:58.307369 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304300    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:10:58.307369 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304303    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:10:58.307369 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304306    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:10:58.307903 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304309    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:10:58.307903 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304312    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:10:58.307903 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304315    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:10:58.307903 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304318    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:10:58.307903 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304321    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:10:58.307903 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304324    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:10:58.307903 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304327    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:10:58.307903 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304329    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:10:58.307903 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304333    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:10:58.307903 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304336    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:10:58.307903 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304339    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:10:58.307903 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304342    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:10:58.307903 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304345    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:10:58.307903 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304347    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:10:58.307903 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304350    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:10:58.307903 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304352    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:10:58.307903 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304355    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:10:58.307903 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304361    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:10:58.307903 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304364    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:10:58.308381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304366    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:10:58.308381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304369    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:10:58.308381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304371    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:10:58.308381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304374    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:10:58.308381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304376    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:10:58.308381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304384    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:10:58.308381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304388    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:10:58.308381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304392    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:10:58.308381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304395    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:10:58.308381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304398    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:10:58.308381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304400    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:10:58.308381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304403    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:10:58.308381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304406    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:10:58.308381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304408    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:10:58.308381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304411    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:10:58.308381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304413    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:10:58.308381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304416    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:10:58.308381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304419    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:10:58.308381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304421    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:10:58.309088 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304424    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:10:58.309088 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304426    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:10:58.309088 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304430    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:10:58.309088 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304433    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:10:58.309088 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304435    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:10:58.309088 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304438    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:10:58.309088 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304440    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:10:58.309088 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304443    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:10:58.309088 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304446    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:10:58.309088 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304449    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:10:58.309088 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304452    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:10:58.309088 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304466    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:10:58.309088 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304470    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:10:58.309088 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304472    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:10:58.309088 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304475    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:10:58.309088 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304477    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:10:58.309088 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304480    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:10:58.309088 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304483    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:10:58.309088 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304485    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:10:58.309088 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304496    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:10:58.309088 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304499    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:10:58.309621 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304502    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:10:58.309621 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304505    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:10:58.309621 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304507    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:10:58.309621 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304510    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:10:58.309621 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304513    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:10:58.309621 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304515    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:10:58.309621 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304518    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:10:58.309621 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304521    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:10:58.309621 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304523    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:10:58.309621 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304526    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:10:58.309621 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304529    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:10:58.309621 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304531    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:10:58.309621 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304534    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:10:58.309621 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304537    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:10:58.309621 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304540    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:10:58.309621 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304542    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:10:58.309621 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304545    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:10:58.309621 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304548    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:10:58.309621 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304551    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:10:58.310215 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.304561    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:10:58.310215 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304722    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:10:58.310215 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304727    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:10:58.310215 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304730    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:10:58.310215 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304733    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:10:58.310215 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304736    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:10:58.310215 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304739    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:10:58.310215 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304742    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:10:58.310215 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304744    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:10:58.310215 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304747    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:10:58.310215 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304750    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:10:58.310215 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304752    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:10:58.310215 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304760    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:10:58.310215 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304763    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:10:58.310215 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304766    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:10:58.310609 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304768    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:10:58.310609 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304771    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:10:58.310609 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304773    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:10:58.310609 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304776    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:10:58.310609 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304779    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:10:58.310609 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304781    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:10:58.310609 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304784    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:10:58.310609 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304786    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:10:58.310609 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304789    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:10:58.310609 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304792    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:10:58.310609 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304794    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:10:58.310609 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304797    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:10:58.310609 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304800    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:10:58.310609 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304803    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:10:58.310609 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304805    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:10:58.310609 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304808    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:10:58.310609 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304810    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:10:58.310609 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304813    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:10:58.310609 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304816    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:10:58.310609 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304819    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:10:58.311322 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304823    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:10:58.311322 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304826    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:10:58.311322 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304828    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:10:58.311322 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304831    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:10:58.311322 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304834    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:10:58.311322 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304836    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:10:58.311322 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304839    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:10:58.311322 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304841    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:10:58.311322 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304844    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:10:58.311322 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304846    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:10:58.311322 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304854    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:10:58.311322 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304857    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:10:58.311322 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304861    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:10:58.311322 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304864    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:10:58.311322 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304867    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:10:58.311322 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304870    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:10:58.311322 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304873    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:10:58.311322 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304876    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:10:58.311322 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304879    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:10:58.311926 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304881    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:10:58.311926 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304884    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:10:58.311926 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304887    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:10:58.311926 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304889    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:10:58.311926 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304892    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:10:58.311926 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304894    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:10:58.311926 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304897    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:10:58.311926 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304900    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:10:58.311926 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304902    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:10:58.311926 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304905    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:10:58.311926 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304907    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:10:58.311926 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304910    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:10:58.311926 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304913    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:10:58.311926 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304915    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:10:58.311926 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304919    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:10:58.311926 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304922    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:10:58.311926 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304925    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:10:58.311926 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304928    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:10:58.311926 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304930    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:10:58.312667 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304933    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:10:58.312667 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304936    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:10:58.312667 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304939    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:10:58.312667 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304941    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:10:58.312667 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304943    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:10:58.312667 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304951    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:10:58.312667 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304954    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:10:58.312667 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304957    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:10:58.312667 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304959    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:10:58.312667 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304962    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:10:58.312667 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304964    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:10:58.312667 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304968    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:10:58.312667 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304970    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:10:58.312667 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304973    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:10:58.312667 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304975    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:10:58.312667 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304977    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:10:58.312667 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304980    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:10:58.312667 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304983    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:10:58.312667 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304985    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:10:58.312667 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304988    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:10:58.313381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304990    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:10:58.313381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304993    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:10:58.313381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304995    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:10:58.313381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.304998    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:10:58.313381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.305000    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:10:58.313381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.305003    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:10:58.313381 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:58.305005    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:10:58.313381 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.305010    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:10:58.313381 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.305147    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 20:10:58.313381 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.307513    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 20:10:58.313381 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.308595    2581 server.go:1019] "Starting client certificate rotation"
Sep 09 20:10:58.313381 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.308716    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:10:58.313381 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.308760    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:10:58.313381 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.311478    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:10:58.317574 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.317551    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:10:58.329069 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.329050    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 09 20:10:58.332569 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.332550    2581 log.go:25] "Validated CRI v1 image API"
Sep 09 20:10:58.333316 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.333301    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 20:10:58.339636 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.339620    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:10:58.340997 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.340975    2581 fs.go:135] Filesystem UUIDs: map[316ecd15-5a34-461e-a020-887a057c4cf2:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 824ad108-6902-4860-86fa-91922a68720f:/dev/nvme0n1p4]
Sep 09 20:10:58.341038 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.340997    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 20:10:58.346154 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.346039    2581 manager.go:216] Machine: {Timestamp:2026-09-09 20:10:58.344609569 +0000 UTC m=+0.349786555 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100293 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a75ae0de158ec919e98f2c7409604 SystemUUID:ec2a75ae-0de1-58ec-919e-98f2c7409604 BootID:2fc3a94d-708f-4fe0-b1aa-10154a175053 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:30:69:45:d5:ff Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:30:69:45:d5:ff Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:75:95:a5:c4:16 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 20:10:58.346154 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.346150    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 20:10:58.346277 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.346263    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 20:10:58.349192 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.349165    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 20:10:58.349320 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.349195    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-84.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 20:10:58.350024 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.350014    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 20:10:58.350056 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.350025    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 20:10:58.350056 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.350033    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:10:58.350056 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.350047    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:10:58.352412 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.352402    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 20:10:58.355872 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.355861    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:10:58.355985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.355976    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 20:10:58.356025 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.356012    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 20:10:58.356025 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.356022    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 20:10:58.356075 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.356039    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 20:10:58.356075 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.356048    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 09 20:10:58.356075 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.356060    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 20:10:58.356438 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.356427    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:10:58.356499 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.356444    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:10:58.359025 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.359010    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 20:10:58.363010 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.362991    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 20:10:58.363096 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.363041    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 20:10:58.363532 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.363519    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 20:10:58.363582 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.363537    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 20:10:58.363582 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.363544    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 20:10:58.363582 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.363550    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 20:10:58.363582 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.363557    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 20:10:58.363582 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.363563    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 20:10:58.364066 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.364057    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 20:10:58.364066 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.364066    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 20:10:58.364126 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.364073    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 20:10:58.364126 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.364079    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 20:10:58.364126 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.364087    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 20:10:58.364126 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.364095    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 20:10:58.364941 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.364931    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 20:10:58.364941 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.364942    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 20:10:58.368342 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.368324    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-84.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 20:10:58.368965 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.368951    2581 server.go:1268] "Started kubelet"
Sep 09 20:10:58.369030 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:58.368984    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 20:10:58.369030 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.369003    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 20:10:58.369117 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.369037    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 20:10:58.369117 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.369097    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 20:10:58.369254 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:58.369061    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-84.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 20:10:58.369717 ip-10-0-134-84 systemd[1]: Started Kubernetes Kubelet.
Sep 09 20:10:58.371309 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.370580    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 20:10:58.371309 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.370886    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 20:10:58.373954 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:58.373933    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 20:10:58.376500 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.376483    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 20:10:58.377901 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.377184    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 20:10:58.378030 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:58.376778    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-84.ec2.internal.18d3c0117eb3fe03  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-84.ec2.internal,UID:ip-10-0-134-84.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-84.ec2.internal,},FirstTimestamp:2026-09-09 20:10:58.368921091 +0000 UTC m=+0.374098077,LastTimestamp:2026-09-09 20:10:58.368921091 +0000 UTC m=+0.374098077,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-84.ec2.internal,}"
Sep 09 20:10:58.378095 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:58.378059    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-84.ec2.internal\" not found"
Sep 09 20:10:58.378328 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.378287    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 20:10:58.378328 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.378323    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 20:10:58.378446 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.378353    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 20:10:58.378446 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.378394    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 20:10:58.378446 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.378402    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 20:10:58.378956 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.378937    2581 factory.go:55] Registering systemd factory
Sep 09 20:10:58.379029 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.378978    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 09 20:10:58.379223 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.379210    2581 factory.go:153] Registering CRI-O factory
Sep 09 20:10:58.379304 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.379227    2581 factory.go:223] Registration of the crio container factory successfully
Sep 09 20:10:58.379304 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.379275    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 20:10:58.379304 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.379296    2581 factory.go:103] Registering Raw factory
Sep 09 20:10:58.379437 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.379313    2581 manager.go:1196] Started watching for new ooms in manager
Sep 09 20:10:58.379862 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.379845    2581 manager.go:349] Starting recovery of all containers
Sep 09 20:10:58.384115 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:58.384090    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 20:10:58.384217 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:58.384194    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-84.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 20:10:58.392441 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.392422    2581 manager.go:354] Recovery completed
Sep 09 20:10:58.394113 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.394093    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ckk42"
Sep 09 20:10:58.397105 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.397093    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:10:58.398850 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.398831    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:10:58.398927 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.398862    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:10:58.398927 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.398875    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:10:58.399427 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.399414    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:10:58.399503 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.399414    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 20:10:58.399503 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.399447    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 20:10:58.399503 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.399479    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:10:58.400089 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.400067    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ckk42"
Sep 09 20:10:58.400185 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.400074    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:10:58.400185 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.400162    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:10:58.400185 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.400181    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:10:58.400912 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.400896    2581 policy_none.go:49] "None policy: Start"
Sep 09 20:10:58.400912 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.400915    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 20:10:58.401050 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.400935    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 20:10:58.401135 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:58.401065    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-84.ec2.internal.18d3c011807ca777  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-84.ec2.internal,UID:ip-10-0-134-84.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-84.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-84.ec2.internal,},FirstTimestamp:2026-09-09 20:10:58.398848887 +0000 UTC m=+0.404025872,LastTimestamp:2026-09-09 20:10:58.398848887 +0000 UTC m=+0.404025872,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-84.ec2.internal,}"
Sep 09 20:10:58.401999 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.401986    2581 policy_none.go:47] "Start"
Sep 09 20:10:58.428249 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.428228    2581 manager.go:337] "Starting Device Plugin manager"
Sep 09 20:10:58.444378 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:58.428280    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 20:10:58.444378 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.428298    2581 server.go:87] "Starting device plugin registration server"
Sep 09 20:10:58.444378 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.428598    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 20:10:58.444378 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.428609    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 20:10:58.444378 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.428856    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 20:10:58.444378 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.429282    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 20:10:58.444378 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.429295    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 20:10:58.444378 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:58.429586    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 20:10:58.444378 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:58.429700    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-84.ec2.internal\" not found"
Sep 09 20:10:58.529664 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.529631    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:10:58.531357 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.531330    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:10:58.531494 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.531365    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:10:58.531494 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.531381    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:10:58.531494 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.531413    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-84.ec2.internal"
Sep 09 20:10:58.539002 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.538917    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-84.ec2.internal"
Sep 09 20:10:58.539002 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:58.538941    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-84.ec2.internal\": node \"ip-10-0-134-84.ec2.internal\" not found"
Sep 09 20:10:58.558245 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:58.558220    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-84.ec2.internal\" not found"
Sep 09 20:10:58.627574 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.627542    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 20:10:58.628800 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.628783    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 20:10:58.628866 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.628843    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 20:10:58.628925 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.628869    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 20:10:58.628925 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:58.628913    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 20:10:58.631139 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.631120    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:10:58.658445 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:58.658411    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-84.ec2.internal\" not found"
Sep 09 20:10:58.729892 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.729863    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-84.ec2.internal"]
Sep 09 20:10:58.730005 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.729948    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:10:58.731196 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.731179    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:10:58.731261 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.731208    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:10:58.731261 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.731218    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:10:58.732293 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.732282    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:10:58.732416 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.732402    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal"
Sep 09 20:10:58.732474 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.732437    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:10:58.733217 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.733202    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:10:58.733294 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.733230    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:10:58.733294 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.733243    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:10:58.733514 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.733499    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:10:58.733567 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.733527    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:10:58.733567 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.733538    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:10:58.734601 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.734589    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-84.ec2.internal"
Sep 09 20:10:58.734647 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.734613    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:10:58.735826 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.735810    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:10:58.735878 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.735840    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:10:58.735878 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.735851    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:10:58.759025 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:58.759002    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-84.ec2.internal\" not found"
Sep 09 20:10:58.759131 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:58.759098    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-84.ec2.internal\" not found" node="ip-10-0-134-84.ec2.internal"
Sep 09 20:10:58.763548 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:58.763534    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-84.ec2.internal\" not found" node="ip-10-0-134-84.ec2.internal"
Sep 09 20:10:58.780548 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.780533    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8226d39d61364b61ba5e1ad9f7514144-config\") pod \"kube-apiserver-proxy-ip-10-0-134-84.ec2.internal\" (UID: \"8226d39d61364b61ba5e1ad9f7514144\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-84.ec2.internal"
Sep 09 20:10:58.780617 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.780556    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45419fbd63c226915cb25a2cb6311f7e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal\" (UID: \"45419fbd63c226915cb25a2cb6311f7e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal"
Sep 09 20:10:58.780617 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.780589    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45419fbd63c226915cb25a2cb6311f7e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal\" (UID: \"45419fbd63c226915cb25a2cb6311f7e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal"
Sep 09 20:10:58.859333 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:58.859268    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-84.ec2.internal\" not found"
Sep 09 20:10:58.881619 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.881595    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45419fbd63c226915cb25a2cb6311f7e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal\" (UID: \"45419fbd63c226915cb25a2cb6311f7e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal"
Sep 09 20:10:58.881682 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.881627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45419fbd63c226915cb25a2cb6311f7e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal\" (UID: \"45419fbd63c226915cb25a2cb6311f7e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal"
Sep 09 20:10:58.881682 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.881647    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8226d39d61364b61ba5e1ad9f7514144-config\") pod \"kube-apiserver-proxy-ip-10-0-134-84.ec2.internal\" (UID: \"8226d39d61364b61ba5e1ad9f7514144\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-84.ec2.internal"
Sep 09 20:10:58.881748 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.881684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8226d39d61364b61ba5e1ad9f7514144-config\") pod \"kube-apiserver-proxy-ip-10-0-134-84.ec2.internal\" (UID: \"8226d39d61364b61ba5e1ad9f7514144\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-84.ec2.internal"
Sep 09 20:10:58.881748 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.881684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45419fbd63c226915cb25a2cb6311f7e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal\" (UID: \"45419fbd63c226915cb25a2cb6311f7e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal"
Sep 09 20:10:58.881748 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:58.881724    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45419fbd63c226915cb25a2cb6311f7e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal\" (UID: \"45419fbd63c226915cb25a2cb6311f7e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal"
Sep 09 20:10:58.959802 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:58.959773    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-84.ec2.internal\" not found"
Sep 09 20:10:59.060253 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:59.060204    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-84.ec2.internal\" not found"
Sep 09 20:10:59.061307 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:59.061291    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal"
Sep 09 20:10:59.066049 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:59.066033    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-84.ec2.internal"
Sep 09 20:10:59.160924 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:59.160833    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-84.ec2.internal\" not found"
Sep 09 20:10:59.267348 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:59.267314    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-84.ec2.internal\" not found"
Sep 09 20:10:59.291808 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:59.291782    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:10:59.308889 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:59.308870    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 20:10:59.309002 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:59.308981    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:10:59.309062 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:59.309034    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:10:59.367731 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:59.367696    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-84.ec2.internal\" not found"
Sep 09 20:10:59.404530 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:59.404496    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 20:05:58 +0000 UTC" deadline="2028-05-23 17:16:46.071414842 +0000 UTC"
Sep 09 20:10:59.404530 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:59.404525    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14925h5m46.666892905s"
Sep 09 20:10:59.468250 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:10:59.468043    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-84.ec2.internal\" not found"
Sep 09 20:10:59.468915 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:59.468896    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:10:59.479029 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:59.479010    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal"
Sep 09 20:10:59.493935 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:59.493910    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:10:59.495556 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:59.495513    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-84.ec2.internal"
Sep 09 20:10:59.504530 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:59.504500    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:10:59.573454 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:59.573420    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8226d39d61364b61ba5e1ad9f7514144.slice/crio-a7c9c143fe16e094449180add29d476aa4d0af215304dce23c75efde45483665 WatchSource:0}: Error finding container a7c9c143fe16e094449180add29d476aa4d0af215304dce23c75efde45483665: Status 404 returned error can't find the container with id a7c9c143fe16e094449180add29d476aa4d0af215304dce23c75efde45483665
Sep 09 20:10:59.573826 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:10:59.573810    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45419fbd63c226915cb25a2cb6311f7e.slice/crio-7682431b6551010023bf256538e4e7f136fbdeb3e295a7a1e222092c57b64bce WatchSource:0}: Error finding container 7682431b6551010023bf256538e4e7f136fbdeb3e295a7a1e222092c57b64bce: Status 404 returned error can't find the container with id 7682431b6551010023bf256538e4e7f136fbdeb3e295a7a1e222092c57b64bce
Sep 09 20:10:59.580060 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:59.580043    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:10:59.632667 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:59.632615    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-84.ec2.internal" event={"ID":"8226d39d61364b61ba5e1ad9f7514144","Type":"ContainerStarted","Data":"a7c9c143fe16e094449180add29d476aa4d0af215304dce23c75efde45483665"}
Sep 09 20:10:59.633664 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:59.633641    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal" event={"ID":"45419fbd63c226915cb25a2cb6311f7e","Type":"ContainerStarted","Data":"7682431b6551010023bf256538e4e7f136fbdeb3e295a7a1e222092c57b64bce"}
Sep 09 20:10:59.712660 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:10:59.712635    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:00.349839 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.349806    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:00.356968 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.356943    2581 apiserver.go:52] "Watching apiserver"
Sep 09 20:11:00.365910 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.365884    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 20:11:00.366496 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.366461    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-lk6j8","openshift-ovn-kubernetes/ovnkube-node-jqqq8","kube-system/global-pull-secret-syncer-z8q9m","kube-system/kube-apiserver-proxy-ip-10-0-134-84.ec2.internal","openshift-cluster-node-tuning-operator/tuned-4k95l","openshift-dns/node-resolver-l7rwd","openshift-image-registry/node-ca-qpmd2","openshift-multus/multus-additional-cni-plugins-6hqtr","openshift-multus/network-metrics-daemon-9vfht","openshift-network-diagnostics/network-check-target-zcrsj","kube-system/konnectivity-agent-f28rc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal","openshift-multus/multus-f7kxd"]
Sep 09 20:11:00.368148 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.368125    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:00.368259 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:00.368217    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcrsj" podUID="6b9953bc-b5e1-4b30-a926-c1a13d4677a7"
Sep 09 20:11:00.369270 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.369247    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qpmd2"
Sep 09 20:11:00.370759 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.370726    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.372150 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.371938    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lk6j8"
Sep 09 20:11:00.375106 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.372827    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 20:11:00.375106 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.372847    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 20:11:00.375106 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.373159    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9vskd\""
Sep 09 20:11:00.375106 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.373491    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 20:11:00.375106 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.373753    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 20:11:00.375106 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.373941    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 20:11:00.375106 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.374143    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 20:11:00.375106 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.374324    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 20:11:00.375106 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.374587    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.375106 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.374649    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 20:11:00.375106 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.374867    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:11:00.376491 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.376470    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dbjdx\""
Sep 09 20:11:00.376853 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.376830    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-chlqn\""
Sep 09 20:11:00.378928 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.377831    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.378928 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.378123    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 20:11:00.378928 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.378348    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7kqgk\""
Sep 09 20:11:00.378928 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.378641    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 20:11:00.380024 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.379997    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:11:00.380262 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.380246    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:00.380769 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:00.380736    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vfht" podUID="7e3d0490-5d57-42a1-858f-1e6bf8830f4c"
Sep 09 20:11:00.381109 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.381089    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 20:11:00.381216 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.381140    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 20:11:00.381406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.381389    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 20:11:00.381770 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.381740    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f28rc"
Sep 09 20:11:00.381968 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.381948    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l7rwd"
Sep 09 20:11:00.382351 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.382328    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m6f54\""
Sep 09 20:11:00.382878 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.382858    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 20:11:00.383739 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.383716    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 20:11:00.383951 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.383934    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 20:11:00.384120 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.384063    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 20:11:00.384190 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.384154    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm"
Sep 09 20:11:00.384671 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.384611    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 20:11:00.386246 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.385434    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6hqtr"
Sep 09 20:11:00.387380 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.386472    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 20:11:00.387380 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.386542    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2fwlt\""
Sep 09 20:11:00.387380 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.386653    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wt9jn\""
Sep 09 20:11:00.387380 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.386818    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 20:11:00.387380 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.386869    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 20:11:00.387380 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.387143    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 20:11:00.387380 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.387220    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 20:11:00.387380 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.387290    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 20:11:00.387380 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.387291    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b946f\""
Sep 09 20:11:00.388558 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.388332    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 20:11:00.392525 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.389185    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 20:11:00.392525 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.389269    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:00.392525 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:00.389328    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8q9m" podUID="5d3756a7-d67d-48ef-aa34-a792650e747d"
Sep 09 20:11:00.392525 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.389428    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dn758\""
Sep 09 20:11:00.392525 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.389757    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-562s2\" (UniqueName: \"kubernetes.io/projected/610a91de-a1e2-445d-8f72-48aba0d8a2bf-kube-api-access-562s2\") pod \"node-ca-qpmd2\" (UID: \"610a91de-a1e2-445d-8f72-48aba0d8a2bf\") " pod="openshift-image-registry/node-ca-qpmd2"
Sep 09 20:11:00.392525 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.389786    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-hostroot\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.392525 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.389813    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-host-run-multus-certs\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.392525 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.389851    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-run-ovn\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.392525 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.389880    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-ovn-node-metrics-cert\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.392525 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.389906    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckvcx\" (UniqueName: \"kubernetes.io/projected/c47994f8-a068-4132-86bc-81d0c9ab827f-kube-api-access-ckvcx\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.392525 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.389940    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-etc-sysctl-d\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.392525 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.389993    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-host-run-ovn-kubernetes\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.392525 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390021    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-host-cni-bin\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.392525 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390045    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bc4b3154-62b5-4d79-bfe4-673807839bf7-agent-certs\") pod \"konnectivity-agent-f28rc\" (UID: \"bc4b3154-62b5-4d79-bfe4-673807839bf7\") " pod="kube-system/konnectivity-agent-f28rc"
Sep 09 20:11:00.392525 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390068    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/610a91de-a1e2-445d-8f72-48aba0d8a2bf-host\") pod \"node-ca-qpmd2\" (UID: \"610a91de-a1e2-445d-8f72-48aba0d8a2bf\") " pod="openshift-image-registry/node-ca-qpmd2"
Sep 09 20:11:00.392525 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390092    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-etc-kubernetes\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.392525 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390117    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-ovnkube-script-lib\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.392525 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390142    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c47994f8-a068-4132-86bc-81d0c9ab827f-cni-binary-copy\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.393964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390172    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-host-run-netns\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.393964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-var-lib-openvswitch\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.393964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390220    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-run-openvswitch\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.393964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390262    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-host-cni-netd\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.393964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390286    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/610a91de-a1e2-445d-8f72-48aba0d8a2bf-serviceca\") pod \"node-ca-qpmd2\" (UID: \"610a91de-a1e2-445d-8f72-48aba0d8a2bf\") " pod="openshift-image-registry/node-ca-qpmd2"
Sep 09 20:11:00.393964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390310    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-run-systemd\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.393964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-etc-kubernetes\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.393964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-env-overrides\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.393964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390396    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-multus-socket-dir-parent\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.393964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390420    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-run\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.393964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-lib-modules\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.393964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390470    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-log-socket\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.393964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390495    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/297425e2-5552-4654-b41b-336d88d5a56c-hosts-file\") pod \"node-resolver-l7rwd\" (UID: \"297425e2-5552-4654-b41b-336d88d5a56c\") " pod="openshift-dns/node-resolver-l7rwd"
Sep 09 20:11:00.393964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390523    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-system-cni-dir\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.393964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390578    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-host-var-lib-cni-bin\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.393964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e4de001-c1ee-47a0-a833-d6ffd119372c-host-slash\") pod \"iptables-alerter-lk6j8\" (UID: \"0e4de001-c1ee-47a0-a833-d6ffd119372c\") " pod="openshift-network-operator/iptables-alerter-lk6j8"
Sep 09 20:11:00.393964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390672    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df278843-3e4c-47b6-ba9d-6def44cf12d8-tmp\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.394719 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390867    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c47994f8-a068-4132-86bc-81d0c9ab827f-multus-daemon-config\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.394719 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390898    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/297425e2-5552-4654-b41b-336d88d5a56c-tmp-dir\") pod \"node-resolver-l7rwd\" (UID: \"297425e2-5552-4654-b41b-336d88d5a56c\") " pod="openshift-dns/node-resolver-l7rwd"
Sep 09 20:11:00.394719 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390933    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bc4b3154-62b5-4d79-bfe4-673807839bf7-konnectivity-ca\") pod \"konnectivity-agent-f28rc\" (UID: \"bc4b3154-62b5-4d79-bfe4-673807839bf7\") " pod="kube-system/konnectivity-agent-f28rc"
Sep 09 20:11:00.394719 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390961    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-host-var-lib-cni-multus\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.394719 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390983    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:11:00.394719 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.390986    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-etc-sysconfig\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.394719 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.391196    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0e4de001-c1ee-47a0-a833-d6ffd119372c-iptables-alerter-script\") pod \"iptables-alerter-lk6j8\" (UID: \"0e4de001-c1ee-47a0-a833-d6ffd119372c\") " pod="openshift-network-operator/iptables-alerter-lk6j8"
Sep 09 20:11:00.394719 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.391222    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95zkm\" (UniqueName: \"kubernetes.io/projected/0e4de001-c1ee-47a0-a833-d6ffd119372c-kube-api-access-95zkm\") pod \"iptables-alerter-lk6j8\" (UID: \"0e4de001-c1ee-47a0-a833-d6ffd119372c\") " pod="openshift-network-operator/iptables-alerter-lk6j8"
Sep 09 20:11:00.394719 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.391250    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-sys\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.394719 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.391311    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-node-log\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.394719 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.391341    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-host-run-k8s-cni-cncf-io\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.394719 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.391366    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-var-lib-kubelet\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.394719 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.391392    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-ovnkube-config\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.394719 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.391421    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flhcr\" (UniqueName: \"kubernetes.io/projected/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-kube-api-access-flhcr\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.394719 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.391477    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-multus-conf-dir\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.394719 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.391504    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-etc-sysctl-conf\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.394719 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.391554    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2w9j\" (UniqueName: \"kubernetes.io/projected/6b9953bc-b5e1-4b30-a926-c1a13d4677a7-kube-api-access-g2w9j\") pod \"network-check-target-zcrsj\" (UID: \"6b9953bc-b5e1-4b30-a926-c1a13d4677a7\") " pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:00.395563 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.391590    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-host\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.395563 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.391617    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-cnibin\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.395563 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.391652    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/df278843-3e4c-47b6-ba9d-6def44cf12d8-etc-tuned\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.395563 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.391678    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-etc-openvswitch\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.395563 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.391702    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-multus-cni-dir\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.395563 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.391752    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwxsj\" (UniqueName: \"kubernetes.io/projected/df278843-3e4c-47b6-ba9d-6def44cf12d8-kube-api-access-lwxsj\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.395563 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.391779    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs\") pod \"network-metrics-daemon-9vfht\" (UID: \"7e3d0490-5d57-42a1-858f-1e6bf8830f4c\") " pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:00.395563 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.391816    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dhnp\" (UniqueName: \"kubernetes.io/projected/297425e2-5552-4654-b41b-336d88d5a56c-kube-api-access-2dhnp\") pod \"node-resolver-l7rwd\" (UID: \"297425e2-5552-4654-b41b-336d88d5a56c\") " pod="openshift-dns/node-resolver-l7rwd"
Sep 09 20:11:00.395563 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.391854    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-etc-systemd\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.395563 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.391889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-host-kubelet\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.395563 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.391914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-systemd-units\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.395563 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.391959    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-host-slash\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.395563 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.391987    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.395563 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.392015    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxwg7\" (UniqueName: \"kubernetes.io/projected/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-kube-api-access-fxwg7\") pod \"network-metrics-daemon-9vfht\" (UID: \"7e3d0490-5d57-42a1-858f-1e6bf8830f4c\") " pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:00.395563 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.392043    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-host-var-lib-kubelet\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.395563 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.392080    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-etc-modprobe-d\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.396344 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.392110    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-host-run-netns\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.410411 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.410392    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-s7sgk"
Sep 09 20:11:00.418389 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.418365    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-s7sgk"
Sep 09 20:11:00.479577 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.479550    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 20:11:00.492992 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.492672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/297425e2-5552-4654-b41b-336d88d5a56c-tmp-dir\") pod \"node-resolver-l7rwd\" (UID: \"297425e2-5552-4654-b41b-336d88d5a56c\") " pod="openshift-dns/node-resolver-l7rwd"
Sep 09 20:11:00.493131 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.493036    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bc4b3154-62b5-4d79-bfe4-673807839bf7-konnectivity-ca\") pod \"konnectivity-agent-f28rc\" (UID: \"bc4b3154-62b5-4d79-bfe4-673807839bf7\") " pod="kube-system/konnectivity-agent-f28rc"
Sep 09 20:11:00.493131 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.493118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-host-var-lib-cni-multus\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.493643 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.493598    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91167988-d0fe-4f04-9716-21432d50f5b7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6hqtr\" (UID: \"91167988-d0fe-4f04-9716-21432d50f5b7\") " pod="openshift-multus/multus-additional-cni-plugins-6hqtr"
Sep 09 20:11:00.493736 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.493701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-etc-sysconfig\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.493796 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.493744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0e4de001-c1ee-47a0-a833-d6ffd119372c-iptables-alerter-script\") pod \"iptables-alerter-lk6j8\" (UID: \"0e4de001-c1ee-47a0-a833-d6ffd119372c\") " pod="openshift-network-operator/iptables-alerter-lk6j8"
Sep 09 20:11:00.493796 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.493775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95zkm\" (UniqueName: \"kubernetes.io/projected/0e4de001-c1ee-47a0-a833-d6ffd119372c-kube-api-access-95zkm\") pod \"iptables-alerter-lk6j8\" (UID: \"0e4de001-c1ee-47a0-a833-d6ffd119372c\") " pod="openshift-network-operator/iptables-alerter-lk6j8"
Sep 09 20:11:00.493915 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.493814    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-sys\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.493971 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.493920    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-etc-sysconfig\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.494027 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.493982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-node-log\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.494082 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.494045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-host-run-k8s-cni-cncf-io\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.494146 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.494094    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a74a9a9-d77a-4dce-8c71-7588372245d8-device-dir\") pod \"aws-ebs-csi-driver-node-s5zpm\" (UID: \"8a74a9a9-d77a-4dce-8c71-7588372245d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm"
Sep 09 20:11:00.494146 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.494141    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-var-lib-kubelet\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.494259 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.494176    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-ovnkube-config\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.494259 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.494211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flhcr\" (UniqueName: \"kubernetes.io/projected/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-kube-api-access-flhcr\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.494361 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.494267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-multus-conf-dir\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.494361 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.494295    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a74a9a9-d77a-4dce-8c71-7588372245d8-sys-fs\") pod \"aws-ebs-csi-driver-node-s5zpm\" (UID: \"8a74a9a9-d77a-4dce-8c71-7588372245d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm"
Sep 09 20:11:00.494472 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.494356    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91167988-d0fe-4f04-9716-21432d50f5b7-system-cni-dir\") pod \"multus-additional-cni-plugins-6hqtr\" (UID: \"91167988-d0fe-4f04-9716-21432d50f5b7\") " pod="openshift-multus/multus-additional-cni-plugins-6hqtr"
Sep 09 20:11:00.494472 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.494424    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d3756a7-d67d-48ef-aa34-a792650e747d-original-pull-secret\") pod \"global-pull-secret-syncer-z8q9m\" (UID: \"5d3756a7-d67d-48ef-aa34-a792650e747d\") " pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:00.494600 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.494504    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-etc-sysctl-conf\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.494659 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.494606    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2w9j\" (UniqueName: \"kubernetes.io/projected/6b9953bc-b5e1-4b30-a926-c1a13d4677a7-kube-api-access-g2w9j\") pod \"network-check-target-zcrsj\" (UID: \"6b9953bc-b5e1-4b30-a926-c1a13d4677a7\") " pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:00.494659 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.494614    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/297425e2-5552-4654-b41b-336d88d5a56c-tmp-dir\") pod \"node-resolver-l7rwd\" (UID: \"297425e2-5552-4654-b41b-336d88d5a56c\") " pod="openshift-dns/node-resolver-l7rwd"
Sep 09 20:11:00.494659 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.494647    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-host\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.494794 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.494692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-cnibin\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.494794 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.494738    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91167988-d0fe-4f04-9716-21432d50f5b7-cni-binary-copy\") pod \"multus-additional-cni-plugins-6hqtr\" (UID: \"91167988-d0fe-4f04-9716-21432d50f5b7\") " pod="openshift-multus/multus-additional-cni-plugins-6hqtr"
Sep 09 20:11:00.494794 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.494781    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5d3756a7-d67d-48ef-aa34-a792650e747d-dbus\") pod \"global-pull-secret-syncer-z8q9m\" (UID: \"5d3756a7-d67d-48ef-aa34-a792650e747d\") " pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:00.494891 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.494824    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/df278843-3e4c-47b6-ba9d-6def44cf12d8-etc-tuned\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.494891 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.494877    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-etc-openvswitch\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.494954 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.494896    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0e4de001-c1ee-47a0-a833-d6ffd119372c-iptables-alerter-script\") pod \"iptables-alerter-lk6j8\" (UID: \"0e4de001-c1ee-47a0-a833-d6ffd119372c\") " pod="openshift-network-operator/iptables-alerter-lk6j8"
Sep 09 20:11:00.494954 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.494916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-multus-cni-dir\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.495028 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.494966    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t7z7\" (UniqueName: \"kubernetes.io/projected/91167988-d0fe-4f04-9716-21432d50f5b7-kube-api-access-7t7z7\") pod \"multus-additional-cni-plugins-6hqtr\" (UID: \"91167988-d0fe-4f04-9716-21432d50f5b7\") " pod="openshift-multus/multus-additional-cni-plugins-6hqtr"
Sep 09 20:11:00.495087 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.495011    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwxsj\" (UniqueName: \"kubernetes.io/projected/df278843-3e4c-47b6-ba9d-6def44cf12d8-kube-api-access-lwxsj\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.495143 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.495069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-sys\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.495143 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.495099    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs\") pod \"network-metrics-daemon-9vfht\" (UID: \"7e3d0490-5d57-42a1-858f-1e6bf8830f4c\") " pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:00.495143 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.495135    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dhnp\" (UniqueName: \"kubernetes.io/projected/297425e2-5552-4654-b41b-336d88d5a56c-kube-api-access-2dhnp\") pod \"node-resolver-l7rwd\" (UID: \"297425e2-5552-4654-b41b-336d88d5a56c\") " pod="openshift-dns/node-resolver-l7rwd"
Sep 09 20:11:00.495296 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.495170    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5d3756a7-d67d-48ef-aa34-a792650e747d-kubelet-config\") pod \"global-pull-secret-syncer-z8q9m\" (UID: \"5d3756a7-d67d-48ef-aa34-a792650e747d\") " pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:00.495296 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.495215    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-etc-systemd\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.495296 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.495284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-host-kubelet\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.495435 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.495316    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-systemd-units\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.495636 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.495613    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-host-slash\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.495735 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.495687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.495853 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.495761    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxwg7\" (UniqueName: \"kubernetes.io/projected/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-kube-api-access-fxwg7\") pod \"network-metrics-daemon-9vfht\" (UID: \"7e3d0490-5d57-42a1-858f-1e6bf8830f4c\") " pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:00.495853 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.495832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-host-var-lib-kubelet\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.495966 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.495861    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bc4b3154-62b5-4d79-bfe4-673807839bf7-konnectivity-ca\") pod \"konnectivity-agent-f28rc\" (UID: \"bc4b3154-62b5-4d79-bfe4-673807839bf7\") " pod="kube-system/konnectivity-agent-f28rc"
Sep 09 20:11:00.495966 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.495877    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-etc-modprobe-d\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.496109 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.496018    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-host-run-netns\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.496109 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.496072    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-562s2\" (UniqueName: \"kubernetes.io/projected/610a91de-a1e2-445d-8f72-48aba0d8a2bf-kube-api-access-562s2\") pod \"node-ca-qpmd2\" (UID: \"610a91de-a1e2-445d-8f72-48aba0d8a2bf\") " pod="openshift-image-registry/node-ca-qpmd2"
Sep 09 20:11:00.496206 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.496109    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-hostroot\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.496206 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.496137    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-cnibin\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.496206 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.496148    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-host-run-multus-certs\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.496350 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.496202    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a74a9a9-d77a-4dce-8c71-7588372245d8-registration-dir\") pod \"aws-ebs-csi-driver-node-s5zpm\" (UID: \"8a74a9a9-d77a-4dce-8c71-7588372245d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm"
Sep 09 20:11:00.496350 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.496213    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-host-run-multus-certs\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.496350 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.496242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-run-ovn\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.496492 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.496352    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-ovn-node-metrics-cert\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.496492 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.496426    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckvcx\" (UniqueName: \"kubernetes.io/projected/c47994f8-a068-4132-86bc-81d0c9ab827f-kube-api-access-ckvcx\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.496492 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.496462    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/91167988-d0fe-4f04-9716-21432d50f5b7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6hqtr\" (UID: \"91167988-d0fe-4f04-9716-21432d50f5b7\") " pod="openshift-multus/multus-additional-cni-plugins-6hqtr"
Sep 09 20:11:00.496671 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.496493    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-etc-sysctl-d\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.496671 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.496557    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-host-run-ovn-kubernetes\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.496671 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.496595    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-host-cni-bin\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.496671 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.496643    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bc4b3154-62b5-4d79-bfe4-673807839bf7-agent-certs\") pod \"konnectivity-agent-f28rc\" (UID: \"bc4b3154-62b5-4d79-bfe4-673807839bf7\") " pod="kube-system/konnectivity-agent-f28rc"
Sep 09 20:11:00.496866 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.496658    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-host-kubelet\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.496866 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.496674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/610a91de-a1e2-445d-8f72-48aba0d8a2bf-host\") pod \"node-ca-qpmd2\" (UID: \"610a91de-a1e2-445d-8f72-48aba0d8a2bf\") " pod="openshift-image-registry/node-ca-qpmd2"
Sep 09 20:11:00.496866 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.496716    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a74a9a9-d77a-4dce-8c71-7588372245d8-socket-dir\") pod \"aws-ebs-csi-driver-node-s5zpm\" (UID: \"8a74a9a9-d77a-4dce-8c71-7588372245d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm"
Sep 09 20:11:00.496866 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.496749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7jvg\" (UniqueName: \"kubernetes.io/projected/8a74a9a9-d77a-4dce-8c71-7588372245d8-kube-api-access-w7jvg\") pod \"aws-ebs-csi-driver-node-s5zpm\" (UID: \"8a74a9a9-d77a-4dce-8c71-7588372245d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm"
Sep 09 20:11:00.496866 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.496775    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 20:11:00.497107 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.496937    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-etc-kubernetes\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.497107 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.496946    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-multus-cni-dir\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.497107 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.497036    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-node-log\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.497253 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.497091    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-host-run-k8s-cni-cncf-io\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.497253 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.497198    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-var-lib-kubelet\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.497508 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:00.497490    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:00.497670 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:00.497657    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs podName:7e3d0490-5d57-42a1-858f-1e6bf8830f4c nodeName:}" failed. No retries permitted until 2026-09-09 20:11:00.997600968 +0000 UTC m=+3.002777957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs") pod "network-metrics-daemon-9vfht" (UID: "7e3d0490-5d57-42a1-858f-1e6bf8830f4c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:00.497854 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.497834    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-ovnkube-config\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.498155 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.498124    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-etc-systemd\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.498292 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.498230    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-host-var-lib-kubelet\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.498357 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.498308    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-multus-conf-dir\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.499048 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.498477    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-host\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.499048 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.498525    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.499048 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.498618    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-host-cni-bin\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.499048 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.498673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-etc-sysctl-conf\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.499048 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.498725    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-run-ovn\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.499048 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.498770    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-systemd-units\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.499048 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.498934    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-host-slash\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.499506 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.499334    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-etc-sysctl-d\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.499506 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.499399    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-host-run-ovn-kubernetes\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.499506 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.499477    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-etc-openvswitch\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.499694 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.499574    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/610a91de-a1e2-445d-8f72-48aba0d8a2bf-host\") pod \"node-ca-qpmd2\" (UID: \"610a91de-a1e2-445d-8f72-48aba0d8a2bf\") " pod="openshift-image-registry/node-ca-qpmd2"
Sep 09 20:11:00.500569 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500023    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-etc-modprobe-d\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.500569 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.496782    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-etc-kubernetes\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.500569 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-host-run-netns\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.500569 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500091    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-ovnkube-script-lib\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.500569 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500120    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-hostroot\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.500569 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500145    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c47994f8-a068-4132-86bc-81d0c9ab827f-cni-binary-copy\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.500569 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500191    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-host-run-netns\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.500569 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500233    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-var-lib-openvswitch\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.500569 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-run-openvswitch\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.500569 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500307    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-host-cni-netd\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.500569 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/610a91de-a1e2-445d-8f72-48aba0d8a2bf-serviceca\") pod \"node-ca-qpmd2\" (UID: \"610a91de-a1e2-445d-8f72-48aba0d8a2bf\") " pod="openshift-image-registry/node-ca-qpmd2"
Sep 09 20:11:00.500569 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500395    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a74a9a9-d77a-4dce-8c71-7588372245d8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s5zpm\" (UID: \"8a74a9a9-d77a-4dce-8c71-7588372245d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm"
Sep 09 20:11:00.500569 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500452    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/91167988-d0fe-4f04-9716-21432d50f5b7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6hqtr\" (UID: \"91167988-d0fe-4f04-9716-21432d50f5b7\") " pod="openshift-multus/multus-additional-cni-plugins-6hqtr"
Sep 09 20:11:00.500569 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500513    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-run-systemd\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.501298 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500599    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-run-openvswitch\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.501298 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-host-run-netns\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.501298 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500646    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-host-var-lib-cni-multus\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.501298 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500671    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-etc-kubernetes\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.501298 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500708    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-env-overrides\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.501298 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-multus-socket-dir-parent\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.501298 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-run\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.501298 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500823    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-lib-modules\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.501298 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500859    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-log-socket\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.501298 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/297425e2-5552-4654-b41b-336d88d5a56c-hosts-file\") pod \"node-resolver-l7rwd\" (UID: \"297425e2-5552-4654-b41b-336d88d5a56c\") " pod="openshift-dns/node-resolver-l7rwd"
Sep 09 20:11:00.501298 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-system-cni-dir\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.501298 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.500975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-host-var-lib-cni-bin\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.501298 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.501012    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a74a9a9-d77a-4dce-8c71-7588372245d8-etc-selinux\") pod \"aws-ebs-csi-driver-node-s5zpm\" (UID: \"8a74a9a9-d77a-4dce-8c71-7588372245d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm"
Sep 09 20:11:00.501298 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.501082    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e4de001-c1ee-47a0-a833-d6ffd119372c-host-slash\") pod \"iptables-alerter-lk6j8\" (UID: \"0e4de001-c1ee-47a0-a833-d6ffd119372c\") " pod="openshift-network-operator/iptables-alerter-lk6j8"
Sep 09 20:11:00.501298 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.501152    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/610a91de-a1e2-445d-8f72-48aba0d8a2bf-serviceca\") pod \"node-ca-qpmd2\" (UID: \"610a91de-a1e2-445d-8f72-48aba0d8a2bf\") " pod="openshift-image-registry/node-ca-qpmd2"
Sep 09 20:11:00.501298 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.501170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df278843-3e4c-47b6-ba9d-6def44cf12d8-tmp\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.501298 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.501218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c47994f8-a068-4132-86bc-81d0c9ab827f-multus-daemon-config\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.501298 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.501235    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-host-cni-netd\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.502254 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.501274    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91167988-d0fe-4f04-9716-21432d50f5b7-cnibin\") pod \"multus-additional-cni-plugins-6hqtr\" (UID: \"91167988-d0fe-4f04-9716-21432d50f5b7\") " pod="openshift-multus/multus-additional-cni-plugins-6hqtr"
Sep 09 20:11:00.502254 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.501312    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-ovnkube-script-lib\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.502254 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.501430    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-etc-kubernetes\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.502254 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.501508    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-system-cni-dir\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.502254 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.501576    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-host-var-lib-cni-bin\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.502254 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.501578    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/297425e2-5552-4654-b41b-336d88d5a56c-hosts-file\") pod \"node-resolver-l7rwd\" (UID: \"297425e2-5552-4654-b41b-336d88d5a56c\") " pod="openshift-dns/node-resolver-l7rwd"
Sep 09 20:11:00.502254 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.501651    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e4de001-c1ee-47a0-a833-d6ffd119372c-host-slash\") pod \"iptables-alerter-lk6j8\" (UID: \"0e4de001-c1ee-47a0-a833-d6ffd119372c\") " pod="openshift-network-operator/iptables-alerter-lk6j8"
Sep 09 20:11:00.502254 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.502210    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-env-overrides\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.502688 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.502325    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c47994f8-a068-4132-86bc-81d0c9ab827f-multus-socket-dir-parent\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.502688 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.502388    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-run-systemd\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.502688 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.502406    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-run\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.502688 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.502387    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-var-lib-openvswitch\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.502688 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.502501    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df278843-3e4c-47b6-ba9d-6def44cf12d8-lib-modules\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.502688 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.502590    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-log-socket\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.503010 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.502971    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c47994f8-a068-4132-86bc-81d0c9ab827f-cni-binary-copy\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.503125 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.503102    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c47994f8-a068-4132-86bc-81d0c9ab827f-multus-daemon-config\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.505045 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.504993    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df278843-3e4c-47b6-ba9d-6def44cf12d8-tmp\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.506324 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:00.506304    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:00.506421 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:00.506327    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:00.506421 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:00.506341    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g2w9j for pod openshift-network-diagnostics/network-check-target-zcrsj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:00.506421 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:00.506415    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b9953bc-b5e1-4b30-a926-c1a13d4677a7-kube-api-access-g2w9j podName:6b9953bc-b5e1-4b30-a926-c1a13d4677a7 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:01.006396265 +0000 UTC m=+3.011573254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g2w9j" (UniqueName: "kubernetes.io/projected/6b9953bc-b5e1-4b30-a926-c1a13d4677a7-kube-api-access-g2w9j") pod "network-check-target-zcrsj" (UID: "6b9953bc-b5e1-4b30-a926-c1a13d4677a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:00.507329 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.507304    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bc4b3154-62b5-4d79-bfe4-673807839bf7-agent-certs\") pod \"konnectivity-agent-f28rc\" (UID: \"bc4b3154-62b5-4d79-bfe4-673807839bf7\") " pod="kube-system/konnectivity-agent-f28rc"
Sep 09 20:11:00.507424 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.507355    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95zkm\" (UniqueName: \"kubernetes.io/projected/0e4de001-c1ee-47a0-a833-d6ffd119372c-kube-api-access-95zkm\") pod \"iptables-alerter-lk6j8\" (UID: \"0e4de001-c1ee-47a0-a833-d6ffd119372c\") " pod="openshift-network-operator/iptables-alerter-lk6j8"
Sep 09 20:11:00.507866 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.507836    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dhnp\" (UniqueName: \"kubernetes.io/projected/297425e2-5552-4654-b41b-336d88d5a56c-kube-api-access-2dhnp\") pod \"node-resolver-l7rwd\" (UID: \"297425e2-5552-4654-b41b-336d88d5a56c\") " pod="openshift-dns/node-resolver-l7rwd"
Sep 09 20:11:00.508983 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.508004    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/df278843-3e4c-47b6-ba9d-6def44cf12d8-etc-tuned\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.508983 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.508224    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-ovn-node-metrics-cert\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.509776 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.509751    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckvcx\" (UniqueName: \"kubernetes.io/projected/c47994f8-a068-4132-86bc-81d0c9ab827f-kube-api-access-ckvcx\") pod \"multus-f7kxd\" (UID: \"c47994f8-a068-4132-86bc-81d0c9ab827f\") " pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.510493 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.510466    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxwg7\" (UniqueName: \"kubernetes.io/projected/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-kube-api-access-fxwg7\") pod \"network-metrics-daemon-9vfht\" (UID: \"7e3d0490-5d57-42a1-858f-1e6bf8830f4c\") " pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:00.511272 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.511245    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-562s2\" (UniqueName: \"kubernetes.io/projected/610a91de-a1e2-445d-8f72-48aba0d8a2bf-kube-api-access-562s2\") pod \"node-ca-qpmd2\" (UID: \"610a91de-a1e2-445d-8f72-48aba0d8a2bf\") " pod="openshift-image-registry/node-ca-qpmd2"
Sep 09 20:11:00.514940 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.514910    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwxsj\" (UniqueName: \"kubernetes.io/projected/df278843-3e4c-47b6-ba9d-6def44cf12d8-kube-api-access-lwxsj\") pod \"tuned-4k95l\" (UID: \"df278843-3e4c-47b6-ba9d-6def44cf12d8\") " pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.516211 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.516190    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flhcr\" (UniqueName: \"kubernetes.io/projected/e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8-kube-api-access-flhcr\") pod \"ovnkube-node-jqqq8\" (UID: \"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.602199 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a74a9a9-d77a-4dce-8c71-7588372245d8-device-dir\") pod \"aws-ebs-csi-driver-node-s5zpm\" (UID: \"8a74a9a9-d77a-4dce-8c71-7588372245d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm"
Sep 09 20:11:00.602199 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602156    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a74a9a9-d77a-4dce-8c71-7588372245d8-sys-fs\") pod \"aws-ebs-csi-driver-node-s5zpm\" (UID: \"8a74a9a9-d77a-4dce-8c71-7588372245d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm"
Sep 09 20:11:00.602199 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91167988-d0fe-4f04-9716-21432d50f5b7-system-cni-dir\") pod \"multus-additional-cni-plugins-6hqtr\" (UID: \"91167988-d0fe-4f04-9716-21432d50f5b7\") " pod="openshift-multus/multus-additional-cni-plugins-6hqtr"
Sep 09 20:11:00.602411 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d3756a7-d67d-48ef-aa34-a792650e747d-original-pull-secret\") pod \"global-pull-secret-syncer-z8q9m\" (UID: \"5d3756a7-d67d-48ef-aa34-a792650e747d\") " pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:00.602411 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602257    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a74a9a9-d77a-4dce-8c71-7588372245d8-device-dir\") pod \"aws-ebs-csi-driver-node-s5zpm\" (UID: \"8a74a9a9-d77a-4dce-8c71-7588372245d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm"
Sep 09 20:11:00.602411 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602260    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a74a9a9-d77a-4dce-8c71-7588372245d8-sys-fs\") pod \"aws-ebs-csi-driver-node-s5zpm\" (UID: \"8a74a9a9-d77a-4dce-8c71-7588372245d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm"
Sep 09 20:11:00.602411 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91167988-d0fe-4f04-9716-21432d50f5b7-cni-binary-copy\") pod \"multus-additional-cni-plugins-6hqtr\" (UID: \"91167988-d0fe-4f04-9716-21432d50f5b7\") " pod="openshift-multus/multus-additional-cni-plugins-6hqtr"
Sep 09 20:11:00.602411 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602310    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91167988-d0fe-4f04-9716-21432d50f5b7-system-cni-dir\") pod \"multus-additional-cni-plugins-6hqtr\" (UID: \"91167988-d0fe-4f04-9716-21432d50f5b7\") " pod="openshift-multus/multus-additional-cni-plugins-6hqtr"
Sep 09 20:11:00.602411 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602318    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5d3756a7-d67d-48ef-aa34-a792650e747d-dbus\") pod \"global-pull-secret-syncer-z8q9m\" (UID: \"5d3756a7-d67d-48ef-aa34-a792650e747d\") " pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:00.602411 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:00.602360    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:00.602411 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602374    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t7z7\" (UniqueName: \"kubernetes.io/projected/91167988-d0fe-4f04-9716-21432d50f5b7-kube-api-access-7t7z7\") pod \"multus-additional-cni-plugins-6hqtr\" (UID: \"91167988-d0fe-4f04-9716-21432d50f5b7\") " pod="openshift-multus/multus-additional-cni-plugins-6hqtr"
Sep 09 20:11:00.602411 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602381    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5d3756a7-d67d-48ef-aa34-a792650e747d-dbus\") pod \"global-pull-secret-syncer-z8q9m\" (UID: \"5d3756a7-d67d-48ef-aa34-a792650e747d\") " pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:00.602741 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:00.602420    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3756a7-d67d-48ef-aa34-a792650e747d-original-pull-secret podName:5d3756a7-d67d-48ef-aa34-a792650e747d nodeName:}" failed. No retries permitted until 2026-09-09 20:11:01.102401035 +0000 UTC m=+3.107578025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d3756a7-d67d-48ef-aa34-a792650e747d-original-pull-secret") pod "global-pull-secret-syncer-z8q9m" (UID: "5d3756a7-d67d-48ef-aa34-a792650e747d") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:00.602741 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602473    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5d3756a7-d67d-48ef-aa34-a792650e747d-kubelet-config\") pod \"global-pull-secret-syncer-z8q9m\" (UID: \"5d3756a7-d67d-48ef-aa34-a792650e747d\") " pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:00.602741 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602510    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a74a9a9-d77a-4dce-8c71-7588372245d8-registration-dir\") pod \"aws-ebs-csi-driver-node-s5zpm\" (UID: \"8a74a9a9-d77a-4dce-8c71-7588372245d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm"
Sep 09 20:11:00.602741 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/91167988-d0fe-4f04-9716-21432d50f5b7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6hqtr\" (UID: \"91167988-d0fe-4f04-9716-21432d50f5b7\") " pod="openshift-multus/multus-additional-cni-plugins-6hqtr"
Sep 09 20:11:00.602741 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a74a9a9-d77a-4dce-8c71-7588372245d8-socket-dir\") pod \"aws-ebs-csi-driver-node-s5zpm\" (UID: \"8a74a9a9-d77a-4dce-8c71-7588372245d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm"
Sep 09 20:11:00.602741 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602616    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7jvg\" (UniqueName: \"kubernetes.io/projected/8a74a9a9-d77a-4dce-8c71-7588372245d8-kube-api-access-w7jvg\") pod \"aws-ebs-csi-driver-node-s5zpm\" (UID: \"8a74a9a9-d77a-4dce-8c71-7588372245d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm"
Sep 09 20:11:00.602741 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602627    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5d3756a7-d67d-48ef-aa34-a792650e747d-kubelet-config\") pod \"global-pull-secret-syncer-z8q9m\" (UID: \"5d3756a7-d67d-48ef-aa34-a792650e747d\") " pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:00.602741 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602663    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a74a9a9-d77a-4dce-8c71-7588372245d8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s5zpm\" (UID: \"8a74a9a9-d77a-4dce-8c71-7588372245d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm"
Sep 09 20:11:00.602741 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602680    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a74a9a9-d77a-4dce-8c71-7588372245d8-registration-dir\") pod \"aws-ebs-csi-driver-node-s5zpm\" (UID: \"8a74a9a9-d77a-4dce-8c71-7588372245d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm"
Sep 09 20:11:00.602741 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602686    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/91167988-d0fe-4f04-9716-21432d50f5b7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6hqtr\" (UID: \"91167988-d0fe-4f04-9716-21432d50f5b7\") " pod="openshift-multus/multus-additional-cni-plugins-6hqtr"
Sep 09 20:11:00.602741 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602726    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a74a9a9-d77a-4dce-8c71-7588372245d8-etc-selinux\") pod \"aws-ebs-csi-driver-node-s5zpm\" (UID: \"8a74a9a9-d77a-4dce-8c71-7588372245d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm"
Sep 09 20:11:00.603135 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602747    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a74a9a9-d77a-4dce-8c71-7588372245d8-socket-dir\") pod \"aws-ebs-csi-driver-node-s5zpm\" (UID: \"8a74a9a9-d77a-4dce-8c71-7588372245d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm"
Sep 09 20:11:00.603135 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91167988-d0fe-4f04-9716-21432d50f5b7-cnibin\") pod \"multus-additional-cni-plugins-6hqtr\" (UID: \"91167988-d0fe-4f04-9716-21432d50f5b7\") " pod="openshift-multus/multus-additional-cni-plugins-6hqtr"
Sep 09 20:11:00.603135 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602776    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91167988-d0fe-4f04-9716-21432d50f5b7-cni-binary-copy\") pod \"multus-additional-cni-plugins-6hqtr\" (UID: \"91167988-d0fe-4f04-9716-21432d50f5b7\") " pod="openshift-multus/multus-additional-cni-plugins-6hqtr"
Sep 09 20:11:00.603135 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602785    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91167988-d0fe-4f04-9716-21432d50f5b7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6hqtr\" (UID: \"91167988-d0fe-4f04-9716-21432d50f5b7\") " pod="openshift-multus/multus-additional-cni-plugins-6hqtr"
Sep 09 20:11:00.603135 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602942    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91167988-d0fe-4f04-9716-21432d50f5b7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6hqtr\" (UID: \"91167988-d0fe-4f04-9716-21432d50f5b7\") " pod="openshift-multus/multus-additional-cni-plugins-6hqtr"
Sep 09 20:11:00.603135 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602993    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91167988-d0fe-4f04-9716-21432d50f5b7-cnibin\") pod \"multus-additional-cni-plugins-6hqtr\" (UID: \"91167988-d0fe-4f04-9716-21432d50f5b7\") " pod="openshift-multus/multus-additional-cni-plugins-6hqtr"
Sep 09 20:11:00.603135 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.602997    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a74a9a9-d77a-4dce-8c71-7588372245d8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s5zpm\" (UID: \"8a74a9a9-d77a-4dce-8c71-7588372245d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm"
Sep 09 20:11:00.603135 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.603074    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/91167988-d0fe-4f04-9716-21432d50f5b7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6hqtr\" (UID: \"91167988-d0fe-4f04-9716-21432d50f5b7\") " pod="openshift-multus/multus-additional-cni-plugins-6hqtr"
Sep 09 20:11:00.603135 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.603126    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a74a9a9-d77a-4dce-8c71-7588372245d8-etc-selinux\") pod \"aws-ebs-csi-driver-node-s5zpm\" (UID: \"8a74a9a9-d77a-4dce-8c71-7588372245d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm"
Sep 09 20:11:00.603524 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.603165    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/91167988-d0fe-4f04-9716-21432d50f5b7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6hqtr\" (UID: \"91167988-d0fe-4f04-9716-21432d50f5b7\") " pod="openshift-multus/multus-additional-cni-plugins-6hqtr"
Sep 09 20:11:00.613025 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.612996    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t7z7\" (UniqueName: \"kubernetes.io/projected/91167988-d0fe-4f04-9716-21432d50f5b7-kube-api-access-7t7z7\") pod \"multus-additional-cni-plugins-6hqtr\" (UID: \"91167988-d0fe-4f04-9716-21432d50f5b7\") " pod="openshift-multus/multus-additional-cni-plugins-6hqtr"
Sep 09 20:11:00.613143 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.612997    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7jvg\" (UniqueName: \"kubernetes.io/projected/8a74a9a9-d77a-4dce-8c71-7588372245d8-kube-api-access-w7jvg\") pod \"aws-ebs-csi-driver-node-s5zpm\" (UID: \"8a74a9a9-d77a-4dce-8c71-7588372245d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm"
Sep 09 20:11:00.688186 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.688139    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qpmd2"
Sep 09 20:11:00.697233 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.697207    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f7kxd"
Sep 09 20:11:00.710622 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.710591    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lk6j8"
Sep 09 20:11:00.716823 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.716706    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4k95l"
Sep 09 20:11:00.728704 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.728681    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:00.736654 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.736632    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f28rc"
Sep 09 20:11:00.746232 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.746211    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l7rwd"
Sep 09 20:11:00.755736 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.755711    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm"
Sep 09 20:11:00.763531 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:00.763505    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6hqtr"
Sep 09 20:11:01.005006 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:01.004967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs\") pod \"network-metrics-daemon-9vfht\" (UID: \"7e3d0490-5d57-42a1-858f-1e6bf8830f4c\") " pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:01.005207 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:01.005113    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:01.005207 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:01.005186    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs podName:7e3d0490-5d57-42a1-858f-1e6bf8830f4c nodeName:}" failed. No retries permitted until 2026-09-09 20:11:02.005165575 +0000 UTC m=+4.010342569 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs") pod "network-metrics-daemon-9vfht" (UID: "7e3d0490-5d57-42a1-858f-1e6bf8830f4c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:01.105339 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:01.105295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d3756a7-d67d-48ef-aa34-a792650e747d-original-pull-secret\") pod \"global-pull-secret-syncer-z8q9m\" (UID: \"5d3756a7-d67d-48ef-aa34-a792650e747d\") " pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:01.105339 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:01.105347    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2w9j\" (UniqueName: \"kubernetes.io/projected/6b9953bc-b5e1-4b30-a926-c1a13d4677a7-kube-api-access-g2w9j\") pod \"network-check-target-zcrsj\" (UID: \"6b9953bc-b5e1-4b30-a926-c1a13d4677a7\") " pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:01.105603 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:01.105497    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:01.105603 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:01.105524    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:01.105603 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:01.105540    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:01.105603 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:01.105563    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g2w9j for pod openshift-network-diagnostics/network-check-target-zcrsj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:01.105603 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:01.105581    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3756a7-d67d-48ef-aa34-a792650e747d-original-pull-secret podName:5d3756a7-d67d-48ef-aa34-a792650e747d nodeName:}" failed. No retries permitted until 2026-09-09 20:11:02.10555988 +0000 UTC m=+4.110736869 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d3756a7-d67d-48ef-aa34-a792650e747d-original-pull-secret") pod "global-pull-secret-syncer-z8q9m" (UID: "5d3756a7-d67d-48ef-aa34-a792650e747d") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:01.105815 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:01.105612    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b9953bc-b5e1-4b30-a926-c1a13d4677a7-kube-api-access-g2w9j podName:6b9953bc-b5e1-4b30-a926-c1a13d4677a7 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:02.105603195 +0000 UTC m=+4.110780171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2w9j" (UniqueName: "kubernetes.io/projected/6b9953bc-b5e1-4b30-a926-c1a13d4677a7-kube-api-access-g2w9j") pod "network-check-target-zcrsj" (UID: "6b9953bc-b5e1-4b30-a926-c1a13d4677a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:01.420864 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:01.420779    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:06:00 +0000 UTC" deadline="2028-03-12 20:10:57.027171431 +0000 UTC"
Sep 09 20:11:01.420864 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:01.420809    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13199h59m55.606365732s"
Sep 09 20:11:01.600187 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:01.600154    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:01.629713 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:01.629683    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:01.629881 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:01.629819    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vfht" podUID="7e3d0490-5d57-42a1-858f-1e6bf8830f4c"
Sep 09 20:11:01.629943 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:01.629879    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:01.630009 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:01.629991    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8q9m" podUID="5d3756a7-d67d-48ef-aa34-a792650e747d"
Sep 09 20:11:01.731959 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:11:01.731913    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf278843_3e4c_47b6_ba9d_6def44cf12d8.slice/crio-b7c270268887062b4290d8d76f1f880f5f3231fcbd0c5c6a66638680fbd50b64 WatchSource:0}: Error finding container b7c270268887062b4290d8d76f1f880f5f3231fcbd0c5c6a66638680fbd50b64: Status 404 returned error can't find the container with id b7c270268887062b4290d8d76f1f880f5f3231fcbd0c5c6a66638680fbd50b64
Sep 09 20:11:01.734153 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:11:01.733923    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc47994f8_a068_4132_86bc_81d0c9ab827f.slice/crio-d3f66d2928aca0bc9fc1d4158e52df1ace60e66c4821c83171c3437199a74f29 WatchSource:0}: Error finding container d3f66d2928aca0bc9fc1d4158e52df1ace60e66c4821c83171c3437199a74f29: Status 404 returned error can't find the container with id d3f66d2928aca0bc9fc1d4158e52df1ace60e66c4821c83171c3437199a74f29
Sep 09 20:11:01.745299 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:11:01.745224    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4de001_c1ee_47a0_a833_d6ffd119372c.slice/crio-259f0f2c7f44c2a6d9124ea2709dcd9ec92079eef61fc8a19e5af99f4ae47fc0 WatchSource:0}: Error finding container 259f0f2c7f44c2a6d9124ea2709dcd9ec92079eef61fc8a19e5af99f4ae47fc0: Status 404 returned error can't find the container with id 259f0f2c7f44c2a6d9124ea2709dcd9ec92079eef61fc8a19e5af99f4ae47fc0
Sep 09 20:11:01.745737 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:11:01.745711    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610a91de_a1e2_445d_8f72_48aba0d8a2bf.slice/crio-5932c140e27d6dd9ecc343ddec536c6d39dd9e3a5f42ccc1e30aa8aeac2d4b58 WatchSource:0}: Error finding container 5932c140e27d6dd9ecc343ddec536c6d39dd9e3a5f42ccc1e30aa8aeac2d4b58: Status 404 returned error can't find the container with id 5932c140e27d6dd9ecc343ddec536c6d39dd9e3a5f42ccc1e30aa8aeac2d4b58
Sep 09 20:11:01.746057 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:11:01.745940    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91167988_d0fe_4f04_9716_21432d50f5b7.slice/crio-141f44cdda842bd26cd3aff6932417a8e1a329dc6ca3f70d18ef1e02a955440e WatchSource:0}: Error finding container 141f44cdda842bd26cd3aff6932417a8e1a329dc6ca3f70d18ef1e02a955440e: Status 404 returned error can't find the container with id 141f44cdda842bd26cd3aff6932417a8e1a329dc6ca3f70d18ef1e02a955440e
Sep 09 20:11:01.747253 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:11:01.747228    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d7bb0f_c4ae_41cc_bc1d_a12071b148b8.slice/crio-e9c747b12907c05899ec980790b2fb807d9de17c3187ea5decfb6da4fd7002a4 WatchSource:0}: Error finding container e9c747b12907c05899ec980790b2fb807d9de17c3187ea5decfb6da4fd7002a4: Status 404 returned error can't find the container with id e9c747b12907c05899ec980790b2fb807d9de17c3187ea5decfb6da4fd7002a4
Sep 09 20:11:02.012042 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:02.012011    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs\") pod \"network-metrics-daemon-9vfht\" (UID: \"7e3d0490-5d57-42a1-858f-1e6bf8830f4c\") " pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:02.012196 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:02.012176    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:02.012257 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:02.012246    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs podName:7e3d0490-5d57-42a1-858f-1e6bf8830f4c nodeName:}" failed. No retries permitted until 2026-09-09 20:11:04.012230756 +0000 UTC m=+6.017407743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs") pod "network-metrics-daemon-9vfht" (UID: "7e3d0490-5d57-42a1-858f-1e6bf8830f4c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:02.113572 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:02.113360    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d3756a7-d67d-48ef-aa34-a792650e747d-original-pull-secret\") pod \"global-pull-secret-syncer-z8q9m\" (UID: \"5d3756a7-d67d-48ef-aa34-a792650e747d\") " pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:02.113765 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:02.113584    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2w9j\" (UniqueName: \"kubernetes.io/projected/6b9953bc-b5e1-4b30-a926-c1a13d4677a7-kube-api-access-g2w9j\") pod \"network-check-target-zcrsj\" (UID: \"6b9953bc-b5e1-4b30-a926-c1a13d4677a7\") " pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:02.113765 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:02.113529    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:02.113765 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:02.113698    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3756a7-d67d-48ef-aa34-a792650e747d-original-pull-secret podName:5d3756a7-d67d-48ef-aa34-a792650e747d nodeName:}" failed. No retries permitted until 2026-09-09 20:11:04.113678219 +0000 UTC m=+6.118855195 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d3756a7-d67d-48ef-aa34-a792650e747d-original-pull-secret") pod "global-pull-secret-syncer-z8q9m" (UID: "5d3756a7-d67d-48ef-aa34-a792650e747d") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:02.113765 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:02.113748    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:02.113765 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:02.113769    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:02.113971 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:02.113782    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g2w9j for pod openshift-network-diagnostics/network-check-target-zcrsj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:02.113971 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:02.113839    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b9953bc-b5e1-4b30-a926-c1a13d4677a7-kube-api-access-g2w9j podName:6b9953bc-b5e1-4b30-a926-c1a13d4677a7 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:04.11382297 +0000 UTC m=+6.118999960 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2w9j" (UniqueName: "kubernetes.io/projected/6b9953bc-b5e1-4b30-a926-c1a13d4677a7-kube-api-access-g2w9j") pod "network-check-target-zcrsj" (UID: "6b9953bc-b5e1-4b30-a926-c1a13d4677a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:02.422067 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:02.421945    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:06:00 +0000 UTC" deadline="2028-05-01 07:32:35.789842074 +0000 UTC"
Sep 09 20:11:02.422067 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:02.422017    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14387h21m33.367829559s"
Sep 09 20:11:02.630616 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:02.630582    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:02.630798 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:02.630722    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcrsj" podUID="6b9953bc-b5e1-4b30-a926-c1a13d4677a7"
Sep 09 20:11:02.648073 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:02.648033    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l7rwd" event={"ID":"297425e2-5552-4654-b41b-336d88d5a56c","Type":"ContainerStarted","Data":"1d70d311dec93bc6191f3fdab0d4b21cdca9b61ce9c2c29b17601daa914fcfdb"}
Sep 09 20:11:02.654284 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:02.654251    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4k95l" event={"ID":"df278843-3e4c-47b6-ba9d-6def44cf12d8","Type":"ContainerStarted","Data":"b7c270268887062b4290d8d76f1f880f5f3231fcbd0c5c6a66638680fbd50b64"}
Sep 09 20:11:02.685703 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:02.685010    2581 generic.go:356] "Generic (PLEG): container finished" podID="45419fbd63c226915cb25a2cb6311f7e" containerID="47e238a20926c72e6474a969d6a81cca1e0849a5232943fe92fe04e010682147" exitCode=0
Sep 09 20:11:02.685703 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:02.685135    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal" event={"ID":"45419fbd63c226915cb25a2cb6311f7e","Type":"ContainerDied","Data":"47e238a20926c72e6474a969d6a81cca1e0849a5232943fe92fe04e010682147"}
Sep 09 20:11:02.691434 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:02.691218    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8" event={"ID":"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8","Type":"ContainerStarted","Data":"e9c747b12907c05899ec980790b2fb807d9de17c3187ea5decfb6da4fd7002a4"}
Sep 09 20:11:02.701519 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:02.701488    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lk6j8" event={"ID":"0e4de001-c1ee-47a0-a833-d6ffd119372c","Type":"ContainerStarted","Data":"259f0f2c7f44c2a6d9124ea2709dcd9ec92079eef61fc8a19e5af99f4ae47fc0"}
Sep 09 20:11:02.708828 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:02.708725    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hqtr" event={"ID":"91167988-d0fe-4f04-9716-21432d50f5b7","Type":"ContainerStarted","Data":"141f44cdda842bd26cd3aff6932417a8e1a329dc6ca3f70d18ef1e02a955440e"}
Sep 09 20:11:02.712029 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:02.712004    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm" event={"ID":"8a74a9a9-d77a-4dce-8c71-7588372245d8","Type":"ContainerStarted","Data":"e52d612b8609d60aff4bb57cb07c317aef8f611e6e5b41aa216ed600b6b01490"}
Sep 09 20:11:02.721045 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:02.721015    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f7kxd" event={"ID":"c47994f8-a068-4132-86bc-81d0c9ab827f","Type":"ContainerStarted","Data":"d3f66d2928aca0bc9fc1d4158e52df1ace60e66c4821c83171c3437199a74f29"}
Sep 09 20:11:02.732491 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:02.732442    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-84.ec2.internal" event={"ID":"8226d39d61364b61ba5e1ad9f7514144","Type":"ContainerStarted","Data":"bf9f5a243f4d41d1bacf3b21f8ddde86b620084dff43128911519a62b3b59462"}
Sep 09 20:11:02.734939 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:02.734912    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f28rc" event={"ID":"bc4b3154-62b5-4d79-bfe4-673807839bf7","Type":"ContainerStarted","Data":"29132ea5add534953be4026fe0e45a83fd8fb6bce1c3ef87649cdaeae93e08d4"}
Sep 09 20:11:02.742827 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:02.742795    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qpmd2" event={"ID":"610a91de-a1e2-445d-8f72-48aba0d8a2bf","Type":"ContainerStarted","Data":"5932c140e27d6dd9ecc343ddec536c6d39dd9e3a5f42ccc1e30aa8aeac2d4b58"}
Sep 09 20:11:02.749184 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:02.749126    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-84.ec2.internal" podStartSLOduration=3.7491076469999998 podStartE2EDuration="3.749107647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:02.748952989 +0000 UTC m=+4.754129982" watchObservedRunningTime="2026-09-09 20:11:02.749107647 +0000 UTC m=+4.754284644"
Sep 09 20:11:03.629991 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:03.629954    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:03.630426 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:03.630086    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vfht" podUID="7e3d0490-5d57-42a1-858f-1e6bf8830f4c"
Sep 09 20:11:03.630529 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:03.630510    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:03.630648 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:03.630611    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8q9m" podUID="5d3756a7-d67d-48ef-aa34-a792650e747d"
Sep 09 20:11:03.770252 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:03.770221    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal" event={"ID":"45419fbd63c226915cb25a2cb6311f7e","Type":"ContainerStarted","Data":"1ac556cb1ac13c7c2aa48e08e0789266f5aaa81dd9c960ddf4629584ba1b7ba6"}
Sep 09 20:11:03.785977 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:03.785920    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-84.ec2.internal" podStartSLOduration=4.785900071 podStartE2EDuration="4.785900071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:03.785530213 +0000 UTC m=+5.790707210" watchObservedRunningTime="2026-09-09 20:11:03.785900071 +0000 UTC m=+5.791077066"
Sep 09 20:11:04.029814 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:04.029779    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs\") pod \"network-metrics-daemon-9vfht\" (UID: \"7e3d0490-5d57-42a1-858f-1e6bf8830f4c\") " pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:04.030005 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:04.029985    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:04.030079 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:04.030064    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs podName:7e3d0490-5d57-42a1-858f-1e6bf8830f4c nodeName:}" failed. No retries permitted until 2026-09-09 20:11:08.030044221 +0000 UTC m=+10.035221217 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs") pod "network-metrics-daemon-9vfht" (UID: "7e3d0490-5d57-42a1-858f-1e6bf8830f4c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:04.130598 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:04.130559    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d3756a7-d67d-48ef-aa34-a792650e747d-original-pull-secret\") pod \"global-pull-secret-syncer-z8q9m\" (UID: \"5d3756a7-d67d-48ef-aa34-a792650e747d\") " pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:04.130598 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:04.130611    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2w9j\" (UniqueName: \"kubernetes.io/projected/6b9953bc-b5e1-4b30-a926-c1a13d4677a7-kube-api-access-g2w9j\") pod \"network-check-target-zcrsj\" (UID: \"6b9953bc-b5e1-4b30-a926-c1a13d4677a7\") " pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:04.130859 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:04.130810    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:04.130859 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:04.130833    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:04.130859 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:04.130847    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g2w9j for pod openshift-network-diagnostics/network-check-target-zcrsj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:04.131018 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:04.130906    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b9953bc-b5e1-4b30-a926-c1a13d4677a7-kube-api-access-g2w9j podName:6b9953bc-b5e1-4b30-a926-c1a13d4677a7 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:08.130888516 +0000 UTC m=+10.136065506 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2w9j" (UniqueName: "kubernetes.io/projected/6b9953bc-b5e1-4b30-a926-c1a13d4677a7-kube-api-access-g2w9j") pod "network-check-target-zcrsj" (UID: "6b9953bc-b5e1-4b30-a926-c1a13d4677a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:04.131319 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:04.131298    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:04.131380 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:04.131358    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3756a7-d67d-48ef-aa34-a792650e747d-original-pull-secret podName:5d3756a7-d67d-48ef-aa34-a792650e747d nodeName:}" failed. No retries permitted until 2026-09-09 20:11:08.131342142 +0000 UTC m=+10.136519132 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d3756a7-d67d-48ef-aa34-a792650e747d-original-pull-secret") pod "global-pull-secret-syncer-z8q9m" (UID: "5d3756a7-d67d-48ef-aa34-a792650e747d") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:04.629667 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:04.629633    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:04.629873 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:04.629771    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcrsj" podUID="6b9953bc-b5e1-4b30-a926-c1a13d4677a7"
Sep 09 20:11:05.629935 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:05.629179    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:05.629935 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:05.629309    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8q9m" podUID="5d3756a7-d67d-48ef-aa34-a792650e747d"
Sep 09 20:11:05.629935 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:05.629706    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:05.629935 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:05.629848    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vfht" podUID="7e3d0490-5d57-42a1-858f-1e6bf8830f4c"
Sep 09 20:11:06.629857 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:06.629733    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:06.629857 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:06.629854    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcrsj" podUID="6b9953bc-b5e1-4b30-a926-c1a13d4677a7"
Sep 09 20:11:07.629983 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:07.629946    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:07.630153 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:07.629994    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:07.630153 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:07.630098    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vfht" podUID="7e3d0490-5d57-42a1-858f-1e6bf8830f4c"
Sep 09 20:11:07.630605 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:07.630234    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8q9m" podUID="5d3756a7-d67d-48ef-aa34-a792650e747d"
Sep 09 20:11:08.067188 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:08.067149    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs\") pod \"network-metrics-daemon-9vfht\" (UID: \"7e3d0490-5d57-42a1-858f-1e6bf8830f4c\") " pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:08.067371 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:08.067289    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:08.067371 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:08.067360    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs podName:7e3d0490-5d57-42a1-858f-1e6bf8830f4c nodeName:}" failed. No retries permitted until 2026-09-09 20:11:16.067339328 +0000 UTC m=+18.072516316 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs") pod "network-metrics-daemon-9vfht" (UID: "7e3d0490-5d57-42a1-858f-1e6bf8830f4c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:08.168570 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:08.168528    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d3756a7-d67d-48ef-aa34-a792650e747d-original-pull-secret\") pod \"global-pull-secret-syncer-z8q9m\" (UID: \"5d3756a7-d67d-48ef-aa34-a792650e747d\") " pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:08.168570 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:08.168575    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2w9j\" (UniqueName: \"kubernetes.io/projected/6b9953bc-b5e1-4b30-a926-c1a13d4677a7-kube-api-access-g2w9j\") pod \"network-check-target-zcrsj\" (UID: \"6b9953bc-b5e1-4b30-a926-c1a13d4677a7\") " pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:08.168810 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:08.168754    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:08.168810 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:08.168773    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:08.168810 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:08.168786    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g2w9j for pod openshift-network-diagnostics/network-check-target-zcrsj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:08.169006 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:08.168842    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b9953bc-b5e1-4b30-a926-c1a13d4677a7-kube-api-access-g2w9j podName:6b9953bc-b5e1-4b30-a926-c1a13d4677a7 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:16.168824735 +0000 UTC m=+18.174001726 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2w9j" (UniqueName: "kubernetes.io/projected/6b9953bc-b5e1-4b30-a926-c1a13d4677a7-kube-api-access-g2w9j") pod "network-check-target-zcrsj" (UID: "6b9953bc-b5e1-4b30-a926-c1a13d4677a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:08.169326 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:08.169218    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:08.169326 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:08.169286    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3756a7-d67d-48ef-aa34-a792650e747d-original-pull-secret podName:5d3756a7-d67d-48ef-aa34-a792650e747d nodeName:}" failed. No retries permitted until 2026-09-09 20:11:16.169269825 +0000 UTC m=+18.174446815 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d3756a7-d67d-48ef-aa34-a792650e747d-original-pull-secret") pod "global-pull-secret-syncer-z8q9m" (UID: "5d3756a7-d67d-48ef-aa34-a792650e747d") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:08.634050 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:08.632392    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:08.634050 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:08.632551    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcrsj" podUID="6b9953bc-b5e1-4b30-a926-c1a13d4677a7"
Sep 09 20:11:09.630041 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:09.629347    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:09.630041 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:09.629506    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vfht" podUID="7e3d0490-5d57-42a1-858f-1e6bf8830f4c"
Sep 09 20:11:09.630041 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:09.629904    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:09.630041 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:09.629994    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8q9m" podUID="5d3756a7-d67d-48ef-aa34-a792650e747d"
Sep 09 20:11:10.629884 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:10.629849    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:10.630267 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:10.629977    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcrsj" podUID="6b9953bc-b5e1-4b30-a926-c1a13d4677a7"
Sep 09 20:11:11.629944 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:11.629911    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:11.630308 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:11.629953    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:11.630308 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:11.630028    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vfht" podUID="7e3d0490-5d57-42a1-858f-1e6bf8830f4c"
Sep 09 20:11:11.630308 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:11.630139    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8q9m" podUID="5d3756a7-d67d-48ef-aa34-a792650e747d"
Sep 09 20:11:12.629554 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:12.629471    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:12.629705 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:12.629590    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcrsj" podUID="6b9953bc-b5e1-4b30-a926-c1a13d4677a7"
Sep 09 20:11:13.629175 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:13.629134    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:13.629642 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:13.629274    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8q9m" podUID="5d3756a7-d67d-48ef-aa34-a792650e747d"
Sep 09 20:11:13.629642 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:13.629383    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:13.629642 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:13.629527    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vfht" podUID="7e3d0490-5d57-42a1-858f-1e6bf8830f4c"
Sep 09 20:11:14.629837 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:14.629799    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:14.630285 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:14.629946    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcrsj" podUID="6b9953bc-b5e1-4b30-a926-c1a13d4677a7"
Sep 09 20:11:15.629361 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:15.629330    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:15.629361 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:15.629354    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:15.629635 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:15.629447    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8q9m" podUID="5d3756a7-d67d-48ef-aa34-a792650e747d"
Sep 09 20:11:15.629635 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:15.629583    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vfht" podUID="7e3d0490-5d57-42a1-858f-1e6bf8830f4c"
Sep 09 20:11:16.129749 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:16.129717    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs\") pod \"network-metrics-daemon-9vfht\" (UID: \"7e3d0490-5d57-42a1-858f-1e6bf8830f4c\") " pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:16.130127 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:16.129855    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:16.130127 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:16.129916    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs podName:7e3d0490-5d57-42a1-858f-1e6bf8830f4c nodeName:}" failed. No retries permitted until 2026-09-09 20:11:32.129901116 +0000 UTC m=+34.135078107 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs") pod "network-metrics-daemon-9vfht" (UID: "7e3d0490-5d57-42a1-858f-1e6bf8830f4c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:16.230543 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:16.230514    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d3756a7-d67d-48ef-aa34-a792650e747d-original-pull-secret\") pod \"global-pull-secret-syncer-z8q9m\" (UID: \"5d3756a7-d67d-48ef-aa34-a792650e747d\") " pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:16.230543 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:16.230550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2w9j\" (UniqueName: \"kubernetes.io/projected/6b9953bc-b5e1-4b30-a926-c1a13d4677a7-kube-api-access-g2w9j\") pod \"network-check-target-zcrsj\" (UID: \"6b9953bc-b5e1-4b30-a926-c1a13d4677a7\") " pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:16.230785 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:16.230665    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:16.230785 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:16.230673    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:16.230785 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:16.230748    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3756a7-d67d-48ef-aa34-a792650e747d-original-pull-secret podName:5d3756a7-d67d-48ef-aa34-a792650e747d nodeName:}" failed. No retries permitted until 2026-09-09 20:11:32.230727435 +0000 UTC m=+34.235904414 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d3756a7-d67d-48ef-aa34-a792650e747d-original-pull-secret") pod "global-pull-secret-syncer-z8q9m" (UID: "5d3756a7-d67d-48ef-aa34-a792650e747d") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:16.230785 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:16.230682    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:16.230785 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:16.230783    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g2w9j for pod openshift-network-diagnostics/network-check-target-zcrsj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:16.231031 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:16.230833    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b9953bc-b5e1-4b30-a926-c1a13d4677a7-kube-api-access-g2w9j podName:6b9953bc-b5e1-4b30-a926-c1a13d4677a7 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:32.230818813 +0000 UTC m=+34.235995806 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2w9j" (UniqueName: "kubernetes.io/projected/6b9953bc-b5e1-4b30-a926-c1a13d4677a7-kube-api-access-g2w9j") pod "network-check-target-zcrsj" (UID: "6b9953bc-b5e1-4b30-a926-c1a13d4677a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:16.629205 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:16.629174    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:16.629365 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:16.629321    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcrsj" podUID="6b9953bc-b5e1-4b30-a926-c1a13d4677a7"
Sep 09 20:11:17.629820 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:17.629789    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:17.630254 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:17.629910    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vfht" podUID="7e3d0490-5d57-42a1-858f-1e6bf8830f4c"
Sep 09 20:11:17.630254 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:17.629968    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:17.630254 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:17.630058    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8q9m" podUID="5d3756a7-d67d-48ef-aa34-a792650e747d"
Sep 09 20:11:18.630660 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:18.630629    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:18.631193 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:18.630728    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcrsj" podUID="6b9953bc-b5e1-4b30-a926-c1a13d4677a7"
Sep 09 20:11:19.630170 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:19.630021    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:19.630260 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:19.630018    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:19.630305 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:19.630256    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vfht" podUID="7e3d0490-5d57-42a1-858f-1e6bf8830f4c"
Sep 09 20:11:19.630378 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:19.630344    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8q9m" podUID="5d3756a7-d67d-48ef-aa34-a792650e747d"
Sep 09 20:11:19.805539 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:19.805442    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qpmd2" event={"ID":"610a91de-a1e2-445d-8f72-48aba0d8a2bf","Type":"ContainerStarted","Data":"bc8ec30534a20a2166f37ffd527d5d7e21d60b106c29aecbf622357cfee6295a"}
Sep 09 20:11:19.807227 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:19.807206    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l7rwd" event={"ID":"297425e2-5552-4654-b41b-336d88d5a56c","Type":"ContainerStarted","Data":"7f0a1e0c5e0842d666a06c5d0bba779492c8b1f14b95f3a3f5c35c8af2fae389"}
Sep 09 20:11:19.808773 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:19.808751    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4k95l" event={"ID":"df278843-3e4c-47b6-ba9d-6def44cf12d8","Type":"ContainerStarted","Data":"cdfd7d61af7e52ecb16fe69a8f4b017b915d7e9ccd8e5d509fc287253f991736"}
Sep 09 20:11:19.810573 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:19.810556    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 20:11:19.810964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:19.810941    2581 generic.go:356] "Generic (PLEG): container finished" podID="e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8" containerID="f13ed2c2b027858291aeb068bb714f53940101c9d3576806094780239976d221" exitCode=1
Sep 09 20:11:19.811038 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:19.811016    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8" event={"ID":"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8","Type":"ContainerDied","Data":"f13ed2c2b027858291aeb068bb714f53940101c9d3576806094780239976d221"}
Sep 09 20:11:19.811143 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:19.811082    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8" event={"ID":"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8","Type":"ContainerStarted","Data":"b693ed43a3c9e3c82f12ab504ae32e6551231284e612e64247394d9004857d77"}
Sep 09 20:11:19.812822 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:19.812797    2581 generic.go:356] "Generic (PLEG): container finished" podID="91167988-d0fe-4f04-9716-21432d50f5b7" containerID="4096df51e27e5f6c32f0cb0d473ebaa686d9d97203b2f9a976de7ee6feb1d2fa" exitCode=0
Sep 09 20:11:19.812904 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:19.812883    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hqtr" event={"ID":"91167988-d0fe-4f04-9716-21432d50f5b7","Type":"ContainerDied","Data":"4096df51e27e5f6c32f0cb0d473ebaa686d9d97203b2f9a976de7ee6feb1d2fa"}
Sep 09 20:11:19.814380 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:19.814360    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm" event={"ID":"8a74a9a9-d77a-4dce-8c71-7588372245d8","Type":"ContainerStarted","Data":"9ded565246f334fa26cb9abfe7fd35362510583b2bc1d5cbb7e1f8504b91260a"}
Sep 09 20:11:19.815953 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:19.815931    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f28rc" event={"ID":"bc4b3154-62b5-4d79-bfe4-673807839bf7","Type":"ContainerStarted","Data":"db8bd88328252bc682f2c624166b03e729bcea8513163417e7a631d1a850f09f"}
Sep 09 20:11:19.843248 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:19.843206    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qpmd2" podStartSLOduration=4.278795234 podStartE2EDuration="21.843192782s" totalImagesPullingTime="17.564397548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:10:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:19.822793465 +0000 UTC m=+21.827970460" watchObservedRunningTime="2026-09-09 20:11:19.843192782 +0000 UTC m=+21.848369789"
Sep 09 20:11:19.843773 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:19.843748    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4k95l" podStartSLOduration=4.266950932 podStartE2EDuration="21.843741512s" totalImagesPullingTime="17.57679058s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:10:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:19.843160979 +0000 UTC m=+21.848337973" watchObservedRunningTime="2026-09-09 20:11:19.843741512 +0000 UTC m=+21.848918507"
Sep 09 20:11:19.858754 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:19.858716    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-f28rc" podStartSLOduration=3.331593651 podStartE2EDuration="20.858706868s" totalImagesPullingTime="17.527113217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:19.858373054 +0000 UTC m=+21.863550048" watchObservedRunningTime="2026-09-09 20:11:19.858706868 +0000 UTC m=+21.863883860"
Sep 09 20:11:19.877056 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:19.877018    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-l7rwd" podStartSLOduration=3.306823588 podStartE2EDuration="20.877005112s" totalImagesPullingTime="17.570181524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:19.87672995 +0000 UTC m=+21.881906946" watchObservedRunningTime="2026-09-09 20:11:19.877005112 +0000 UTC m=+21.882182107"
Sep 09 20:11:20.629871 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:20.629804    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:20.630015 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:20.629908    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcrsj" podUID="6b9953bc-b5e1-4b30-a926-c1a13d4677a7"
Sep 09 20:11:20.819504 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:20.819450    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f7kxd" event={"ID":"c47994f8-a068-4132-86bc-81d0c9ab827f","Type":"ContainerStarted","Data":"4cb3de254a96de7644f8942832da28ebe0707dcbb500e2a8b7e8b4c12bc265c5"}
Sep 09 20:11:20.822663 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:20.822642    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 20:11:20.823106 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:20.823079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8" event={"ID":"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8","Type":"ContainerStarted","Data":"895a96beb0181b999599ac91ea0c09ca04b4e9988c7a9d121d662fe06e9375cf"}
Sep 09 20:11:20.823228 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:20.823158    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8" event={"ID":"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8","Type":"ContainerStarted","Data":"d31f3b07d121fd25864b584b2e03127c3fd9b271fd8bf359fc9bcb5eab75b1a9"}
Sep 09 20:11:20.823228 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:20.823209    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8" event={"ID":"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8","Type":"ContainerStarted","Data":"ae831283d809f8a9aa77588bf023372ee976a5d88f836d73071b5026654ca8c9"}
Sep 09 20:11:20.823337 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:20.823255    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8" event={"ID":"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8","Type":"ContainerStarted","Data":"3bb574f745c5955c9f976c63bf9272719572ba97f8e14b5f4faf4cd19303e2b0"}
Sep 09 20:11:20.824835 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:20.824769    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lk6j8" event={"ID":"0e4de001-c1ee-47a0-a833-d6ffd119372c","Type":"ContainerStarted","Data":"c60df62e596b1a232abcd83d8a037c1e31b73f2316916420b2099416586591c3"}
Sep 09 20:11:20.838899 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:20.838851    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-f7kxd" podStartSLOduration=4.69495236 podStartE2EDuration="22.83883464s" totalImagesPullingTime="18.14388228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:10:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:20.837903299 +0000 UTC m=+22.843080293" watchObservedRunningTime="2026-09-09 20:11:20.83883464 +0000 UTC m=+22.844011637"
Sep 09 20:11:20.853337 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:20.853293    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lk6j8" podStartSLOduration=5.28376041 podStartE2EDuration="22.853281666s" totalImagesPullingTime="17.569521256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:10:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:20.85259287 +0000 UTC m=+22.857769866" watchObservedRunningTime="2026-09-09 20:11:20.853281666 +0000 UTC m=+22.858458660"
Sep 09 20:11:21.009902 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:21.009860    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 20:11:21.440915 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:21.440803    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T20:11:21.009879942Z","UUID":"03c39c1f-b9ff-4150-8fc6-ff231dd7a786","Handler":null,"Name":"","Endpoint":""}
Sep 09 20:11:21.442469 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:21.442436    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 20:11:21.442598 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:21.442487    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 20:11:21.629724 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:21.629624    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:21.629724 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:21.629666    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:21.629947 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:21.629751    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vfht" podUID="7e3d0490-5d57-42a1-858f-1e6bf8830f4c"
Sep 09 20:11:21.629947 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:21.629840    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8q9m" podUID="5d3756a7-d67d-48ef-aa34-a792650e747d"
Sep 09 20:11:21.829765 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:21.829728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm" event={"ID":"8a74a9a9-d77a-4dce-8c71-7588372245d8","Type":"ContainerStarted","Data":"73c5fc03a6aba88e2a1009186804ca5a8ac80baacb86e0e6c9b704bf647d8de0"}
Sep 09 20:11:22.629631 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:22.629602    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:22.629817 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:22.629715    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcrsj" podUID="6b9953bc-b5e1-4b30-a926-c1a13d4677a7"
Sep 09 20:11:22.836443 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:22.836416    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 20:11:22.836904 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:22.836876    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8" event={"ID":"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8","Type":"ContainerStarted","Data":"4d769f3ea179d5b2faad61271b604cd0980b3c155c6e43ba99947b598e83a21c"}
Sep 09 20:11:22.839138 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:22.839111    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm" event={"ID":"8a74a9a9-d77a-4dce-8c71-7588372245d8","Type":"ContainerStarted","Data":"cb9f5f9f942ac2e6e7e350a98edad5d786fa0a2e445b12b4f0875ee87dc7a607"}
Sep 09 20:11:22.858574 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:22.858526    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s5zpm" podStartSLOduration=4.058278027 podStartE2EDuration="23.858512192s" totalImagesPullingTime="19.800234165s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:10:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:22.858054811 +0000 UTC m=+24.863231808" watchObservedRunningTime="2026-09-09 20:11:22.858512192 +0000 UTC m=+24.863689187"
Sep 09 20:11:23.629546 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:23.629512    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:23.629546 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:23.629547    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:23.629777 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:23.629657    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8q9m" podUID="5d3756a7-d67d-48ef-aa34-a792650e747d"
Sep 09 20:11:23.629842 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:23.629782    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vfht" podUID="7e3d0490-5d57-42a1-858f-1e6bf8830f4c"
Sep 09 20:11:23.740027 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:23.739992    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-f28rc"
Sep 09 20:11:23.740725 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:23.740708    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-f28rc"
Sep 09 20:11:23.841789 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:23.841611    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-f28rc"
Sep 09 20:11:23.842225 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:23.842023    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f28rc"
Sep 09 20:11:24.629701 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:24.629482    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:24.629829 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:24.629803    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcrsj" podUID="6b9953bc-b5e1-4b30-a926-c1a13d4677a7"
Sep 09 20:11:24.845763 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:24.845691    2581 generic.go:356] "Generic (PLEG): container finished" podID="91167988-d0fe-4f04-9716-21432d50f5b7" containerID="1a74b5b0232ae0879cfc1c9cecfdcd757116154043cbc63e49dd9caad7d04688" exitCode=0
Sep 09 20:11:24.846592 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:24.845772    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hqtr" event={"ID":"91167988-d0fe-4f04-9716-21432d50f5b7","Type":"ContainerDied","Data":"1a74b5b0232ae0879cfc1c9cecfdcd757116154043cbc63e49dd9caad7d04688"}
Sep 09 20:11:24.849407 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:24.849387    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 20:11:24.849801 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:24.849778    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8" event={"ID":"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8","Type":"ContainerStarted","Data":"f5c7955bb39ef3efe73769bc333dcd0b97fb8b8e2c4da2f0034973e6b7092135"}
Sep 09 20:11:24.850349 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:24.850326    2581 scope.go:117] "RemoveContainer" containerID="f13ed2c2b027858291aeb068bb714f53940101c9d3576806094780239976d221"
Sep 09 20:11:25.629453 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:25.629412    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:25.629453 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:25.629420    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:25.629738 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:25.629568    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8q9m" podUID="5d3756a7-d67d-48ef-aa34-a792650e747d"
Sep 09 20:11:25.629913 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:25.629863    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vfht" podUID="7e3d0490-5d57-42a1-858f-1e6bf8830f4c"
Sep 09 20:11:25.855194 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:25.855035    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 20:11:25.855636 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:25.855480    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8" event={"ID":"e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8","Type":"ContainerStarted","Data":"72aa354f9661daf0476b3d9d23e61671071319578064352bb52f490284a15e21"}
Sep 09 20:11:25.855636 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:25.855592    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 20:11:25.855838 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:25.855813    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:25.855957 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:25.855879    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:25.871177 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:25.871156    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:25.871290 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:25.871253    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:25.885001 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:25.884887    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8" podStartSLOduration=10.293572344 podStartE2EDuration="27.884874341s" totalImagesPullingTime="17.591301997s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:10:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:25.883350713 +0000 UTC m=+27.888527708" watchObservedRunningTime="2026-09-09 20:11:25.884874341 +0000 UTC m=+27.890051360"
Sep 09 20:11:26.223575 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:26.223505    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:11:26.319912 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:26.319881    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z8q9m"]
Sep 09 20:11:26.320088 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:26.319997    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:26.320150 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:26.320110    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8q9m" podUID="5d3756a7-d67d-48ef-aa34-a792650e747d"
Sep 09 20:11:26.323125 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:26.323090    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zcrsj"]
Sep 09 20:11:26.323235 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:26.323198    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:26.323316 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:26.323295    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcrsj" podUID="6b9953bc-b5e1-4b30-a926-c1a13d4677a7"
Sep 09 20:11:26.323773 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:26.323749    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9vfht"]
Sep 09 20:11:26.323880 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:26.323844    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:26.323971 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:26.323949    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vfht" podUID="7e3d0490-5d57-42a1-858f-1e6bf8830f4c"
Sep 09 20:11:26.859185 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:26.859144    2581 generic.go:356] "Generic (PLEG): container finished" podID="91167988-d0fe-4f04-9716-21432d50f5b7" containerID="cd0535c91e05f18f86f36604cb43525fd39f7b7fa066b6e2a9488256b75a32fc" exitCode=0
Sep 09 20:11:26.859621 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:26.859224    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hqtr" event={"ID":"91167988-d0fe-4f04-9716-21432d50f5b7","Type":"ContainerDied","Data":"cd0535c91e05f18f86f36604cb43525fd39f7b7fa066b6e2a9488256b75a32fc"}
Sep 09 20:11:27.629887 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:27.629857    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:27.630047 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:27.629850    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:27.630047 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:27.629977    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8q9m" podUID="5d3756a7-d67d-48ef-aa34-a792650e747d"
Sep 09 20:11:27.630047 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:27.630039    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vfht" podUID="7e3d0490-5d57-42a1-858f-1e6bf8830f4c"
Sep 09 20:11:28.632382 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:28.632311    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:28.632816 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:28.632424    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcrsj" podUID="6b9953bc-b5e1-4b30-a926-c1a13d4677a7"
Sep 09 20:11:28.865846 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:28.865814    2581 generic.go:356] "Generic (PLEG): container finished" podID="91167988-d0fe-4f04-9716-21432d50f5b7" containerID="9a1f7d5b6a919e7a97d5c7276fd698c1d669f2a813ba2ddd8bec7aa374fdd5d6" exitCode=0
Sep 09 20:11:28.865991 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:28.865867    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hqtr" event={"ID":"91167988-d0fe-4f04-9716-21432d50f5b7","Type":"ContainerDied","Data":"9a1f7d5b6a919e7a97d5c7276fd698c1d669f2a813ba2ddd8bec7aa374fdd5d6"}
Sep 09 20:11:29.630252 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:29.630069    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:29.630415 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:29.630070    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:29.630415 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:29.630329    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8q9m" podUID="5d3756a7-d67d-48ef-aa34-a792650e747d"
Sep 09 20:11:29.630577 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:29.630426    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vfht" podUID="7e3d0490-5d57-42a1-858f-1e6bf8830f4c"
Sep 09 20:11:30.629954 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:30.629919    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:30.630375 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:30.630021    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcrsj" podUID="6b9953bc-b5e1-4b30-a926-c1a13d4677a7"
Sep 09 20:11:31.629249 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:31.629221    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:31.629448 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:31.629265    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:31.629448 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:31.629351    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vfht" podUID="7e3d0490-5d57-42a1-858f-1e6bf8830f4c"
Sep 09 20:11:31.629595 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:31.629477    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8q9m" podUID="5d3756a7-d67d-48ef-aa34-a792650e747d"
Sep 09 20:11:32.145391 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.145355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs\") pod \"network-metrics-daemon-9vfht\" (UID: \"7e3d0490-5d57-42a1-858f-1e6bf8830f4c\") " pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:32.145921 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:32.145602    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:32.145921 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:32.145668    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs podName:7e3d0490-5d57-42a1-858f-1e6bf8830f4c nodeName:}" failed. No retries permitted until 2026-09-09 20:12:04.145654298 +0000 UTC m=+66.150831271 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs") pod "network-metrics-daemon-9vfht" (UID: "7e3d0490-5d57-42a1-858f-1e6bf8830f4c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:32.245810 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.245775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d3756a7-d67d-48ef-aa34-a792650e747d-original-pull-secret\") pod \"global-pull-secret-syncer-z8q9m\" (UID: \"5d3756a7-d67d-48ef-aa34-a792650e747d\") " pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:32.245974 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.245826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2w9j\" (UniqueName: \"kubernetes.io/projected/6b9953bc-b5e1-4b30-a926-c1a13d4677a7-kube-api-access-g2w9j\") pod \"network-check-target-zcrsj\" (UID: \"6b9953bc-b5e1-4b30-a926-c1a13d4677a7\") " pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:32.245974 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:32.245932    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:32.245974 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:32.245940    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:32.245974 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:32.245959    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:32.245974 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:32.245971    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g2w9j for pod openshift-network-diagnostics/network-check-target-zcrsj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:32.246220 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:32.246001    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3756a7-d67d-48ef-aa34-a792650e747d-original-pull-secret podName:5d3756a7-d67d-48ef-aa34-a792650e747d nodeName:}" failed. No retries permitted until 2026-09-09 20:12:04.24598047 +0000 UTC m=+66.251157463 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d3756a7-d67d-48ef-aa34-a792650e747d-original-pull-secret") pod "global-pull-secret-syncer-z8q9m" (UID: "5d3756a7-d67d-48ef-aa34-a792650e747d") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:32.246220 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:32.246072    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b9953bc-b5e1-4b30-a926-c1a13d4677a7-kube-api-access-g2w9j podName:6b9953bc-b5e1-4b30-a926-c1a13d4677a7 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:04.246060184 +0000 UTC m=+66.251237172 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2w9j" (UniqueName: "kubernetes.io/projected/6b9953bc-b5e1-4b30-a926-c1a13d4677a7-kube-api-access-g2w9j") pod "network-check-target-zcrsj" (UID: "6b9953bc-b5e1-4b30-a926-c1a13d4677a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:32.629796 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.629764    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:32.629982 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:32.629899    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcrsj" podUID="6b9953bc-b5e1-4b30-a926-c1a13d4677a7"
Sep 09 20:11:32.742527 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.742500    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-84.ec2.internal" event="NodeReady"
Sep 09 20:11:32.742687 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.742642    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 20:11:32.788278 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.788239    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qd6j5"]
Sep 09 20:11:32.814149 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.814119    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4prkf"]
Sep 09 20:11:32.814314 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.814253    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qd6j5"
Sep 09 20:11:32.817411 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.817200    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ftsch\""
Sep 09 20:11:32.819770 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.819675    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 20:11:32.819899 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.819847    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 20:11:32.832119 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.832098    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qd6j5"]
Sep 09 20:11:32.832230 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.832132    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4prkf"]
Sep 09 20:11:32.832230 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.832219    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4prkf"
Sep 09 20:11:32.834924 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.834903    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7rf7m\""
Sep 09 20:11:32.834924 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.834914    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 20:11:32.834924 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.834932    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 20:11:32.835223 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.835207    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 20:11:32.950044 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.949955    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2bff29a8-0416-4090-af52-b62924509cd0-config-volume\") pod \"dns-default-qd6j5\" (UID: \"2bff29a8-0416-4090-af52-b62924509cd0\") " pod="openshift-dns/dns-default-qd6j5"
Sep 09 20:11:32.950044 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.949996    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bff29a8-0416-4090-af52-b62924509cd0-tmp-dir\") pod \"dns-default-qd6j5\" (UID: \"2bff29a8-0416-4090-af52-b62924509cd0\") " pod="openshift-dns/dns-default-qd6j5"
Sep 09 20:11:32.950254 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.950096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4rls\" (UniqueName: \"kubernetes.io/projected/6d63ce9d-307d-4a8f-bcf1-9f53a3d3c956-kube-api-access-m4rls\") pod \"ingress-canary-4prkf\" (UID: \"6d63ce9d-307d-4a8f-bcf1-9f53a3d3c956\") " pod="openshift-ingress-canary/ingress-canary-4prkf"
Sep 09 20:11:32.950254 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.950139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bff29a8-0416-4090-af52-b62924509cd0-metrics-tls\") pod \"dns-default-qd6j5\" (UID: \"2bff29a8-0416-4090-af52-b62924509cd0\") " pod="openshift-dns/dns-default-qd6j5"
Sep 09 20:11:32.950254 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.950241    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcshj\" (UniqueName: \"kubernetes.io/projected/2bff29a8-0416-4090-af52-b62924509cd0-kube-api-access-kcshj\") pod \"dns-default-qd6j5\" (UID: \"2bff29a8-0416-4090-af52-b62924509cd0\") " pod="openshift-dns/dns-default-qd6j5"
Sep 09 20:11:32.950393 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:32.950267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d63ce9d-307d-4a8f-bcf1-9f53a3d3c956-cert\") pod \"ingress-canary-4prkf\" (UID: \"6d63ce9d-307d-4a8f-bcf1-9f53a3d3c956\") " pod="openshift-ingress-canary/ingress-canary-4prkf"
Sep 09 20:11:33.051273 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:33.051236    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcshj\" (UniqueName: \"kubernetes.io/projected/2bff29a8-0416-4090-af52-b62924509cd0-kube-api-access-kcshj\") pod \"dns-default-qd6j5\" (UID: \"2bff29a8-0416-4090-af52-b62924509cd0\") " pod="openshift-dns/dns-default-qd6j5"
Sep 09 20:11:33.051273 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:33.051280    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d63ce9d-307d-4a8f-bcf1-9f53a3d3c956-cert\") pod \"ingress-canary-4prkf\" (UID: \"6d63ce9d-307d-4a8f-bcf1-9f53a3d3c956\") " pod="openshift-ingress-canary/ingress-canary-4prkf"
Sep 09 20:11:33.051513 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:33.051363    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2bff29a8-0416-4090-af52-b62924509cd0-config-volume\") pod \"dns-default-qd6j5\" (UID: \"2bff29a8-0416-4090-af52-b62924509cd0\") " pod="openshift-dns/dns-default-qd6j5"
Sep 09 20:11:33.051513 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:33.051394    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bff29a8-0416-4090-af52-b62924509cd0-tmp-dir\") pod \"dns-default-qd6j5\" (UID: \"2bff29a8-0416-4090-af52-b62924509cd0\") " pod="openshift-dns/dns-default-qd6j5"
Sep 09 20:11:33.051513 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:33.051425    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4rls\" (UniqueName: \"kubernetes.io/projected/6d63ce9d-307d-4a8f-bcf1-9f53a3d3c956-kube-api-access-m4rls\") pod \"ingress-canary-4prkf\" (UID: \"6d63ce9d-307d-4a8f-bcf1-9f53a3d3c956\") " pod="openshift-ingress-canary/ingress-canary-4prkf"
Sep 09 20:11:33.051513 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:33.051441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bff29a8-0416-4090-af52-b62924509cd0-metrics-tls\") pod \"dns-default-qd6j5\" (UID: \"2bff29a8-0416-4090-af52-b62924509cd0\") " pod="openshift-dns/dns-default-qd6j5"
Sep 09 20:11:33.051653 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:33.051559    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:33.051653 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:33.051608    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bff29a8-0416-4090-af52-b62924509cd0-metrics-tls podName:2bff29a8-0416-4090-af52-b62924509cd0 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:33.55159393 +0000 UTC m=+35.556770916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2bff29a8-0416-4090-af52-b62924509cd0-metrics-tls") pod "dns-default-qd6j5" (UID: "2bff29a8-0416-4090-af52-b62924509cd0") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:33.051897 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:33.051873    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bff29a8-0416-4090-af52-b62924509cd0-tmp-dir\") pod \"dns-default-qd6j5\" (UID: \"2bff29a8-0416-4090-af52-b62924509cd0\") " pod="openshift-dns/dns-default-qd6j5"
Sep 09 20:11:33.052361 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:33.052336    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2bff29a8-0416-4090-af52-b62924509cd0-config-volume\") pod \"dns-default-qd6j5\" (UID: \"2bff29a8-0416-4090-af52-b62924509cd0\") " pod="openshift-dns/dns-default-qd6j5"
Sep 09 20:11:33.056005 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:33.055976    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d63ce9d-307d-4a8f-bcf1-9f53a3d3c956-cert\") pod \"ingress-canary-4prkf\" (UID: \"6d63ce9d-307d-4a8f-bcf1-9f53a3d3c956\") " pod="openshift-ingress-canary/ingress-canary-4prkf"
Sep 09 20:11:33.060800 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:33.060777    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4rls\" (UniqueName: \"kubernetes.io/projected/6d63ce9d-307d-4a8f-bcf1-9f53a3d3c956-kube-api-access-m4rls\") pod \"ingress-canary-4prkf\" (UID: \"6d63ce9d-307d-4a8f-bcf1-9f53a3d3c956\") " pod="openshift-ingress-canary/ingress-canary-4prkf"
Sep 09 20:11:33.061156 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:33.061136    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcshj\" (UniqueName: \"kubernetes.io/projected/2bff29a8-0416-4090-af52-b62924509cd0-kube-api-access-kcshj\") pod \"dns-default-qd6j5\" (UID: \"2bff29a8-0416-4090-af52-b62924509cd0\") " pod="openshift-dns/dns-default-qd6j5"
Sep 09 20:11:33.141661 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:33.141620    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4prkf"
Sep 09 20:11:33.553641 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:33.553473    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4prkf"]
Sep 09 20:11:33.555736 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:33.555655    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bff29a8-0416-4090-af52-b62924509cd0-metrics-tls\") pod \"dns-default-qd6j5\" (UID: \"2bff29a8-0416-4090-af52-b62924509cd0\") " pod="openshift-dns/dns-default-qd6j5"
Sep 09 20:11:33.555800 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:33.555786    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:33.555851 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:33.555842    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bff29a8-0416-4090-af52-b62924509cd0-metrics-tls podName:2bff29a8-0416-4090-af52-b62924509cd0 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:34.555827202 +0000 UTC m=+36.561004175 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2bff29a8-0416-4090-af52-b62924509cd0-metrics-tls") pod "dns-default-qd6j5" (UID: "2bff29a8-0416-4090-af52-b62924509cd0") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:33.557520 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:11:33.557498    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d63ce9d_307d_4a8f_bcf1_9f53a3d3c956.slice/crio-0b5f2f0d8e42be88c59ac93d5961b2161f0357a64024ac9b3e61622c7da30c8f WatchSource:0}: Error finding container 0b5f2f0d8e42be88c59ac93d5961b2161f0357a64024ac9b3e61622c7da30c8f: Status 404 returned error can't find the container with id 0b5f2f0d8e42be88c59ac93d5961b2161f0357a64024ac9b3e61622c7da30c8f
Sep 09 20:11:33.629889 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:33.629843    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:11:33.630050 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:33.629843    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:11:33.632589 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:33.632564    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:11:33.632734 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:33.632632    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4t9vt\""
Sep 09 20:11:33.632734 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:33.632662    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:11:33.878513 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:33.878384    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4prkf" event={"ID":"6d63ce9d-307d-4a8f-bcf1-9f53a3d3c956","Type":"ContainerStarted","Data":"0b5f2f0d8e42be88c59ac93d5961b2161f0357a64024ac9b3e61622c7da30c8f"}
Sep 09 20:11:33.880929 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:33.880906    2581 generic.go:356] "Generic (PLEG): container finished" podID="91167988-d0fe-4f04-9716-21432d50f5b7" containerID="9167e23d64e203e2328f581eed791946307c4592bcc8f2ccc24132eb89ce1e7d" exitCode=0
Sep 09 20:11:33.881026 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:33.880950    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hqtr" event={"ID":"91167988-d0fe-4f04-9716-21432d50f5b7","Type":"ContainerDied","Data":"9167e23d64e203e2328f581eed791946307c4592bcc8f2ccc24132eb89ce1e7d"}
Sep 09 20:11:34.563359 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:34.563322    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bff29a8-0416-4090-af52-b62924509cd0-metrics-tls\") pod \"dns-default-qd6j5\" (UID: \"2bff29a8-0416-4090-af52-b62924509cd0\") " pod="openshift-dns/dns-default-qd6j5"
Sep 09 20:11:34.563855 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:34.563450    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:34.563855 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:34.563541    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bff29a8-0416-4090-af52-b62924509cd0-metrics-tls podName:2bff29a8-0416-4090-af52-b62924509cd0 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:36.563516161 +0000 UTC m=+38.568693137 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2bff29a8-0416-4090-af52-b62924509cd0-metrics-tls") pod "dns-default-qd6j5" (UID: "2bff29a8-0416-4090-af52-b62924509cd0") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:34.629811 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:34.629777    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:11:34.632687 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:34.632650    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:11:34.632687 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:34.632681    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:11:34.632875 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:34.632728    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wr6wr\""
Sep 09 20:11:34.887821 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:34.887754    2581 generic.go:356] "Generic (PLEG): container finished" podID="91167988-d0fe-4f04-9716-21432d50f5b7" containerID="eca9ab02e1393c8d627fd6af3fa8f5cc9ac62aa0513bbab9eb0a833e8bf8686b" exitCode=0
Sep 09 20:11:34.887821 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:34.887806    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hqtr" event={"ID":"91167988-d0fe-4f04-9716-21432d50f5b7","Type":"ContainerDied","Data":"eca9ab02e1393c8d627fd6af3fa8f5cc9ac62aa0513bbab9eb0a833e8bf8686b"}
Sep 09 20:11:35.893358 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:35.893323    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hqtr" event={"ID":"91167988-d0fe-4f04-9716-21432d50f5b7","Type":"ContainerStarted","Data":"e0244467bdd9ac6aa9a0a6d38325f5cdd9b836ef67351488d4943292f5cee2a3"}
Sep 09 20:11:35.894787 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:35.894763    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4prkf" event={"ID":"6d63ce9d-307d-4a8f-bcf1-9f53a3d3c956","Type":"ContainerStarted","Data":"fcffb04ffdc5d832e3f632eab96ad7acfe1102a2d67443987212830bfa632cde"}
Sep 09 20:11:35.917428 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:35.917380    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6hqtr" podStartSLOduration=7.704024007 podStartE2EDuration="36.917365219s" totalImagesPullingTime="29.213341212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:10:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:35.915298024 +0000 UTC m=+37.920475020" watchObservedRunningTime="2026-09-09 20:11:35.917365219 +0000 UTC m=+37.922542215"
Sep 09 20:11:35.931528 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:35.931484    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4prkf" podStartSLOduration=1.9050557590000001 podStartE2EDuration="3.931452495s" totalImagesPullingTime="2.026396736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:35.930780608 +0000 UTC m=+37.935957605" watchObservedRunningTime="2026-09-09 20:11:35.931452495 +0000 UTC m=+37.936629494"
Sep 09 20:11:36.579214 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:36.579171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bff29a8-0416-4090-af52-b62924509cd0-metrics-tls\") pod \"dns-default-qd6j5\" (UID: \"2bff29a8-0416-4090-af52-b62924509cd0\") " pod="openshift-dns/dns-default-qd6j5"
Sep 09 20:11:36.579353 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:36.579310    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:36.579484 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:36.579372    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bff29a8-0416-4090-af52-b62924509cd0-metrics-tls podName:2bff29a8-0416-4090-af52-b62924509cd0 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:40.579356566 +0000 UTC m=+42.584533539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2bff29a8-0416-4090-af52-b62924509cd0-metrics-tls") pod "dns-default-qd6j5" (UID: "2bff29a8-0416-4090-af52-b62924509cd0") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:40.613647 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:40.613603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bff29a8-0416-4090-af52-b62924509cd0-metrics-tls\") pod \"dns-default-qd6j5\" (UID: \"2bff29a8-0416-4090-af52-b62924509cd0\") " pod="openshift-dns/dns-default-qd6j5"
Sep 09 20:11:40.614028 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:40.613724    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:40.614028 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:40.613777    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bff29a8-0416-4090-af52-b62924509cd0-metrics-tls podName:2bff29a8-0416-4090-af52-b62924509cd0 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:48.613763874 +0000 UTC m=+50.618940847 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2bff29a8-0416-4090-af52-b62924509cd0-metrics-tls") pod "dns-default-qd6j5" (UID: "2bff29a8-0416-4090-af52-b62924509cd0") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:48.665674 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:48.665638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bff29a8-0416-4090-af52-b62924509cd0-metrics-tls\") pod \"dns-default-qd6j5\" (UID: \"2bff29a8-0416-4090-af52-b62924509cd0\") " pod="openshift-dns/dns-default-qd6j5"
Sep 09 20:11:48.666067 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:48.665751    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:48.666067 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:11:48.665813    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bff29a8-0416-4090-af52-b62924509cd0-metrics-tls podName:2bff29a8-0416-4090-af52-b62924509cd0 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:04.66579814 +0000 UTC m=+66.670975132 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2bff29a8-0416-4090-af52-b62924509cd0-metrics-tls") pod "dns-default-qd6j5" (UID: "2bff29a8-0416-4090-af52-b62924509cd0") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:57.877910 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:11:57.877873    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jqqq8"
Sep 09 20:12:01.731332 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.731291    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2fqt"]
Sep 09 20:12:01.733727 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.733706    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j4s66"]
Sep 09 20:12:01.733837 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.733818    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2fqt"
Sep 09 20:12:01.735165 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.735145    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j4s66"
Sep 09 20:12:01.736250 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.736228    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:12:01.736365 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.736325    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 09 20:12:01.736896 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.736871    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-tfmnf\""
Sep 09 20:12:01.737426 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.737405    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 09 20:12:01.737944 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.737924    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 09 20:12:01.738298 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.738277    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:12:01.739447 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.739422    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-d24pg\""
Sep 09 20:12:01.746247 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.746219    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2fqt"]
Sep 09 20:12:01.746918 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.746898    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j4s66"]
Sep 09 20:12:01.754850 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.754826    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-ldv8m"]
Sep 09 20:12:01.755063 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.755044    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29gwj\" (UniqueName: \"kubernetes.io/projected/ae4c0694-26b1-4a2a-a5fa-b134191927fc-kube-api-access-29gwj\") pod \"volume-data-source-validator-695c895bd6-r2fqt\" (UID: \"ae4c0694-26b1-4a2a-a5fa-b134191927fc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2fqt"
Sep 09 20:12:01.755135 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.755118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/43ddfbde-d8f2-4412-941e-98bab3b4856d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j4s66\" (UID: \"43ddfbde-d8f2-4412-941e-98bab3b4856d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j4s66"
Sep 09 20:12:01.755172 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.755148    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4hch\" (UniqueName: \"kubernetes.io/projected/43ddfbde-d8f2-4412-941e-98bab3b4856d-kube-api-access-s4hch\") pod \"cluster-samples-operator-7cdcfcf76c-j4s66\" (UID: \"43ddfbde-d8f2-4412-941e-98bab3b4856d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j4s66"
Sep 09 20:12:01.756425 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.756409    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-ldv8m"
Sep 09 20:12:01.758990 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.758973    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 09 20:12:01.759898 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.759880    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 09 20:12:01.760659 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.760640    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 09 20:12:01.760805 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.760663    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:12:01.761032 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.761003    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-gb44c\""
Sep 09 20:12:01.764795 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.764775    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 09 20:12:01.769579 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.769557    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-ldv8m"]
Sep 09 20:12:01.855471 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.855423    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b60c4c0d-ce10-44ef-a2e7-a7562b103e46-config\") pod \"console-operator-85b454874b-ldv8m\" (UID: \"b60c4c0d-ce10-44ef-a2e7-a7562b103e46\") " pod="openshift-console-operator/console-operator-85b454874b-ldv8m"
Sep 09 20:12:01.855633 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.855482    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b60c4c0d-ce10-44ef-a2e7-a7562b103e46-trusted-ca\") pod \"console-operator-85b454874b-ldv8m\" (UID: \"b60c4c0d-ce10-44ef-a2e7-a7562b103e46\") " pod="openshift-console-operator/console-operator-85b454874b-ldv8m"
Sep 09 20:12:01.855633 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.855542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/43ddfbde-d8f2-4412-941e-98bab3b4856d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j4s66\" (UID: \"43ddfbde-d8f2-4412-941e-98bab3b4856d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j4s66"
Sep 09 20:12:01.855633 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.855585    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4hch\" (UniqueName: \"kubernetes.io/projected/43ddfbde-d8f2-4412-941e-98bab3b4856d-kube-api-access-s4hch\") pod \"cluster-samples-operator-7cdcfcf76c-j4s66\" (UID: \"43ddfbde-d8f2-4412-941e-98bab3b4856d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j4s66"
Sep 09 20:12:01.855633 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.855626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29gwj\" (UniqueName: \"kubernetes.io/projected/ae4c0694-26b1-4a2a-a5fa-b134191927fc-kube-api-access-29gwj\") pod \"volume-data-source-validator-695c895bd6-r2fqt\" (UID: \"ae4c0694-26b1-4a2a-a5fa-b134191927fc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2fqt"
Sep 09 20:12:01.855763 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:01.855652    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:12:01.855763 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.855659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b60c4c0d-ce10-44ef-a2e7-a7562b103e46-serving-cert\") pod \"console-operator-85b454874b-ldv8m\" (UID: \"b60c4c0d-ce10-44ef-a2e7-a7562b103e46\") " pod="openshift-console-operator/console-operator-85b454874b-ldv8m"
Sep 09 20:12:01.855763 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.855706    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsswv\" (UniqueName: \"kubernetes.io/projected/b60c4c0d-ce10-44ef-a2e7-a7562b103e46-kube-api-access-gsswv\") pod \"console-operator-85b454874b-ldv8m\" (UID: \"b60c4c0d-ce10-44ef-a2e7-a7562b103e46\") " pod="openshift-console-operator/console-operator-85b454874b-ldv8m"
Sep 09 20:12:01.855763 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:01.855720    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43ddfbde-d8f2-4412-941e-98bab3b4856d-samples-operator-tls podName:43ddfbde-d8f2-4412-941e-98bab3b4856d nodeName:}" failed. No retries permitted until 2026-09-09 20:12:02.355705089 +0000 UTC m=+64.360882066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/43ddfbde-d8f2-4412-941e-98bab3b4856d-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-j4s66" (UID: "43ddfbde-d8f2-4412-941e-98bab3b4856d") : secret "samples-operator-tls" not found
Sep 09 20:12:01.860851 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.860824    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8vcxj"]
Sep 09 20:12:01.862751 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.862736    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8vcxj"
Sep 09 20:12:01.864174 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.864151    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-974b4ddbd-dpnx9"]
Sep 09 20:12:01.865723 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.865710    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:01.866278 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.866243    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 09 20:12:01.866377 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.866318    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 09 20:12:01.866377 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.866245    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 09 20:12:01.866601 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.866581    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-jjbwb\""
Sep 09 20:12:01.867058 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.867045    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:12:01.870227 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.870207    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 20:12:01.870227 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.870223    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 20:12:01.870432 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.870418    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 20:12:01.870506 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.870439    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gslg8\""
Sep 09 20:12:01.875816 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.875795    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 20:12:01.878221 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.878200    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8vcxj"]
Sep 09 20:12:01.879189 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.879167    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4hch\" (UniqueName: \"kubernetes.io/projected/43ddfbde-d8f2-4412-941e-98bab3b4856d-kube-api-access-s4hch\") pod \"cluster-samples-operator-7cdcfcf76c-j4s66\" (UID: \"43ddfbde-d8f2-4412-941e-98bab3b4856d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j4s66"
Sep 09 20:12:01.879302 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.879284    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29gwj\" (UniqueName: \"kubernetes.io/projected/ae4c0694-26b1-4a2a-a5fa-b134191927fc-kube-api-access-29gwj\") pod \"volume-data-source-validator-695c895bd6-r2fqt\" (UID: \"ae4c0694-26b1-4a2a-a5fa-b134191927fc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2fqt"
Sep 09 20:12:01.886149 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.886123    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-974b4ddbd-dpnx9"]
Sep 09 20:12:01.956051 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.956018    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-certificates\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:01.956224 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.956104    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b60c4c0d-ce10-44ef-a2e7-a7562b103e46-serving-cert\") pod \"console-operator-85b454874b-ldv8m\" (UID: \"b60c4c0d-ce10-44ef-a2e7-a7562b103e46\") " pod="openshift-console-operator/console-operator-85b454874b-ldv8m"
Sep 09 20:12:01.956224 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.956162    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-tls\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:01.956224 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.956219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsswv\" (UniqueName: \"kubernetes.io/projected/b60c4c0d-ce10-44ef-a2e7-a7562b103e46-kube-api-access-gsswv\") pod \"console-operator-85b454874b-ldv8m\" (UID: \"b60c4c0d-ce10-44ef-a2e7-a7562b103e46\") " pod="openshift-console-operator/console-operator-85b454874b-ldv8m"
Sep 09 20:12:01.956340 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.956250    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b60c4c0d-ce10-44ef-a2e7-a7562b103e46-config\") pod \"console-operator-85b454874b-ldv8m\" (UID: \"b60c4c0d-ce10-44ef-a2e7-a7562b103e46\") " pod="openshift-console-operator/console-operator-85b454874b-ldv8m"
Sep 09 20:12:01.956340 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.956286    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b60c4c0d-ce10-44ef-a2e7-a7562b103e46-trusted-ca\") pod \"console-operator-85b454874b-ldv8m\" (UID: \"b60c4c0d-ce10-44ef-a2e7-a7562b103e46\") " pod="openshift-console-operator/console-operator-85b454874b-ldv8m"
Sep 09 20:12:01.956340 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.956311    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmsxd\" (UniqueName: \"kubernetes.io/projected/71bd9823-f654-4f3d-97b4-9c186b4e4028-kube-api-access-zmsxd\") pod \"kube-storage-version-migrator-operator-8d7dd4675-8vcxj\" (UID: \"71bd9823-f654-4f3d-97b4-9c186b4e4028\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8vcxj"
Sep 09 20:12:01.956340 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.956333    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/113a1f91-274e-4844-9598-21a40cfdcdb2-ca-trust-extracted\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:01.956540 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.956379    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71bd9823-f654-4f3d-97b4-9c186b4e4028-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-8vcxj\" (UID: \"71bd9823-f654-4f3d-97b4-9c186b4e4028\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8vcxj"
Sep 09 20:12:01.957204 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.956791    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/113a1f91-274e-4844-9598-21a40cfdcdb2-installation-pull-secrets\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:01.957204 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.956870    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccw8c\" (UniqueName: \"kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-kube-api-access-ccw8c\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:01.957204 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.956904    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71bd9823-f654-4f3d-97b4-9c186b4e4028-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-8vcxj\" (UID: \"71bd9823-f654-4f3d-97b4-9c186b4e4028\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8vcxj"
Sep 09 20:12:01.957204 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.956956    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/113a1f91-274e-4844-9598-21a40cfdcdb2-image-registry-private-configuration\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:01.957204 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.956993    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/113a1f91-274e-4844-9598-21a40cfdcdb2-trusted-ca\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:01.957204 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.957042    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-bound-sa-token\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:01.957614 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.957482    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b60c4c0d-ce10-44ef-a2e7-a7562b103e46-config\") pod \"console-operator-85b454874b-ldv8m\" (UID: \"b60c4c0d-ce10-44ef-a2e7-a7562b103e46\") " pod="openshift-console-operator/console-operator-85b454874b-ldv8m"
Sep 09 20:12:01.958065 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.958039    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b60c4c0d-ce10-44ef-a2e7-a7562b103e46-trusted-ca\") pod \"console-operator-85b454874b-ldv8m\" (UID: \"b60c4c0d-ce10-44ef-a2e7-a7562b103e46\") " pod="openshift-console-operator/console-operator-85b454874b-ldv8m"
Sep 09 20:12:01.959554 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.959529    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b60c4c0d-ce10-44ef-a2e7-a7562b103e46-serving-cert\") pod \"console-operator-85b454874b-ldv8m\" (UID: \"b60c4c0d-ce10-44ef-a2e7-a7562b103e46\") " pod="openshift-console-operator/console-operator-85b454874b-ldv8m"
Sep 09 20:12:01.965258 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:01.965228    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsswv\" (UniqueName: \"kubernetes.io/projected/b60c4c0d-ce10-44ef-a2e7-a7562b103e46-kube-api-access-gsswv\") pod \"console-operator-85b454874b-ldv8m\" (UID: \"b60c4c0d-ce10-44ef-a2e7-a7562b103e46\") " pod="openshift-console-operator/console-operator-85b454874b-ldv8m"
Sep 09 20:12:02.044963 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.044878    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2fqt"
Sep 09 20:12:02.057651 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.057622    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71bd9823-f654-4f3d-97b4-9c186b4e4028-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-8vcxj\" (UID: \"71bd9823-f654-4f3d-97b4-9c186b4e4028\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8vcxj"
Sep 09 20:12:02.057762 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.057661    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/113a1f91-274e-4844-9598-21a40cfdcdb2-installation-pull-secrets\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:02.057762 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.057681    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccw8c\" (UniqueName: \"kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-kube-api-access-ccw8c\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:02.057762 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.057698    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71bd9823-f654-4f3d-97b4-9c186b4e4028-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-8vcxj\" (UID: \"71bd9823-f654-4f3d-97b4-9c186b4e4028\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8vcxj"
Sep 09 20:12:02.057762 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.057721    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/113a1f91-274e-4844-9598-21a40cfdcdb2-image-registry-private-configuration\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:02.057762 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.057740    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/113a1f91-274e-4844-9598-21a40cfdcdb2-trusted-ca\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:02.057762 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.057756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-bound-sa-token\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:02.057976 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.057810    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-certificates\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:02.057976 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.057828    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-tls\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:02.057976 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.057873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmsxd\" (UniqueName: \"kubernetes.io/projected/71bd9823-f654-4f3d-97b4-9c186b4e4028-kube-api-access-zmsxd\") pod \"kube-storage-version-migrator-operator-8d7dd4675-8vcxj\" (UID: \"71bd9823-f654-4f3d-97b4-9c186b4e4028\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8vcxj"
Sep 09 20:12:02.057976 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.057891    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/113a1f91-274e-4844-9598-21a40cfdcdb2-ca-trust-extracted\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:02.058483 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.058232    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/113a1f91-274e-4844-9598-21a40cfdcdb2-ca-trust-extracted\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:02.058483 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:02.058341    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:12:02.058483 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:02.058357    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-974b4ddbd-dpnx9: secret "image-registry-tls" not found
Sep 09 20:12:02.058483 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:02.058422    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-tls podName:113a1f91-274e-4844-9598-21a40cfdcdb2 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:02.558400606 +0000 UTC m=+64.563577593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-tls") pod "image-registry-974b4ddbd-dpnx9" (UID: "113a1f91-274e-4844-9598-21a40cfdcdb2") : secret "image-registry-tls" not found
Sep 09 20:12:02.058758 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.058733    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71bd9823-f654-4f3d-97b4-9c186b4e4028-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-8vcxj\" (UID: \"71bd9823-f654-4f3d-97b4-9c186b4e4028\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8vcxj"
Sep 09 20:12:02.058968 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.058909    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-certificates\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:02.059087 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.059068    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/113a1f91-274e-4844-9598-21a40cfdcdb2-trusted-ca\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:02.060391 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.060368    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71bd9823-f654-4f3d-97b4-9c186b4e4028-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-8vcxj\" (UID: \"71bd9823-f654-4f3d-97b4-9c186b4e4028\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8vcxj"
Sep 09 20:12:02.060998 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.060976    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/113a1f91-274e-4844-9598-21a40cfdcdb2-installation-pull-secrets\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:02.061076 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.061058    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/113a1f91-274e-4844-9598-21a40cfdcdb2-image-registry-private-configuration\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:02.066150 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.066131    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-ldv8m"
Sep 09 20:12:02.067140 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.067119    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-bound-sa-token\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:02.067226 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.067171    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmsxd\" (UniqueName: \"kubernetes.io/projected/71bd9823-f654-4f3d-97b4-9c186b4e4028-kube-api-access-zmsxd\") pod \"kube-storage-version-migrator-operator-8d7dd4675-8vcxj\" (UID: \"71bd9823-f654-4f3d-97b4-9c186b4e4028\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8vcxj"
Sep 09 20:12:02.067808 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.067791    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccw8c\" (UniqueName: \"kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-kube-api-access-ccw8c\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:02.173419 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.173378    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8vcxj"
Sep 09 20:12:02.316150 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.316077    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2fqt"]
Sep 09 20:12:02.319848 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:12:02.319822    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4c0694_26b1_4a2a_a5fa_b134191927fc.slice/crio-2b2e4173ad09adc1290b8dbac82e6cd2bd1c1ff32bb64513261828a4ce7d4f16 WatchSource:0}: Error finding container 2b2e4173ad09adc1290b8dbac82e6cd2bd1c1ff32bb64513261828a4ce7d4f16: Status 404 returned error can't find the container with id 2b2e4173ad09adc1290b8dbac82e6cd2bd1c1ff32bb64513261828a4ce7d4f16
Sep 09 20:12:02.340607 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.340539    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-ldv8m"]
Sep 09 20:12:02.344944 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:12:02.344920    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb60c4c0d_ce10_44ef_a2e7_a7562b103e46.slice/crio-4eb391c2e58249a95c0385093337a209010923adf6f8c36469a975a9e4a80179 WatchSource:0}: Error finding container 4eb391c2e58249a95c0385093337a209010923adf6f8c36469a975a9e4a80179: Status 404 returned error can't find the container with id 4eb391c2e58249a95c0385093337a209010923adf6f8c36469a975a9e4a80179
Sep 09 20:12:02.360344 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.360324    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/43ddfbde-d8f2-4412-941e-98bab3b4856d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j4s66\" (UID: \"43ddfbde-d8f2-4412-941e-98bab3b4856d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j4s66"
Sep 09 20:12:02.360440 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:02.360427    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:12:02.360501 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:02.360493    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43ddfbde-d8f2-4412-941e-98bab3b4856d-samples-operator-tls podName:43ddfbde-d8f2-4412-941e-98bab3b4856d nodeName:}" failed. No retries permitted until 2026-09-09 20:12:03.360477268 +0000 UTC m=+65.365654254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/43ddfbde-d8f2-4412-941e-98bab3b4856d-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-j4s66" (UID: "43ddfbde-d8f2-4412-941e-98bab3b4856d") : secret "samples-operator-tls" not found
Sep 09 20:12:02.438687 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.438656    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8vcxj"]
Sep 09 20:12:02.441394 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:12:02.441372    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71bd9823_f654_4f3d_97b4_9c186b4e4028.slice/crio-1943836cd45e1f5911f8b5ed6322afae03900146628e07a25d6d6ee5a435073b WatchSource:0}: Error finding container 1943836cd45e1f5911f8b5ed6322afae03900146628e07a25d6d6ee5a435073b: Status 404 returned error can't find the container with id 1943836cd45e1f5911f8b5ed6322afae03900146628e07a25d6d6ee5a435073b
Sep 09 20:12:02.561787 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.561748    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-tls\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:02.561945 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:02.561902    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:12:02.561945 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:02.561925    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-974b4ddbd-dpnx9: secret "image-registry-tls" not found
Sep 09 20:12:02.562022 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:02.561981    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-tls podName:113a1f91-274e-4844-9598-21a40cfdcdb2 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:03.561964538 +0000 UTC m=+65.567141515 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-tls") pod "image-registry-974b4ddbd-dpnx9" (UID: "113a1f91-274e-4844-9598-21a40cfdcdb2") : secret "image-registry-tls" not found
Sep 09 20:12:02.962865 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.962827    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8vcxj" event={"ID":"71bd9823-f654-4f3d-97b4-9c186b4e4028","Type":"ContainerStarted","Data":"1943836cd45e1f5911f8b5ed6322afae03900146628e07a25d6d6ee5a435073b"}
Sep 09 20:12:02.966087 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.966023    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-ldv8m" event={"ID":"b60c4c0d-ce10-44ef-a2e7-a7562b103e46","Type":"ContainerStarted","Data":"4eb391c2e58249a95c0385093337a209010923adf6f8c36469a975a9e4a80179"}
Sep 09 20:12:02.967379 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:02.967328    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2fqt" event={"ID":"ae4c0694-26b1-4a2a-a5fa-b134191927fc","Type":"ContainerStarted","Data":"2b2e4173ad09adc1290b8dbac82e6cd2bd1c1ff32bb64513261828a4ce7d4f16"}
Sep 09 20:12:03.367708 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:03.367666    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/43ddfbde-d8f2-4412-941e-98bab3b4856d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j4s66\" (UID: \"43ddfbde-d8f2-4412-941e-98bab3b4856d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j4s66"
Sep 09 20:12:03.367945 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:03.367924    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:12:03.368020 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:03.368003    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43ddfbde-d8f2-4412-941e-98bab3b4856d-samples-operator-tls podName:43ddfbde-d8f2-4412-941e-98bab3b4856d nodeName:}" failed. No retries permitted until 2026-09-09 20:12:05.36797986 +0000 UTC m=+67.373156854 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/43ddfbde-d8f2-4412-941e-98bab3b4856d-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-j4s66" (UID: "43ddfbde-d8f2-4412-941e-98bab3b4856d") : secret "samples-operator-tls" not found
Sep 09 20:12:03.570086 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:03.570049    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-tls\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:03.570272 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:03.570220    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:12:03.570272 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:03.570242    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-974b4ddbd-dpnx9: secret "image-registry-tls" not found
Sep 09 20:12:03.570378 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:03.570309    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-tls podName:113a1f91-274e-4844-9598-21a40cfdcdb2 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:05.570288826 +0000 UTC m=+67.575465805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-tls") pod "image-registry-974b4ddbd-dpnx9" (UID: "113a1f91-274e-4844-9598-21a40cfdcdb2") : secret "image-registry-tls" not found
Sep 09 20:12:03.972098 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:03.972028    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2fqt" event={"ID":"ae4c0694-26b1-4a2a-a5fa-b134191927fc","Type":"ContainerStarted","Data":"f840cbb05698ff9ed90c16cfcacb35cb27242689ba723336fbf075de85175414"}
Sep 09 20:12:03.988163 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:03.988114    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2fqt" podStartSLOduration=1.888258494 podStartE2EDuration="2.988097642s" totalImagesPullingTime="1.099839148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:03.987098144 +0000 UTC m=+65.992275193" watchObservedRunningTime="2026-09-09 20:12:03.988097642 +0000 UTC m=+65.993274639"
Sep 09 20:12:04.176148 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:04.176098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs\") pod \"network-metrics-daemon-9vfht\" (UID: \"7e3d0490-5d57-42a1-858f-1e6bf8830f4c\") " pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:12:04.178759 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:04.178727    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:12:04.186805 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:04.186782    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 20:12:04.186930 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:04.186852    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs podName:7e3d0490-5d57-42a1-858f-1e6bf8830f4c nodeName:}" failed. No retries permitted until 2026-09-09 20:13:08.186832015 +0000 UTC m=+130.192009006 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs") pod "network-metrics-daemon-9vfht" (UID: "7e3d0490-5d57-42a1-858f-1e6bf8830f4c") : secret "metrics-daemon-secret" not found
Sep 09 20:12:04.277004 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:04.276943    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d3756a7-d67d-48ef-aa34-a792650e747d-original-pull-secret\") pod \"global-pull-secret-syncer-z8q9m\" (UID: \"5d3756a7-d67d-48ef-aa34-a792650e747d\") " pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:12:04.277004 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:04.277010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2w9j\" (UniqueName: \"kubernetes.io/projected/6b9953bc-b5e1-4b30-a926-c1a13d4677a7-kube-api-access-g2w9j\") pod \"network-check-target-zcrsj\" (UID: \"6b9953bc-b5e1-4b30-a926-c1a13d4677a7\") " pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:12:04.280074 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:04.280050    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:12:04.280190 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:04.280078    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:12:04.290253 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:04.290228    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:12:04.290408 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:04.290233    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d3756a7-d67d-48ef-aa34-a792650e747d-original-pull-secret\") pod \"global-pull-secret-syncer-z8q9m\" (UID: \"5d3756a7-d67d-48ef-aa34-a792650e747d\") " pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:12:04.300634 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:04.300587    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2w9j\" (UniqueName: \"kubernetes.io/projected/6b9953bc-b5e1-4b30-a926-c1a13d4677a7-kube-api-access-g2w9j\") pod \"network-check-target-zcrsj\" (UID: \"6b9953bc-b5e1-4b30-a926-c1a13d4677a7\") " pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:12:04.343510 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:04.343479    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wr6wr\""
Sep 09 20:12:04.350705 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:04.350679    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:12:04.544811 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:04.544721    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8q9m"
Sep 09 20:12:04.680837 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:04.680799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bff29a8-0416-4090-af52-b62924509cd0-metrics-tls\") pod \"dns-default-qd6j5\" (UID: \"2bff29a8-0416-4090-af52-b62924509cd0\") " pod="openshift-dns/dns-default-qd6j5"
Sep 09 20:12:04.681010 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:04.680967    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:12:04.681082 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:04.681044    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bff29a8-0416-4090-af52-b62924509cd0-metrics-tls podName:2bff29a8-0416-4090-af52-b62924509cd0 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:36.681022795 +0000 UTC m=+98.686199780 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2bff29a8-0416-4090-af52-b62924509cd0-metrics-tls") pod "dns-default-qd6j5" (UID: "2bff29a8-0416-4090-af52-b62924509cd0") : secret "dns-default-metrics-tls" not found
Sep 09 20:12:05.263667 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.263638    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z8q9m"]
Sep 09 20:12:05.266441 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:12:05.266413    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3756a7_d67d_48ef_aa34_a792650e747d.slice/crio-ea2d951651d7d3710a9e6558d4be338c88e2e30f661235d78b76863a8a34a998 WatchSource:0}: Error finding container ea2d951651d7d3710a9e6558d4be338c88e2e30f661235d78b76863a8a34a998: Status 404 returned error can't find the container with id ea2d951651d7d3710a9e6558d4be338c88e2e30f661235d78b76863a8a34a998
Sep 09 20:12:05.266595 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.266449    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zcrsj"]
Sep 09 20:12:05.270468 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:12:05.270433    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b9953bc_b5e1_4b30_a926_c1a13d4677a7.slice/crio-b337dfc62d6134e4f4e3c382ee572b68b3a66acec4a8500f3e48ad0fcc1f7eb7 WatchSource:0}: Error finding container b337dfc62d6134e4f4e3c382ee572b68b3a66acec4a8500f3e48ad0fcc1f7eb7: Status 404 returned error can't find the container with id b337dfc62d6134e4f4e3c382ee572b68b3a66acec4a8500f3e48ad0fcc1f7eb7
Sep 09 20:12:05.386177 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.386145    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/43ddfbde-d8f2-4412-941e-98bab3b4856d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j4s66\" (UID: \"43ddfbde-d8f2-4412-941e-98bab3b4856d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j4s66"
Sep 09 20:12:05.386295 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:05.386253    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:12:05.386337 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:05.386300    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43ddfbde-d8f2-4412-941e-98bab3b4856d-samples-operator-tls podName:43ddfbde-d8f2-4412-941e-98bab3b4856d nodeName:}" failed. No retries permitted until 2026-09-09 20:12:09.38628799 +0000 UTC m=+71.391464976 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/43ddfbde-d8f2-4412-941e-98bab3b4856d-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-j4s66" (UID: "43ddfbde-d8f2-4412-941e-98bab3b4856d") : secret "samples-operator-tls" not found
Sep 09 20:12:05.588067 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.587979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-tls\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:05.588207 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:05.588149    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:12:05.588207 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:05.588172    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-974b4ddbd-dpnx9: secret "image-registry-tls" not found
Sep 09 20:12:05.588317 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:05.588240    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-tls podName:113a1f91-274e-4844-9598-21a40cfdcdb2 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:09.588221002 +0000 UTC m=+71.593397980 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-tls") pod "image-registry-974b4ddbd-dpnx9" (UID: "113a1f91-274e-4844-9598-21a40cfdcdb2") : secret "image-registry-tls" not found
Sep 09 20:12:05.689700 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.689666    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-999gg"]
Sep 09 20:12:05.712075 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.712052    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-999gg"]
Sep 09 20:12:05.712198 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.712161    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:05.714560 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.714543    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 20:12:05.714713 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.714690    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j97b5\""
Sep 09 20:12:05.714771 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.714706    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 20:12:05.715675 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.715659    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 20:12:05.715773 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.715696    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 20:12:05.789944 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.789908    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f211a633-063b-4450-bdb9-26288976f42f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:05.790070 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.790050    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f211a633-063b-4450-bdb9-26288976f42f-crio-socket\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:05.790114 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.790085    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6569c\" (UniqueName: \"kubernetes.io/projected/f211a633-063b-4450-bdb9-26288976f42f-kube-api-access-6569c\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:05.790114 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.790107    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f211a633-063b-4450-bdb9-26288976f42f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:05.790197 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.790172    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f211a633-063b-4450-bdb9-26288976f42f-data-volume\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:05.891037 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.890962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f211a633-063b-4450-bdb9-26288976f42f-crio-socket\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:05.891037 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.891007    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6569c\" (UniqueName: \"kubernetes.io/projected/f211a633-063b-4450-bdb9-26288976f42f-kube-api-access-6569c\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:05.891235 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.891040    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f211a633-063b-4450-bdb9-26288976f42f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:05.891235 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.891079    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f211a633-063b-4450-bdb9-26288976f42f-data-volume\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:05.891235 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.891112    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f211a633-063b-4450-bdb9-26288976f42f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:05.891235 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:05.891186    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:12:05.891425 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:05.891245    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f211a633-063b-4450-bdb9-26288976f42f-kube-rbac-proxy-cm podName:f211a633-063b-4450-bdb9-26288976f42f nodeName:}" failed. No retries permitted until 2026-09-09 20:12:06.391228096 +0000 UTC m=+68.396405085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f211a633-063b-4450-bdb9-26288976f42f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-999gg" (UID: "f211a633-063b-4450-bdb9-26288976f42f") : configmap "kube-rbac-proxy" not found
Sep 09 20:12:05.891425 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:05.891377    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:12:05.891425 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.891079    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f211a633-063b-4450-bdb9-26288976f42f-crio-socket\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:05.891425 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:05.891421    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f211a633-063b-4450-bdb9-26288976f42f-insights-runtime-extractor-tls podName:f211a633-063b-4450-bdb9-26288976f42f nodeName:}" failed. No retries permitted until 2026-09-09 20:12:06.391406146 +0000 UTC m=+68.396583136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f211a633-063b-4450-bdb9-26288976f42f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-999gg" (UID: "f211a633-063b-4450-bdb9-26288976f42f") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:12:05.891763 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.891741    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f211a633-063b-4450-bdb9-26288976f42f-data-volume\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:05.899476 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.899436    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6569c\" (UniqueName: \"kubernetes.io/projected/f211a633-063b-4450-bdb9-26288976f42f-kube-api-access-6569c\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:05.978924 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.978880    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8vcxj" event={"ID":"71bd9823-f654-4f3d-97b4-9c186b4e4028","Type":"ContainerStarted","Data":"90ccf4e4bcb95eb0982031d21c6332f1b1d22edf9f15e880edb5ebcec40b74c0"}
Sep 09 20:12:05.981343 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.981312    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/0.log"
Sep 09 20:12:05.981484 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.981357    2581 generic.go:356] "Generic (PLEG): container finished" podID="b60c4c0d-ce10-44ef-a2e7-a7562b103e46" containerID="cdcd8c1a3105168fa731b5e1b784102c55afec9a3b83ae1849381510c8e639fd" exitCode=255
Sep 09 20:12:05.981484 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.981436    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-ldv8m" event={"ID":"b60c4c0d-ce10-44ef-a2e7-a7562b103e46","Type":"ContainerDied","Data":"cdcd8c1a3105168fa731b5e1b784102c55afec9a3b83ae1849381510c8e639fd"}
Sep 09 20:12:05.981674 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.981655    2581 scope.go:117] "RemoveContainer" containerID="cdcd8c1a3105168fa731b5e1b784102c55afec9a3b83ae1849381510c8e639fd"
Sep 09 20:12:05.982907 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.982878    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zcrsj" event={"ID":"6b9953bc-b5e1-4b30-a926-c1a13d4677a7","Type":"ContainerStarted","Data":"b337dfc62d6134e4f4e3c382ee572b68b3a66acec4a8500f3e48ad0fcc1f7eb7"}
Sep 09 20:12:05.984403 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.984374    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z8q9m" event={"ID":"5d3756a7-d67d-48ef-aa34-a792650e747d","Type":"ContainerStarted","Data":"ea2d951651d7d3710a9e6558d4be338c88e2e30f661235d78b76863a8a34a998"}
Sep 09 20:12:05.995149 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:05.995095    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8vcxj" podStartSLOduration=2.435677922 podStartE2EDuration="4.995066229s" totalImagesPullingTime="2.559388307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:05.994085206 +0000 UTC m=+67.999262198" watchObservedRunningTime="2026-09-09 20:12:05.995066229 +0000 UTC m=+68.000243216"
Sep 09 20:12:06.395020 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:06.394974    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f211a633-063b-4450-bdb9-26288976f42f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:06.395447 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:06.395052    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f211a633-063b-4450-bdb9-26288976f42f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:06.395447 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:06.395139    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:12:06.395447 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:06.395168    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:12:06.395447 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:06.395206    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f211a633-063b-4450-bdb9-26288976f42f-kube-rbac-proxy-cm podName:f211a633-063b-4450-bdb9-26288976f42f nodeName:}" failed. No retries permitted until 2026-09-09 20:12:07.395186537 +0000 UTC m=+69.400363523 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f211a633-063b-4450-bdb9-26288976f42f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-999gg" (UID: "f211a633-063b-4450-bdb9-26288976f42f") : configmap "kube-rbac-proxy" not found
Sep 09 20:12:06.395447 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:06.395225    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f211a633-063b-4450-bdb9-26288976f42f-insights-runtime-extractor-tls podName:f211a633-063b-4450-bdb9-26288976f42f nodeName:}" failed. No retries permitted until 2026-09-09 20:12:07.395215148 +0000 UTC m=+69.400392131 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f211a633-063b-4450-bdb9-26288976f42f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-999gg" (UID: "f211a633-063b-4450-bdb9-26288976f42f") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:12:06.990310 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:06.990278    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/1.log"
Sep 09 20:12:06.991030 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:06.990878    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/0.log"
Sep 09 20:12:06.991030 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:06.990928    2581 generic.go:356] "Generic (PLEG): container finished" podID="b60c4c0d-ce10-44ef-a2e7-a7562b103e46" containerID="22744a4ffaa62f0feba0ddee48e01038cac9ca95a295756c812755edd603d1af" exitCode=255
Sep 09 20:12:06.991214 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:06.991025    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-ldv8m" event={"ID":"b60c4c0d-ce10-44ef-a2e7-a7562b103e46","Type":"ContainerDied","Data":"22744a4ffaa62f0feba0ddee48e01038cac9ca95a295756c812755edd603d1af"}
Sep 09 20:12:06.991214 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:06.991105    2581 scope.go:117] "RemoveContainer" containerID="cdcd8c1a3105168fa731b5e1b784102c55afec9a3b83ae1849381510c8e639fd"
Sep 09 20:12:06.991419 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:06.991404    2581 scope.go:117] "RemoveContainer" containerID="22744a4ffaa62f0feba0ddee48e01038cac9ca95a295756c812755edd603d1af"
Sep 09 20:12:06.991713 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:06.991689    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-ldv8m_openshift-console-operator(b60c4c0d-ce10-44ef-a2e7-a7562b103e46)\"" pod="openshift-console-operator/console-operator-85b454874b-ldv8m" podUID="b60c4c0d-ce10-44ef-a2e7-a7562b103e46"
Sep 09 20:12:07.405365 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:07.405285    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f211a633-063b-4450-bdb9-26288976f42f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:07.405365 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:07.405353    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f211a633-063b-4450-bdb9-26288976f42f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:07.405790 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:07.405450    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:12:07.405790 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:07.405491    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:12:07.405790 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:07.405540    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f211a633-063b-4450-bdb9-26288976f42f-kube-rbac-proxy-cm podName:f211a633-063b-4450-bdb9-26288976f42f nodeName:}" failed. No retries permitted until 2026-09-09 20:12:09.405522733 +0000 UTC m=+71.410699717 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f211a633-063b-4450-bdb9-26288976f42f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-999gg" (UID: "f211a633-063b-4450-bdb9-26288976f42f") : configmap "kube-rbac-proxy" not found
Sep 09 20:12:07.405790 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:07.405558    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f211a633-063b-4450-bdb9-26288976f42f-insights-runtime-extractor-tls podName:f211a633-063b-4450-bdb9-26288976f42f nodeName:}" failed. No retries permitted until 2026-09-09 20:12:09.405548622 +0000 UTC m=+71.410725603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f211a633-063b-4450-bdb9-26288976f42f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-999gg" (UID: "f211a633-063b-4450-bdb9-26288976f42f") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:12:07.645976 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:07.645948    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l7rwd_297425e2-5552-4654-b41b-336d88d5a56c/dns-node-resolver/0.log"
Sep 09 20:12:07.995472 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:07.995431    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/1.log"
Sep 09 20:12:07.995821 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:07.995804    2581 scope.go:117] "RemoveContainer" containerID="22744a4ffaa62f0feba0ddee48e01038cac9ca95a295756c812755edd603d1af"
Sep 09 20:12:07.995993 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:07.995973    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-ldv8m_openshift-console-operator(b60c4c0d-ce10-44ef-a2e7-a7562b103e46)\"" pod="openshift-console-operator/console-operator-85b454874b-ldv8m" podUID="b60c4c0d-ce10-44ef-a2e7-a7562b103e46"
Sep 09 20:12:08.445800 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:08.445718    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qpmd2_610a91de-a1e2-445d-8f72-48aba0d8a2bf/node-ca/0.log"
Sep 09 20:12:09.245646 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:09.245619    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4prkf_6d63ce9d-307d-4a8f-bcf1-9f53a3d3c956/serve-healthcheck-canary/0.log"
Sep 09 20:12:09.423531 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:09.423500    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f211a633-063b-4450-bdb9-26288976f42f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:09.423692 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:09.423588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/43ddfbde-d8f2-4412-941e-98bab3b4856d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j4s66\" (UID: \"43ddfbde-d8f2-4412-941e-98bab3b4856d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j4s66"
Sep 09 20:12:09.423692 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:09.423620    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f211a633-063b-4450-bdb9-26288976f42f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:09.423692 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:09.423648    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:12:09.423852 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:09.423739    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f211a633-063b-4450-bdb9-26288976f42f-kube-rbac-proxy-cm podName:f211a633-063b-4450-bdb9-26288976f42f nodeName:}" failed. No retries permitted until 2026-09-09 20:12:13.423713231 +0000 UTC m=+75.428890220 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f211a633-063b-4450-bdb9-26288976f42f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-999gg" (UID: "f211a633-063b-4450-bdb9-26288976f42f") : configmap "kube-rbac-proxy" not found
Sep 09 20:12:09.423852 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:09.423746    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:12:09.423852 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:09.423807    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43ddfbde-d8f2-4412-941e-98bab3b4856d-samples-operator-tls podName:43ddfbde-d8f2-4412-941e-98bab3b4856d nodeName:}" failed. No retries permitted until 2026-09-09 20:12:17.423789617 +0000 UTC m=+79.428966608 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/43ddfbde-d8f2-4412-941e-98bab3b4856d-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-j4s66" (UID: "43ddfbde-d8f2-4412-941e-98bab3b4856d") : secret "samples-operator-tls" not found
Sep 09 20:12:09.423852 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:09.423740    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:12:09.423990 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:09.423883    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f211a633-063b-4450-bdb9-26288976f42f-insights-runtime-extractor-tls podName:f211a633-063b-4450-bdb9-26288976f42f nodeName:}" failed. No retries permitted until 2026-09-09 20:12:13.42386745 +0000 UTC m=+75.429044449 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f211a633-063b-4450-bdb9-26288976f42f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-999gg" (UID: "f211a633-063b-4450-bdb9-26288976f42f") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:12:09.625252 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:09.625175    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-tls\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:09.625677 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:09.625304    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:12:09.625677 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:09.625324    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-974b4ddbd-dpnx9: secret "image-registry-tls" not found
Sep 09 20:12:09.625677 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:09.625391    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-tls podName:113a1f91-274e-4844-9598-21a40cfdcdb2 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:17.625374058 +0000 UTC m=+79.630551046 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-tls") pod "image-registry-974b4ddbd-dpnx9" (UID: "113a1f91-274e-4844-9598-21a40cfdcdb2") : secret "image-registry-tls" not found
Sep 09 20:12:11.006556 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:11.006508    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zcrsj" event={"ID":"6b9953bc-b5e1-4b30-a926-c1a13d4677a7","Type":"ContainerStarted","Data":"ce393541303b5c65f9bbd0880f2c6ba2377e5183283386fce9d8090a7152ed98"}
Sep 09 20:12:11.006988 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:11.006678    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:12:11.007933 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:11.007915    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z8q9m" event={"ID":"5d3756a7-d67d-48ef-aa34-a792650e747d","Type":"ContainerStarted","Data":"4ac90648fbfabf7b1fd20a60b221ea3035b75085af817a7519501f6d3d5295ad"}
Sep 09 20:12:11.023449 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:11.023408    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zcrsj" podStartSLOduration=67.942894908 podStartE2EDuration="1m13.023396595s" totalImagesPullingTime="5.080501687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:10:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:11.021611645 +0000 UTC m=+73.026788641" watchObservedRunningTime="2026-09-09 20:12:11.023396595 +0000 UTC m=+73.028573580"
Sep 09 20:12:11.038524 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:11.038483    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-z8q9m" podStartSLOduration=66.953315617 podStartE2EDuration="1m12.038447838s" totalImagesPullingTime="5.085132221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:11.037554911 +0000 UTC m=+73.042731906" watchObservedRunningTime="2026-09-09 20:12:11.038447838 +0000 UTC m=+73.043624830"
Sep 09 20:12:12.066902 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:12.066872    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-ldv8m"
Sep 09 20:12:12.067265 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:12.066922    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-ldv8m"
Sep 09 20:12:12.067265 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:12.067217    2581 scope.go:117] "RemoveContainer" containerID="22744a4ffaa62f0feba0ddee48e01038cac9ca95a295756c812755edd603d1af"
Sep 09 20:12:12.067363 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:12.067345    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-ldv8m_openshift-console-operator(b60c4c0d-ce10-44ef-a2e7-a7562b103e46)\"" pod="openshift-console-operator/console-operator-85b454874b-ldv8m" podUID="b60c4c0d-ce10-44ef-a2e7-a7562b103e46"
Sep 09 20:12:13.458009 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:13.457978    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f211a633-063b-4450-bdb9-26288976f42f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:13.458392 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:13.458028    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f211a633-063b-4450-bdb9-26288976f42f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:13.458392 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:13.458136    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:12:13.458392 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:13.458196    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f211a633-063b-4450-bdb9-26288976f42f-insights-runtime-extractor-tls podName:f211a633-063b-4450-bdb9-26288976f42f nodeName:}" failed. No retries permitted until 2026-09-09 20:12:21.45818206 +0000 UTC m=+83.463359036 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f211a633-063b-4450-bdb9-26288976f42f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-999gg" (UID: "f211a633-063b-4450-bdb9-26288976f42f") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:12:13.458392 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:13.458136    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:12:13.458392 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:13.458266    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f211a633-063b-4450-bdb9-26288976f42f-kube-rbac-proxy-cm podName:f211a633-063b-4450-bdb9-26288976f42f nodeName:}" failed. No retries permitted until 2026-09-09 20:12:21.458253021 +0000 UTC m=+83.463430010 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f211a633-063b-4450-bdb9-26288976f42f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-999gg" (UID: "f211a633-063b-4450-bdb9-26288976f42f") : configmap "kube-rbac-proxy" not found
Sep 09 20:12:17.488666 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:17.488613    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/43ddfbde-d8f2-4412-941e-98bab3b4856d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j4s66\" (UID: \"43ddfbde-d8f2-4412-941e-98bab3b4856d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j4s66"
Sep 09 20:12:17.490968 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:17.490946    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/43ddfbde-d8f2-4412-941e-98bab3b4856d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-j4s66\" (UID: \"43ddfbde-d8f2-4412-941e-98bab3b4856d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j4s66"
Sep 09 20:12:17.651168 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:17.651127    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j4s66"
Sep 09 20:12:17.690838 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:17.690808    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-tls\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:17.693295 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:17.693266    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-tls\") pod \"image-registry-974b4ddbd-dpnx9\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") " pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:17.780532 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:17.780424    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:17.918662 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:17.918630    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j4s66"]
Sep 09 20:12:18.029973 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:18.029944    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j4s66" event={"ID":"43ddfbde-d8f2-4412-941e-98bab3b4856d","Type":"ContainerStarted","Data":"2329b1427c050267babce3fa91d7ccc44cc69149c351c16ed477a5bcbdba6c2c"}
Sep 09 20:12:18.052897 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:18.052848    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-974b4ddbd-dpnx9"]
Sep 09 20:12:18.055137 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:12:18.055117    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod113a1f91_274e_4844_9598_21a40cfdcdb2.slice/crio-1665700deef22d4dcf8db5854ff1cf213a0790ada101d4598b17e884e46a1320 WatchSource:0}: Error finding container 1665700deef22d4dcf8db5854ff1cf213a0790ada101d4598b17e884e46a1320: Status 404 returned error can't find the container with id 1665700deef22d4dcf8db5854ff1cf213a0790ada101d4598b17e884e46a1320
Sep 09 20:12:19.037470 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:19.037433    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9" event={"ID":"113a1f91-274e-4844-9598-21a40cfdcdb2","Type":"ContainerStarted","Data":"a3ba277c44a1a7119df923c338cba7d931203223677f38473a8c6d5ffc685d26"}
Sep 09 20:12:19.037934 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:19.037495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9" event={"ID":"113a1f91-274e-4844-9598-21a40cfdcdb2","Type":"ContainerStarted","Data":"1665700deef22d4dcf8db5854ff1cf213a0790ada101d4598b17e884e46a1320"}
Sep 09 20:12:19.037934 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:19.037570    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:19.059205 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:19.059160    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9" podStartSLOduration=18.059146655 podStartE2EDuration="18.059146655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:19.057529404 +0000 UTC m=+81.062706397" watchObservedRunningTime="2026-09-09 20:12:19.059146655 +0000 UTC m=+81.064323650"
Sep 09 20:12:20.045589 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:20.045550    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j4s66" event={"ID":"43ddfbde-d8f2-4412-941e-98bab3b4856d","Type":"ContainerStarted","Data":"79d0e6228ba2f1106907ba05eb64fb39fd2f8495420f9d3c046c67898a7482c2"}
Sep 09 20:12:20.046036 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:20.045641    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j4s66" event={"ID":"43ddfbde-d8f2-4412-941e-98bab3b4856d","Type":"ContainerStarted","Data":"e046da0f9241e7d35d78d16d10df8d0ca8f063fc2172b75f1041724218a82206"}
Sep 09 20:12:20.063718 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:20.063673    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-j4s66" podStartSLOduration=17.315632098000002 podStartE2EDuration="19.06365766s" totalImagesPullingTime="1.748025562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:20.06280703 +0000 UTC m=+82.067984025" watchObservedRunningTime="2026-09-09 20:12:20.06365766 +0000 UTC m=+82.068834654"
Sep 09 20:12:21.521719 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:21.521578    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f211a633-063b-4450-bdb9-26288976f42f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:21.521719 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:21.521677    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f211a633-063b-4450-bdb9-26288976f42f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:21.522212 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:21.521793    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:12:21.522212 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:21.521876    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f211a633-063b-4450-bdb9-26288976f42f-insights-runtime-extractor-tls podName:f211a633-063b-4450-bdb9-26288976f42f nodeName:}" failed. No retries permitted until 2026-09-09 20:12:37.521857392 +0000 UTC m=+99.527034369 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f211a633-063b-4450-bdb9-26288976f42f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-999gg" (UID: "f211a633-063b-4450-bdb9-26288976f42f") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:12:21.522212 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:21.522097    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f211a633-063b-4450-bdb9-26288976f42f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:24.629949 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:24.629915    2581 scope.go:117] "RemoveContainer" containerID="22744a4ffaa62f0feba0ddee48e01038cac9ca95a295756c812755edd603d1af"
Sep 09 20:12:25.062607 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:25.062581    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 20:12:25.062966 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:25.062952    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/1.log"
Sep 09 20:12:25.063025 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:25.062983    2581 generic.go:356] "Generic (PLEG): container finished" podID="b60c4c0d-ce10-44ef-a2e7-a7562b103e46" containerID="3ab081941ff1a804a3a9e44dc5342a3cd62ca07b41d04ae91dc4a527b573e2ac" exitCode=255
Sep 09 20:12:25.063069 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:25.063050    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-ldv8m" event={"ID":"b60c4c0d-ce10-44ef-a2e7-a7562b103e46","Type":"ContainerDied","Data":"3ab081941ff1a804a3a9e44dc5342a3cd62ca07b41d04ae91dc4a527b573e2ac"}
Sep 09 20:12:25.063108 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:25.063098    2581 scope.go:117] "RemoveContainer" containerID="22744a4ffaa62f0feba0ddee48e01038cac9ca95a295756c812755edd603d1af"
Sep 09 20:12:25.063421 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:25.063398    2581 scope.go:117] "RemoveContainer" containerID="3ab081941ff1a804a3a9e44dc5342a3cd62ca07b41d04ae91dc4a527b573e2ac"
Sep 09 20:12:25.063611 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:25.063592    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-ldv8m_openshift-console-operator(b60c4c0d-ce10-44ef-a2e7-a7562b103e46)\"" pod="openshift-console-operator/console-operator-85b454874b-ldv8m" podUID="b60c4c0d-ce10-44ef-a2e7-a7562b103e46"
Sep 09 20:12:26.069149 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:26.069124    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 20:12:29.404948 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.404914    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-974b4ddbd-dpnx9"]
Sep 09 20:12:29.405754 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.405733    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb"]
Sep 09 20:12:29.408795 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.408772    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb"
Sep 09 20:12:29.412315 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.412288    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 20:12:29.412505 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.412298    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 20:12:29.412505 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.412341    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 20:12:29.412692 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.412674    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 20:12:29.412749 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.412696    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 20:12:29.412822 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.412805    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 20:12:29.413636 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.413618    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 20:12:29.423307 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.423284    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb"]
Sep 09 20:12:29.576623 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.576595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8df9ff4a-c815-4823-adc6-0fbde64b2bc6-ca\") pod \"cluster-proxy-proxy-agent-76b74665fd-d52nb\" (UID: \"8df9ff4a-c815-4823-adc6-0fbde64b2bc6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb"
Sep 09 20:12:29.576803 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.576629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8df9ff4a-c815-4823-adc6-0fbde64b2bc6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76b74665fd-d52nb\" (UID: \"8df9ff4a-c815-4823-adc6-0fbde64b2bc6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb"
Sep 09 20:12:29.576803 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.576653    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8df9ff4a-c815-4823-adc6-0fbde64b2bc6-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76b74665fd-d52nb\" (UID: \"8df9ff4a-c815-4823-adc6-0fbde64b2bc6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb"
Sep 09 20:12:29.576803 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.576764    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg9nl\" (UniqueName: \"kubernetes.io/projected/8df9ff4a-c815-4823-adc6-0fbde64b2bc6-kube-api-access-mg9nl\") pod \"cluster-proxy-proxy-agent-76b74665fd-d52nb\" (UID: \"8df9ff4a-c815-4823-adc6-0fbde64b2bc6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb"
Sep 09 20:12:29.576953 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.576803    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8df9ff4a-c815-4823-adc6-0fbde64b2bc6-hub\") pod \"cluster-proxy-proxy-agent-76b74665fd-d52nb\" (UID: \"8df9ff4a-c815-4823-adc6-0fbde64b2bc6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb"
Sep 09 20:12:29.576953 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.576857    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8df9ff4a-c815-4823-adc6-0fbde64b2bc6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76b74665fd-d52nb\" (UID: \"8df9ff4a-c815-4823-adc6-0fbde64b2bc6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb"
Sep 09 20:12:29.677654 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.677576    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8df9ff4a-c815-4823-adc6-0fbde64b2bc6-ca\") pod \"cluster-proxy-proxy-agent-76b74665fd-d52nb\" (UID: \"8df9ff4a-c815-4823-adc6-0fbde64b2bc6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb"
Sep 09 20:12:29.677654 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.677604    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8df9ff4a-c815-4823-adc6-0fbde64b2bc6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76b74665fd-d52nb\" (UID: \"8df9ff4a-c815-4823-adc6-0fbde64b2bc6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb"
Sep 09 20:12:29.677654 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.677627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8df9ff4a-c815-4823-adc6-0fbde64b2bc6-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76b74665fd-d52nb\" (UID: \"8df9ff4a-c815-4823-adc6-0fbde64b2bc6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb"
Sep 09 20:12:29.677896 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.677671    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg9nl\" (UniqueName: \"kubernetes.io/projected/8df9ff4a-c815-4823-adc6-0fbde64b2bc6-kube-api-access-mg9nl\") pod \"cluster-proxy-proxy-agent-76b74665fd-d52nb\" (UID: \"8df9ff4a-c815-4823-adc6-0fbde64b2bc6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb"
Sep 09 20:12:29.677896 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.677802    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8df9ff4a-c815-4823-adc6-0fbde64b2bc6-hub\") pod \"cluster-proxy-proxy-agent-76b74665fd-d52nb\" (UID: \"8df9ff4a-c815-4823-adc6-0fbde64b2bc6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb"
Sep 09 20:12:29.677896 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.677840    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8df9ff4a-c815-4823-adc6-0fbde64b2bc6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76b74665fd-d52nb\" (UID: \"8df9ff4a-c815-4823-adc6-0fbde64b2bc6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb"
Sep 09 20:12:29.678334 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.678309    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8df9ff4a-c815-4823-adc6-0fbde64b2bc6-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76b74665fd-d52nb\" (UID: \"8df9ff4a-c815-4823-adc6-0fbde64b2bc6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb"
Sep 09 20:12:29.680179 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.680155    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8df9ff4a-c815-4823-adc6-0fbde64b2bc6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76b74665fd-d52nb\" (UID: \"8df9ff4a-c815-4823-adc6-0fbde64b2bc6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb"
Sep 09 20:12:29.680179 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.680175    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8df9ff4a-c815-4823-adc6-0fbde64b2bc6-hub\") pod \"cluster-proxy-proxy-agent-76b74665fd-d52nb\" (UID: \"8df9ff4a-c815-4823-adc6-0fbde64b2bc6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb"
Sep 09 20:12:29.680307 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.680247    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8df9ff4a-c815-4823-adc6-0fbde64b2bc6-ca\") pod \"cluster-proxy-proxy-agent-76b74665fd-d52nb\" (UID: \"8df9ff4a-c815-4823-adc6-0fbde64b2bc6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb"
Sep 09 20:12:29.680307 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.680271    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8df9ff4a-c815-4823-adc6-0fbde64b2bc6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76b74665fd-d52nb\" (UID: \"8df9ff4a-c815-4823-adc6-0fbde64b2bc6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb"
Sep 09 20:12:29.688655 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.688638    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg9nl\" (UniqueName: \"kubernetes.io/projected/8df9ff4a-c815-4823-adc6-0fbde64b2bc6-kube-api-access-mg9nl\") pod \"cluster-proxy-proxy-agent-76b74665fd-d52nb\" (UID: \"8df9ff4a-c815-4823-adc6-0fbde64b2bc6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb"
Sep 09 20:12:29.729154 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:29.729132    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb"
Sep 09 20:12:30.009855 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:30.009824    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb"]
Sep 09 20:12:30.012835 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:12:30.012809    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8df9ff4a_c815_4823_adc6_0fbde64b2bc6.slice/crio-00dccc8441f07fff610aadf3c1a6abb543f0101afe787ae1b4c2166c3aaef340 WatchSource:0}: Error finding container 00dccc8441f07fff610aadf3c1a6abb543f0101afe787ae1b4c2166c3aaef340: Status 404 returned error can't find the container with id 00dccc8441f07fff610aadf3c1a6abb543f0101afe787ae1b4c2166c3aaef340
Sep 09 20:12:30.082062 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:30.082028    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb" event={"ID":"8df9ff4a-c815-4823-adc6-0fbde64b2bc6","Type":"ContainerStarted","Data":"00dccc8441f07fff610aadf3c1a6abb543f0101afe787ae1b4c2166c3aaef340"}
Sep 09 20:12:32.067309 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:32.067274    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-ldv8m"
Sep 09 20:12:32.067765 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:32.067361    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-ldv8m"
Sep 09 20:12:32.067814 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:32.067765    2581 scope.go:117] "RemoveContainer" containerID="3ab081941ff1a804a3a9e44dc5342a3cd62ca07b41d04ae91dc4a527b573e2ac"
Sep 09 20:12:32.068008 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:32.067979    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-ldv8m_openshift-console-operator(b60c4c0d-ce10-44ef-a2e7-a7562b103e46)\"" pod="openshift-console-operator/console-operator-85b454874b-ldv8m" podUID="b60c4c0d-ce10-44ef-a2e7-a7562b103e46"
Sep 09 20:12:34.097233 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:34.097202    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb" event={"ID":"8df9ff4a-c815-4823-adc6-0fbde64b2bc6","Type":"ContainerStarted","Data":"6d8579e237b3d53cbf1a34b7fb908d1ba407e5387a2700294bf078f65d120725"}
Sep 09 20:12:34.097600 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:34.097246    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb" event={"ID":"8df9ff4a-c815-4823-adc6-0fbde64b2bc6","Type":"ContainerStarted","Data":"7e8a7efa0a42845d9b2a510c945922523078faa99ba54d746cb2663d07c00113"}
Sep 09 20:12:34.097600 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:34.097260    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb" event={"ID":"8df9ff4a-c815-4823-adc6-0fbde64b2bc6","Type":"ContainerStarted","Data":"7cfba329c722629daabc4a35262c07538ccfee0e8e46fbc0bd1c55ea53aa395c"}
Sep 09 20:12:34.117569 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:34.117510    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76b74665fd-d52nb" podStartSLOduration=1.229654346 podStartE2EDuration="5.117494524s" totalImagesPullingTime="3.887840178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:34.11573676 +0000 UTC m=+96.120913755" watchObservedRunningTime="2026-09-09 20:12:34.117494524 +0000 UTC m=+96.122671518"
Sep 09 20:12:36.742684 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:36.742642    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bff29a8-0416-4090-af52-b62924509cd0-metrics-tls\") pod \"dns-default-qd6j5\" (UID: \"2bff29a8-0416-4090-af52-b62924509cd0\") " pod="openshift-dns/dns-default-qd6j5"
Sep 09 20:12:36.744968 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:36.744943    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bff29a8-0416-4090-af52-b62924509cd0-metrics-tls\") pod \"dns-default-qd6j5\" (UID: \"2bff29a8-0416-4090-af52-b62924509cd0\") " pod="openshift-dns/dns-default-qd6j5"
Sep 09 20:12:37.029115 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:37.029020    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ftsch\""
Sep 09 20:12:37.037033 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:37.037007    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qd6j5"
Sep 09 20:12:37.305666 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:37.305510    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qd6j5"]
Sep 09 20:12:37.308119 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:12:37.308097    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bff29a8_0416_4090_af52_b62924509cd0.slice/crio-99fca76d0457d41ecfae1db453b9d78fffcf523711b19470786ecc0f1a6d278d WatchSource:0}: Error finding container 99fca76d0457d41ecfae1db453b9d78fffcf523711b19470786ecc0f1a6d278d: Status 404 returned error can't find the container with id 99fca76d0457d41ecfae1db453b9d78fffcf523711b19470786ecc0f1a6d278d
Sep 09 20:12:37.550032 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:37.549996    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f211a633-063b-4450-bdb9-26288976f42f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:37.552294 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:37.552269    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f211a633-063b-4450-bdb9-26288976f42f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-999gg\" (UID: \"f211a633-063b-4450-bdb9-26288976f42f\") " pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:37.821309 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:37.821272    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-999gg"
Sep 09 20:12:38.098863 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:38.098618    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-999gg"]
Sep 09 20:12:38.101558 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:12:38.101524    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf211a633_063b_4450_bdb9_26288976f42f.slice/crio-31704e5da3201f71e32503d2d9712b86226ead0a843649b58bd5b7e56e55e71c WatchSource:0}: Error finding container 31704e5da3201f71e32503d2d9712b86226ead0a843649b58bd5b7e56e55e71c: Status 404 returned error can't find the container with id 31704e5da3201f71e32503d2d9712b86226ead0a843649b58bd5b7e56e55e71c
Sep 09 20:12:38.111561 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:38.111533    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qd6j5" event={"ID":"2bff29a8-0416-4090-af52-b62924509cd0","Type":"ContainerStarted","Data":"99fca76d0457d41ecfae1db453b9d78fffcf523711b19470786ecc0f1a6d278d"}
Sep 09 20:12:38.112838 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:38.112813    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-999gg" event={"ID":"f211a633-063b-4450-bdb9-26288976f42f","Type":"ContainerStarted","Data":"31704e5da3201f71e32503d2d9712b86226ead0a843649b58bd5b7e56e55e71c"}
Sep 09 20:12:39.118015 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:39.117983    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qd6j5" event={"ID":"2bff29a8-0416-4090-af52-b62924509cd0","Type":"ContainerStarted","Data":"6f41c3151b583bec3264fe889d3a2eba60a0d8d633d26d050b691ed55d44eb15"}
Sep 09 20:12:39.118365 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:39.118041    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qd6j5" event={"ID":"2bff29a8-0416-4090-af52-b62924509cd0","Type":"ContainerStarted","Data":"5ad3dc20fb673d3744d4435c8dbec1159fe16362b7d28aee7f8d5c584a3343cc"}
Sep 09 20:12:39.118365 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:39.118093    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qd6j5"
Sep 09 20:12:39.119920 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:39.119898    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-999gg" event={"ID":"f211a633-063b-4450-bdb9-26288976f42f","Type":"ContainerStarted","Data":"174bfa386a5a34448ead990b420a6b634150553bc8e947701efd02bdbfd993d3"}
Sep 09 20:12:39.120003 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:39.119940    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-999gg" event={"ID":"f211a633-063b-4450-bdb9-26288976f42f","Type":"ContainerStarted","Data":"52b75e450d9426ea9bc3326c0af3ab40ae6108b37ebd8970cc94cd7f94afec5c"}
Sep 09 20:12:39.135760 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:39.135718    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qd6j5" podStartSLOduration=65.692471813 podStartE2EDuration="1m7.135706237s" totalImagesPullingTime="1.443234424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:39.134611424 +0000 UTC m=+101.139788420" watchObservedRunningTime="2026-09-09 20:12:39.135706237 +0000 UTC m=+101.140883232"
Sep 09 20:12:39.410505 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:39.410396    2581 patch_prober.go:28] interesting pod/image-registry-974b4ddbd-dpnx9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 20:12:39.410505 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:39.410453    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9" podUID="113a1f91-274e-4844-9598-21a40cfdcdb2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:12:41.129084 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:41.129051    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-999gg" event={"ID":"f211a633-063b-4450-bdb9-26288976f42f","Type":"ContainerStarted","Data":"437b40e3ea5c2a3b2a41d79d299c7e3ce7b60e98f9f13706fdba4f0a922d2991"}
Sep 09 20:12:41.146950 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:41.146902    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-999gg" podStartSLOduration=33.700635207 podStartE2EDuration="36.146887345s" totalImagesPullingTime="2.446252138s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:12:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:41.146427387 +0000 UTC m=+103.151604381" watchObservedRunningTime="2026-09-09 20:12:41.146887345 +0000 UTC m=+103.152064337"
Sep 09 20:12:42.013586 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:42.013548    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zcrsj"
Sep 09 20:12:47.178611 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.178584    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4znjx"]
Sep 09 20:12:47.181034 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.181014    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.183353 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.183324    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 20:12:47.183561 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.183540    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9nvqc\""
Sep 09 20:12:47.183691 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.183675    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 20:12:47.183856 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.183730    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 20:12:47.184385 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.184369    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 20:12:47.184477 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.184445    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 20:12:47.184545 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.184446    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 20:12:47.228449 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.228422    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2e553183-00ef-479b-a711-f43feb043bc6-root\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.228605 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.228449    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2e553183-00ef-479b-a711-f43feb043bc6-node-exporter-wtmp\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.228605 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.228493    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2e553183-00ef-479b-a711-f43feb043bc6-node-exporter-textfile\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.228605 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.228509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htf28\" (UniqueName: \"kubernetes.io/projected/2e553183-00ef-479b-a711-f43feb043bc6-kube-api-access-htf28\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.228727 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.228603    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2e553183-00ef-479b-a711-f43feb043bc6-node-exporter-accelerators-collector-config\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.228727 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.228659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2e553183-00ef-479b-a711-f43feb043bc6-node-exporter-tls\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.228727 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.228704    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e553183-00ef-479b-a711-f43feb043bc6-metrics-client-ca\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.228727 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.228723    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e553183-00ef-479b-a711-f43feb043bc6-sys\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.228864 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.228737    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e553183-00ef-479b-a711-f43feb043bc6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.329934 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.329902    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2e553183-00ef-479b-a711-f43feb043bc6-node-exporter-tls\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.330118 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.329961    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e553183-00ef-479b-a711-f43feb043bc6-metrics-client-ca\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.330118 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.329990    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e553183-00ef-479b-a711-f43feb043bc6-sys\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.330118 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.330010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e553183-00ef-479b-a711-f43feb043bc6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.330118 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.330043    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2e553183-00ef-479b-a711-f43feb043bc6-root\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.330118 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:47.330060    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 20:12:47.330118 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.330106    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2e553183-00ef-479b-a711-f43feb043bc6-root\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.330389 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.330116    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e553183-00ef-479b-a711-f43feb043bc6-sys\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.330389 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:47.330123    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e553183-00ef-479b-a711-f43feb043bc6-node-exporter-tls podName:2e553183-00ef-479b-a711-f43feb043bc6 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:47.830106938 +0000 UTC m=+109.835283916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2e553183-00ef-479b-a711-f43feb043bc6-node-exporter-tls") pod "node-exporter-4znjx" (UID: "2e553183-00ef-479b-a711-f43feb043bc6") : secret "node-exporter-tls" not found
Sep 09 20:12:47.330389 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.330064    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2e553183-00ef-479b-a711-f43feb043bc6-node-exporter-wtmp\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.330389 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.330189    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2e553183-00ef-479b-a711-f43feb043bc6-node-exporter-textfile\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.330389 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.330212    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htf28\" (UniqueName: \"kubernetes.io/projected/2e553183-00ef-479b-a711-f43feb043bc6-kube-api-access-htf28\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.330389 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.330261    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2e553183-00ef-479b-a711-f43feb043bc6-node-exporter-accelerators-collector-config\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.330389 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.330271    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2e553183-00ef-479b-a711-f43feb043bc6-node-exporter-wtmp\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.330750 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.330558    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2e553183-00ef-479b-a711-f43feb043bc6-node-exporter-textfile\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.330750 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.330719    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e553183-00ef-479b-a711-f43feb043bc6-metrics-client-ca\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.330815 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.330766    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2e553183-00ef-479b-a711-f43feb043bc6-node-exporter-accelerators-collector-config\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.332266 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.332243    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e553183-00ef-479b-a711-f43feb043bc6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.338924 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.338902    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htf28\" (UniqueName: \"kubernetes.io/projected/2e553183-00ef-479b-a711-f43feb043bc6-kube-api-access-htf28\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.629718 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.629683    2581 scope.go:117] "RemoveContainer" containerID="3ab081941ff1a804a3a9e44dc5342a3cd62ca07b41d04ae91dc4a527b573e2ac"
Sep 09 20:12:47.833698 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.833661    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2e553183-00ef-479b-a711-f43feb043bc6-node-exporter-tls\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:47.835871 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:47.835841    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2e553183-00ef-479b-a711-f43feb043bc6-node-exporter-tls\") pod \"node-exporter-4znjx\" (UID: \"2e553183-00ef-479b-a711-f43feb043bc6\") " pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:48.090252 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:48.090218    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4znjx"
Sep 09 20:12:48.099504 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:12:48.099450    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e553183_00ef_479b_a711_f43feb043bc6.slice/crio-bda1e657966b40b2a7cd3490c864c58eae9187a4a118f329f84df296ad213c6c WatchSource:0}: Error finding container bda1e657966b40b2a7cd3490c864c58eae9187a4a118f329f84df296ad213c6c: Status 404 returned error can't find the container with id bda1e657966b40b2a7cd3490c864c58eae9187a4a118f329f84df296ad213c6c
Sep 09 20:12:48.153593 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:48.153570    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 20:12:48.153722 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:48.153670    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-ldv8m" event={"ID":"b60c4c0d-ce10-44ef-a2e7-a7562b103e46","Type":"ContainerStarted","Data":"8007615c93feec39344c4300cd7b009bbe7aeddf04b89bc105ba3c10601d73d5"}
Sep 09 20:12:48.153960 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:48.153944    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-ldv8m"
Sep 09 20:12:48.154976 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:48.154957    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4znjx" event={"ID":"2e553183-00ef-479b-a711-f43feb043bc6","Type":"ContainerStarted","Data":"bda1e657966b40b2a7cd3490c864c58eae9187a4a118f329f84df296ad213c6c"}
Sep 09 20:12:48.158816 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:48.158798    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-ldv8m"
Sep 09 20:12:48.204012 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:48.203960    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-ldv8m" podStartSLOduration=44.55366296 podStartE2EDuration="47.203940046s" totalImagesPullingTime="2.650277086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:48.174182331 +0000 UTC m=+110.179359340" watchObservedRunningTime="2026-09-09 20:12:48.203940046 +0000 UTC m=+110.209117034"
Sep 09 20:12:49.125776 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:49.125753    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qd6j5"
Sep 09 20:12:49.161586 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:49.161560    2581 generic.go:356] "Generic (PLEG): container finished" podID="2e553183-00ef-479b-a711-f43feb043bc6" containerID="ce9ffc890e3826cde10459608d9ae56721734a642e9f592bb5f5efa373e7fdb8" exitCode=0
Sep 09 20:12:49.161746 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:49.161668    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4znjx" event={"ID":"2e553183-00ef-479b-a711-f43feb043bc6","Type":"ContainerDied","Data":"ce9ffc890e3826cde10459608d9ae56721734a642e9f592bb5f5efa373e7fdb8"}
Sep 09 20:12:49.411180 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:49.411147    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:50.167617 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:50.167577    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4znjx" event={"ID":"2e553183-00ef-479b-a711-f43feb043bc6","Type":"ContainerStarted","Data":"f52799b062d603ddb141ac458c2848ff4fb7c3ce716252366de8400e5e346c04"}
Sep 09 20:12:50.167813 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:50.167643    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4znjx" event={"ID":"2e553183-00ef-479b-a711-f43feb043bc6","Type":"ContainerStarted","Data":"fdbe2f1650b467f0916528482a73f317da21b30505443c17a04d636c4978169c"}
Sep 09 20:12:50.190396 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:50.190350    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4znjx" podStartSLOduration=2.268379041 podStartE2EDuration="3.190334747s" totalImagesPullingTime="858.782133ms" totalInitContainerRuntime="63.173573ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:50.188526512 +0000 UTC m=+112.193703504" watchObservedRunningTime="2026-09-09 20:12:50.190334747 +0000 UTC m=+112.195511741"
Sep 09 20:12:54.425296 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.425232    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9" podUID="113a1f91-274e-4844-9598-21a40cfdcdb2" containerName="registry" containerID="cri-o://a3ba277c44a1a7119df923c338cba7d931203223677f38473a8c6d5ffc685d26" gracePeriod=30
Sep 09 20:12:54.656673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.656649    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:54.692215 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.692158    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/113a1f91-274e-4844-9598-21a40cfdcdb2-ca-trust-extracted\") pod \"113a1f91-274e-4844-9598-21a40cfdcdb2\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") "
Sep 09 20:12:54.692215 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.692188    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccw8c\" (UniqueName: \"kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-kube-api-access-ccw8c\") pod \"113a1f91-274e-4844-9598-21a40cfdcdb2\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") "
Sep 09 20:12:54.692403 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.692225    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/113a1f91-274e-4844-9598-21a40cfdcdb2-trusted-ca\") pod \"113a1f91-274e-4844-9598-21a40cfdcdb2\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") "
Sep 09 20:12:54.692403 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.692276    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-tls\") pod \"113a1f91-274e-4844-9598-21a40cfdcdb2\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") "
Sep 09 20:12:54.692403 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.692305    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-bound-sa-token\") pod \"113a1f91-274e-4844-9598-21a40cfdcdb2\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") "
Sep 09 20:12:54.692403 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.692347    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/113a1f91-274e-4844-9598-21a40cfdcdb2-image-registry-private-configuration\") pod \"113a1f91-274e-4844-9598-21a40cfdcdb2\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") "
Sep 09 20:12:54.692403 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.692387    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-certificates\") pod \"113a1f91-274e-4844-9598-21a40cfdcdb2\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") "
Sep 09 20:12:54.692679 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.692552    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/113a1f91-274e-4844-9598-21a40cfdcdb2-installation-pull-secrets\") pod \"113a1f91-274e-4844-9598-21a40cfdcdb2\" (UID: \"113a1f91-274e-4844-9598-21a40cfdcdb2\") "
Sep 09 20:12:54.692832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.692742    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/113a1f91-274e-4844-9598-21a40cfdcdb2-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "113a1f91-274e-4844-9598-21a40cfdcdb2" (UID: "113a1f91-274e-4844-9598-21a40cfdcdb2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:12:54.692832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.692799    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "113a1f91-274e-4844-9598-21a40cfdcdb2" (UID: "113a1f91-274e-4844-9598-21a40cfdcdb2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:12:54.693034 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.692879    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/113a1f91-274e-4844-9598-21a40cfdcdb2-trusted-ca\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:12:54.693034 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.692895    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-certificates\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:12:54.694893 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.694869    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "113a1f91-274e-4844-9598-21a40cfdcdb2" (UID: "113a1f91-274e-4844-9598-21a40cfdcdb2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:12:54.695006 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.694908    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "113a1f91-274e-4844-9598-21a40cfdcdb2" (UID: "113a1f91-274e-4844-9598-21a40cfdcdb2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:12:54.695058 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.695006    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-kube-api-access-ccw8c" (OuterVolumeSpecName: "kube-api-access-ccw8c") pod "113a1f91-274e-4844-9598-21a40cfdcdb2" (UID: "113a1f91-274e-4844-9598-21a40cfdcdb2"). InnerVolumeSpecName "kube-api-access-ccw8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:12:54.695058 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.695017    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/113a1f91-274e-4844-9598-21a40cfdcdb2-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "113a1f91-274e-4844-9598-21a40cfdcdb2" (UID: "113a1f91-274e-4844-9598-21a40cfdcdb2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:12:54.695175 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.695156    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/113a1f91-274e-4844-9598-21a40cfdcdb2-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "113a1f91-274e-4844-9598-21a40cfdcdb2" (UID: "113a1f91-274e-4844-9598-21a40cfdcdb2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:12:54.700710 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.700686    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113a1f91-274e-4844-9598-21a40cfdcdb2-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "113a1f91-274e-4844-9598-21a40cfdcdb2" (UID: "113a1f91-274e-4844-9598-21a40cfdcdb2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:12:54.793632 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.793598    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/113a1f91-274e-4844-9598-21a40cfdcdb2-installation-pull-secrets\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:12:54.793632 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.793623    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/113a1f91-274e-4844-9598-21a40cfdcdb2-ca-trust-extracted\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:12:54.793632 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.793635    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccw8c\" (UniqueName: \"kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-kube-api-access-ccw8c\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:12:54.793817 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.793653    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-registry-tls\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:12:54.793817 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.793665    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/113a1f91-274e-4844-9598-21a40cfdcdb2-bound-sa-token\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:12:54.793817 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:54.793676    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/113a1f91-274e-4844-9598-21a40cfdcdb2-image-registry-private-configuration\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:12:55.187496 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:55.187442    2581 generic.go:356] "Generic (PLEG): container finished" podID="113a1f91-274e-4844-9598-21a40cfdcdb2" containerID="a3ba277c44a1a7119df923c338cba7d931203223677f38473a8c6d5ffc685d26" exitCode=0
Sep 09 20:12:55.187687 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:55.187518    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9"
Sep 09 20:12:55.187687 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:55.187538    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9" event={"ID":"113a1f91-274e-4844-9598-21a40cfdcdb2","Type":"ContainerDied","Data":"a3ba277c44a1a7119df923c338cba7d931203223677f38473a8c6d5ffc685d26"}
Sep 09 20:12:55.187687 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:55.187626    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-974b4ddbd-dpnx9" event={"ID":"113a1f91-274e-4844-9598-21a40cfdcdb2","Type":"ContainerDied","Data":"1665700deef22d4dcf8db5854ff1cf213a0790ada101d4598b17e884e46a1320"}
Sep 09 20:12:55.187687 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:55.187657    2581 scope.go:117] "RemoveContainer" containerID="a3ba277c44a1a7119df923c338cba7d931203223677f38473a8c6d5ffc685d26"
Sep 09 20:12:55.196207 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:55.196188    2581 scope.go:117] "RemoveContainer" containerID="a3ba277c44a1a7119df923c338cba7d931203223677f38473a8c6d5ffc685d26"
Sep 09 20:12:55.196511 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:12:55.196487    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3ba277c44a1a7119df923c338cba7d931203223677f38473a8c6d5ffc685d26\": container with ID starting with a3ba277c44a1a7119df923c338cba7d931203223677f38473a8c6d5ffc685d26 not found: ID does not exist" containerID="a3ba277c44a1a7119df923c338cba7d931203223677f38473a8c6d5ffc685d26"
Sep 09 20:12:55.196584 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:55.196523    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3ba277c44a1a7119df923c338cba7d931203223677f38473a8c6d5ffc685d26"} err="failed to get container status \"a3ba277c44a1a7119df923c338cba7d931203223677f38473a8c6d5ffc685d26\": rpc error: code = NotFound desc = could not find container \"a3ba277c44a1a7119df923c338cba7d931203223677f38473a8c6d5ffc685d26\": container with ID starting with a3ba277c44a1a7119df923c338cba7d931203223677f38473a8c6d5ffc685d26 not found: ID does not exist"
Sep 09 20:12:55.212893 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:55.212872    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-974b4ddbd-dpnx9"]
Sep 09 20:12:55.220504 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:55.220484    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-974b4ddbd-dpnx9"]
Sep 09 20:12:56.633402 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:56.633368    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="113a1f91-274e-4844-9598-21a40cfdcdb2" path="/var/lib/kubelet/pods/113a1f91-274e-4844-9598-21a40cfdcdb2/volumes"
Sep 09 20:12:57.885055 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:57.885027    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85bff69fd5-9bj2w"]
Sep 09 20:12:57.885471 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:57.885259    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="113a1f91-274e-4844-9598-21a40cfdcdb2" containerName="registry"
Sep 09 20:12:57.885471 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:57.885269    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="113a1f91-274e-4844-9598-21a40cfdcdb2" containerName="registry"
Sep 09 20:12:57.885471 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:57.885321    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="113a1f91-274e-4844-9598-21a40cfdcdb2" containerName="registry"
Sep 09 20:12:57.889935 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:57.889915    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:12:57.892473 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:57.892444    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 20:12:57.893493 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:57.893471    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 20:12:57.893578 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:57.893529    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 20:12:57.893735 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:57.893707    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xjkqr\""
Sep 09 20:12:57.893735 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:57.893730    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 20:12:57.893842 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:57.893708    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 20:12:57.893842 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:57.893828    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 20:12:57.893919 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:57.893847    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 20:12:57.897251 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:57.897229    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85bff69fd5-9bj2w"]
Sep 09 20:12:58.013055 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:58.013020    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5kk8\" (UniqueName: \"kubernetes.io/projected/8f7a7ba9-516a-4631-a246-df1e310f86b5-kube-api-access-b5kk8\") pod \"console-85bff69fd5-9bj2w\" (UID: \"8f7a7ba9-516a-4631-a246-df1e310f86b5\") " pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:12:58.013212 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:58.013100    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7a7ba9-516a-4631-a246-df1e310f86b5-service-ca\") pod \"console-85bff69fd5-9bj2w\" (UID: \"8f7a7ba9-516a-4631-a246-df1e310f86b5\") " pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:12:58.013212 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:58.013131    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f7a7ba9-516a-4631-a246-df1e310f86b5-oauth-serving-cert\") pod \"console-85bff69fd5-9bj2w\" (UID: \"8f7a7ba9-516a-4631-a246-df1e310f86b5\") " pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:12:58.013212 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:58.013148    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f7a7ba9-516a-4631-a246-df1e310f86b5-console-config\") pod \"console-85bff69fd5-9bj2w\" (UID: \"8f7a7ba9-516a-4631-a246-df1e310f86b5\") " pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:12:58.013212 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:58.013176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f7a7ba9-516a-4631-a246-df1e310f86b5-console-oauth-config\") pod \"console-85bff69fd5-9bj2w\" (UID: \"8f7a7ba9-516a-4631-a246-df1e310f86b5\") " pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:12:58.013212 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:58.013205    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f7a7ba9-516a-4631-a246-df1e310f86b5-console-serving-cert\") pod \"console-85bff69fd5-9bj2w\" (UID: \"8f7a7ba9-516a-4631-a246-df1e310f86b5\") " pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:12:58.114341 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:58.114306    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f7a7ba9-516a-4631-a246-df1e310f86b5-console-oauth-config\") pod \"console-85bff69fd5-9bj2w\" (UID: \"8f7a7ba9-516a-4631-a246-df1e310f86b5\") " pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:12:58.114341 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:58.114347    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f7a7ba9-516a-4631-a246-df1e310f86b5-console-serving-cert\") pod \"console-85bff69fd5-9bj2w\" (UID: \"8f7a7ba9-516a-4631-a246-df1e310f86b5\") " pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:12:58.114546 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:58.114370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5kk8\" (UniqueName: \"kubernetes.io/projected/8f7a7ba9-516a-4631-a246-df1e310f86b5-kube-api-access-b5kk8\") pod \"console-85bff69fd5-9bj2w\" (UID: \"8f7a7ba9-516a-4631-a246-df1e310f86b5\") " pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:12:58.114546 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:58.114503    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7a7ba9-516a-4631-a246-df1e310f86b5-service-ca\") pod \"console-85bff69fd5-9bj2w\" (UID: \"8f7a7ba9-516a-4631-a246-df1e310f86b5\") " pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:12:58.114622 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:58.114557    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f7a7ba9-516a-4631-a246-df1e310f86b5-oauth-serving-cert\") pod \"console-85bff69fd5-9bj2w\" (UID: \"8f7a7ba9-516a-4631-a246-df1e310f86b5\") " pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:12:58.114622 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:58.114587    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f7a7ba9-516a-4631-a246-df1e310f86b5-console-config\") pod \"console-85bff69fd5-9bj2w\" (UID: \"8f7a7ba9-516a-4631-a246-df1e310f86b5\") " pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:12:58.115159 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:58.115137    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7a7ba9-516a-4631-a246-df1e310f86b5-service-ca\") pod \"console-85bff69fd5-9bj2w\" (UID: \"8f7a7ba9-516a-4631-a246-df1e310f86b5\") " pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:12:58.115159 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:58.115151    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f7a7ba9-516a-4631-a246-df1e310f86b5-console-config\") pod \"console-85bff69fd5-9bj2w\" (UID: \"8f7a7ba9-516a-4631-a246-df1e310f86b5\") " pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:12:58.115342 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:58.115259    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f7a7ba9-516a-4631-a246-df1e310f86b5-oauth-serving-cert\") pod \"console-85bff69fd5-9bj2w\" (UID: \"8f7a7ba9-516a-4631-a246-df1e310f86b5\") " pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:12:58.116826 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:58.116808    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f7a7ba9-516a-4631-a246-df1e310f86b5-console-oauth-config\") pod \"console-85bff69fd5-9bj2w\" (UID: \"8f7a7ba9-516a-4631-a246-df1e310f86b5\") " pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:12:58.116826 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:58.116820    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f7a7ba9-516a-4631-a246-df1e310f86b5-console-serving-cert\") pod \"console-85bff69fd5-9bj2w\" (UID: \"8f7a7ba9-516a-4631-a246-df1e310f86b5\") " pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:12:58.122707 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:58.122685    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5kk8\" (UniqueName: \"kubernetes.io/projected/8f7a7ba9-516a-4631-a246-df1e310f86b5-kube-api-access-b5kk8\") pod \"console-85bff69fd5-9bj2w\" (UID: \"8f7a7ba9-516a-4631-a246-df1e310f86b5\") " pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:12:58.199304 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:58.199233    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:12:58.468337 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:58.468315    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85bff69fd5-9bj2w"]
Sep 09 20:12:58.470229 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:12:58.470206    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7a7ba9_516a_4631_a246_df1e310f86b5.slice/crio-ad6e1a4490f930540f2092ee23145ee3b2e23a04a5887ea716bd7a21f30f4b62 WatchSource:0}: Error finding container ad6e1a4490f930540f2092ee23145ee3b2e23a04a5887ea716bd7a21f30f4b62: Status 404 returned error can't find the container with id ad6e1a4490f930540f2092ee23145ee3b2e23a04a5887ea716bd7a21f30f4b62
Sep 09 20:12:59.203454 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:12:59.203421    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85bff69fd5-9bj2w" event={"ID":"8f7a7ba9-516a-4631-a246-df1e310f86b5","Type":"ContainerStarted","Data":"ad6e1a4490f930540f2092ee23145ee3b2e23a04a5887ea716bd7a21f30f4b62"}
Sep 09 20:13:02.215577 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:02.215548    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85bff69fd5-9bj2w" event={"ID":"8f7a7ba9-516a-4631-a246-df1e310f86b5","Type":"ContainerStarted","Data":"2f660cb52c41106d0ab333eb476bf9473a399799c40f144deac67178554905e6"}
Sep 09 20:13:02.233486 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:02.233414    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85bff69fd5-9bj2w" podStartSLOduration=2.447445136 podStartE2EDuration="5.233396202s" totalImagesPullingTime="2.785951066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:13:02.232213294 +0000 UTC m=+124.237390289" watchObservedRunningTime="2026-09-09 20:13:02.233396202 +0000 UTC m=+124.238573197"
Sep 09 20:13:06.121347 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.121316    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66d5c49969-szhzv"]
Sep 09 20:13:06.128401 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.128382    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:06.136522 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.136501    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 20:13:06.136973 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.136950    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66d5c49969-szhzv"]
Sep 09 20:13:06.173541 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.173509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87569eba-fd45-4826-80b5-d0201e63129d-console-oauth-config\") pod \"console-66d5c49969-szhzv\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") " pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:06.173662 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.173603    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87569eba-fd45-4826-80b5-d0201e63129d-trusted-ca-bundle\") pod \"console-66d5c49969-szhzv\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") " pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:06.173662 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.173647    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87569eba-fd45-4826-80b5-d0201e63129d-console-serving-cert\") pod \"console-66d5c49969-szhzv\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") " pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:06.173782 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.173695    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87569eba-fd45-4826-80b5-d0201e63129d-oauth-serving-cert\") pod \"console-66d5c49969-szhzv\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") " pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:06.173782 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.173719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5dvt\" (UniqueName: \"kubernetes.io/projected/87569eba-fd45-4826-80b5-d0201e63129d-kube-api-access-x5dvt\") pod \"console-66d5c49969-szhzv\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") " pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:06.173782 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.173773    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87569eba-fd45-4826-80b5-d0201e63129d-console-config\") pod \"console-66d5c49969-szhzv\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") " pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:06.173907 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.173797    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87569eba-fd45-4826-80b5-d0201e63129d-service-ca\") pod \"console-66d5c49969-szhzv\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") " pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:06.274909 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.274877    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87569eba-fd45-4826-80b5-d0201e63129d-console-config\") pod \"console-66d5c49969-szhzv\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") " pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:06.275081 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.274912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87569eba-fd45-4826-80b5-d0201e63129d-service-ca\") pod \"console-66d5c49969-szhzv\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") " pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:06.275081 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.274939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87569eba-fd45-4826-80b5-d0201e63129d-console-oauth-config\") pod \"console-66d5c49969-szhzv\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") " pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:06.275081 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.274989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87569eba-fd45-4826-80b5-d0201e63129d-trusted-ca-bundle\") pod \"console-66d5c49969-szhzv\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") " pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:06.275081 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.275020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87569eba-fd45-4826-80b5-d0201e63129d-console-serving-cert\") pod \"console-66d5c49969-szhzv\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") " pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:06.275296 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.275109    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87569eba-fd45-4826-80b5-d0201e63129d-oauth-serving-cert\") pod \"console-66d5c49969-szhzv\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") " pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:06.275296 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.275146    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5dvt\" (UniqueName: \"kubernetes.io/projected/87569eba-fd45-4826-80b5-d0201e63129d-kube-api-access-x5dvt\") pod \"console-66d5c49969-szhzv\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") " pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:06.275751 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.275715    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87569eba-fd45-4826-80b5-d0201e63129d-console-config\") pod \"console-66d5c49969-szhzv\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") " pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:06.275919 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.275833    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87569eba-fd45-4826-80b5-d0201e63129d-service-ca\") pod \"console-66d5c49969-szhzv\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") " pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:06.275919 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.275859    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87569eba-fd45-4826-80b5-d0201e63129d-oauth-serving-cert\") pod \"console-66d5c49969-szhzv\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") " pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:06.276057 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.276039    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87569eba-fd45-4826-80b5-d0201e63129d-trusted-ca-bundle\") pod \"console-66d5c49969-szhzv\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") " pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:06.277910 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.277892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87569eba-fd45-4826-80b5-d0201e63129d-console-serving-cert\") pod \"console-66d5c49969-szhzv\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") " pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:06.278014 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.277996    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87569eba-fd45-4826-80b5-d0201e63129d-console-oauth-config\") pod \"console-66d5c49969-szhzv\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") " pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:06.283041 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.283021    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5dvt\" (UniqueName: \"kubernetes.io/projected/87569eba-fd45-4826-80b5-d0201e63129d-kube-api-access-x5dvt\") pod \"console-66d5c49969-szhzv\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") " pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:06.438293 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.438207    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:06.728204 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:06.728160    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66d5c49969-szhzv"]
Sep 09 20:13:06.730292 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:13:06.730261    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87569eba_fd45_4826_80b5_d0201e63129d.slice/crio-81190eb1dbfe6f1b380f93e9f66cc7a896eaaca9d5fa24d9076b81bf40fd0cf4 WatchSource:0}: Error finding container 81190eb1dbfe6f1b380f93e9f66cc7a896eaaca9d5fa24d9076b81bf40fd0cf4: Status 404 returned error can't find the container with id 81190eb1dbfe6f1b380f93e9f66cc7a896eaaca9d5fa24d9076b81bf40fd0cf4
Sep 09 20:13:07.233498 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:07.233444    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66d5c49969-szhzv" event={"ID":"87569eba-fd45-4826-80b5-d0201e63129d","Type":"ContainerStarted","Data":"e6d8c399841f3540f27032a4a4b88005ba987883e2ceff0f6e70c20fac238687"}
Sep 09 20:13:07.233859 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:07.233555    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66d5c49969-szhzv" event={"ID":"87569eba-fd45-4826-80b5-d0201e63129d","Type":"ContainerStarted","Data":"81190eb1dbfe6f1b380f93e9f66cc7a896eaaca9d5fa24d9076b81bf40fd0cf4"}
Sep 09 20:13:07.252400 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:07.252356    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66d5c49969-szhzv" podStartSLOduration=1.252342007 podStartE2EDuration="1.252342007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:13:07.251748684 +0000 UTC m=+129.256925680" watchObservedRunningTime="2026-09-09 20:13:07.252342007 +0000 UTC m=+129.257519002"
Sep 09 20:13:08.193066 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:08.193030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs\") pod \"network-metrics-daemon-9vfht\" (UID: \"7e3d0490-5d57-42a1-858f-1e6bf8830f4c\") " pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:13:08.195301 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:08.195278    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e3d0490-5d57-42a1-858f-1e6bf8830f4c-metrics-certs\") pod \"network-metrics-daemon-9vfht\" (UID: \"7e3d0490-5d57-42a1-858f-1e6bf8830f4c\") " pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:13:08.200237 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:08.200212    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:13:08.200357 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:08.200272    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:13:08.205187 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:08.205165    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:13:08.240612 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:08.240589    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:13:08.442767 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:08.442737    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4t9vt\""
Sep 09 20:13:08.451005 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:08.450939    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vfht"
Sep 09 20:13:08.719808 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:08.719723    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9vfht"]
Sep 09 20:13:08.722409 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:13:08.722382    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3d0490_5d57_42a1_858f_1e6bf8830f4c.slice/crio-4e16c7533014f01e99a2f5a9b253355b67f74f01a2aed1c28aee1e79762a68b6 WatchSource:0}: Error finding container 4e16c7533014f01e99a2f5a9b253355b67f74f01a2aed1c28aee1e79762a68b6: Status 404 returned error can't find the container with id 4e16c7533014f01e99a2f5a9b253355b67f74f01a2aed1c28aee1e79762a68b6
Sep 09 20:13:09.241537 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:09.241496    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9vfht" event={"ID":"7e3d0490-5d57-42a1-858f-1e6bf8830f4c","Type":"ContainerStarted","Data":"4e16c7533014f01e99a2f5a9b253355b67f74f01a2aed1c28aee1e79762a68b6"}
Sep 09 20:13:10.246473 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:10.246425    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9vfht" event={"ID":"7e3d0490-5d57-42a1-858f-1e6bf8830f4c","Type":"ContainerStarted","Data":"2cdf93f8e4476019e6cc6c6956d3732a6d7e237ee5f8060b289e335ada6b15d8"}
Sep 09 20:13:10.246856 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:10.246525    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9vfht" event={"ID":"7e3d0490-5d57-42a1-858f-1e6bf8830f4c","Type":"ContainerStarted","Data":"ac4e8b28935e7b4ec8683a19e2cbc2033e314e6d842b88ae952a677ecb2f9e2b"}
Sep 09 20:13:10.263907 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:10.263776    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9vfht" podStartSLOduration=130.391834273 podStartE2EDuration="2m11.263758778s" totalImagesPullingTime="871.924505ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:13:10.263156731 +0000 UTC m=+132.268333729" watchObservedRunningTime="2026-09-09 20:13:10.263758778 +0000 UTC m=+132.268935773"
Sep 09 20:13:12.255479 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:12.255428    2581 generic.go:356] "Generic (PLEG): container finished" podID="71bd9823-f654-4f3d-97b4-9c186b4e4028" containerID="90ccf4e4bcb95eb0982031d21c6332f1b1d22edf9f15e880edb5ebcec40b74c0" exitCode=0
Sep 09 20:13:12.255842 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:12.255485    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8vcxj" event={"ID":"71bd9823-f654-4f3d-97b4-9c186b4e4028","Type":"ContainerDied","Data":"90ccf4e4bcb95eb0982031d21c6332f1b1d22edf9f15e880edb5ebcec40b74c0"}
Sep 09 20:13:12.255842 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:12.255768    2581 scope.go:117] "RemoveContainer" containerID="90ccf4e4bcb95eb0982031d21c6332f1b1d22edf9f15e880edb5ebcec40b74c0"
Sep 09 20:13:13.260656 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:13.260623    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-8vcxj" event={"ID":"71bd9823-f654-4f3d-97b4-9c186b4e4028","Type":"ContainerStarted","Data":"32a724a7e3e79932ca884a9d662ff8f5b8539e6c23d584063440cc111abe0d6d"}
Sep 09 20:13:16.438736 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:16.438691    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:16.438736 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:16.438755    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:16.443452 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:16.443432    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:17.279167 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:17.279140    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:13:17.327139 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:17.327106    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85bff69fd5-9bj2w"]
Sep 09 20:13:42.352719 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:42.352667    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-85bff69fd5-9bj2w" podUID="8f7a7ba9-516a-4631-a246-df1e310f86b5" containerName="console" containerID="cri-o://2f660cb52c41106d0ab333eb476bf9473a399799c40f144deac67178554905e6" gracePeriod=15
Sep 09 20:13:42.591317 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:42.591295    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:13:42.750778 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:42.750752    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f7a7ba9-516a-4631-a246-df1e310f86b5-oauth-serving-cert\") pod \"8f7a7ba9-516a-4631-a246-df1e310f86b5\" (UID: \"8f7a7ba9-516a-4631-a246-df1e310f86b5\") "
Sep 09 20:13:42.750940 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:42.750784    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f7a7ba9-516a-4631-a246-df1e310f86b5-console-serving-cert\") pod \"8f7a7ba9-516a-4631-a246-df1e310f86b5\" (UID: \"8f7a7ba9-516a-4631-a246-df1e310f86b5\") "
Sep 09 20:13:42.750940 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:42.750804    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7a7ba9-516a-4631-a246-df1e310f86b5-service-ca\") pod \"8f7a7ba9-516a-4631-a246-df1e310f86b5\" (UID: \"8f7a7ba9-516a-4631-a246-df1e310f86b5\") "
Sep 09 20:13:42.750940 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:42.750822    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f7a7ba9-516a-4631-a246-df1e310f86b5-console-config\") pod \"8f7a7ba9-516a-4631-a246-df1e310f86b5\" (UID: \"8f7a7ba9-516a-4631-a246-df1e310f86b5\") "
Sep 09 20:13:42.750940 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:42.750845    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5kk8\" (UniqueName: \"kubernetes.io/projected/8f7a7ba9-516a-4631-a246-df1e310f86b5-kube-api-access-b5kk8\") pod \"8f7a7ba9-516a-4631-a246-df1e310f86b5\" (UID: \"8f7a7ba9-516a-4631-a246-df1e310f86b5\") "
Sep 09 20:13:42.750940 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:42.750886    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f7a7ba9-516a-4631-a246-df1e310f86b5-console-oauth-config\") pod \"8f7a7ba9-516a-4631-a246-df1e310f86b5\" (UID: \"8f7a7ba9-516a-4631-a246-df1e310f86b5\") "
Sep 09 20:13:42.751284 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:42.751250    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f7a7ba9-516a-4631-a246-df1e310f86b5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8f7a7ba9-516a-4631-a246-df1e310f86b5" (UID: "8f7a7ba9-516a-4631-a246-df1e310f86b5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:13:42.751384 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:42.751253    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f7a7ba9-516a-4631-a246-df1e310f86b5-console-config" (OuterVolumeSpecName: "console-config") pod "8f7a7ba9-516a-4631-a246-df1e310f86b5" (UID: "8f7a7ba9-516a-4631-a246-df1e310f86b5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:13:42.751384 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:42.751259    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f7a7ba9-516a-4631-a246-df1e310f86b5-service-ca" (OuterVolumeSpecName: "service-ca") pod "8f7a7ba9-516a-4631-a246-df1e310f86b5" (UID: "8f7a7ba9-516a-4631-a246-df1e310f86b5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:13:42.753104 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:42.753081    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f7a7ba9-516a-4631-a246-df1e310f86b5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8f7a7ba9-516a-4631-a246-df1e310f86b5" (UID: "8f7a7ba9-516a-4631-a246-df1e310f86b5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:42.753195 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:42.753147    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f7a7ba9-516a-4631-a246-df1e310f86b5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8f7a7ba9-516a-4631-a246-df1e310f86b5" (UID: "8f7a7ba9-516a-4631-a246-df1e310f86b5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:42.753195 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:42.753152    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7a7ba9-516a-4631-a246-df1e310f86b5-kube-api-access-b5kk8" (OuterVolumeSpecName: "kube-api-access-b5kk8") pod "8f7a7ba9-516a-4631-a246-df1e310f86b5" (UID: "8f7a7ba9-516a-4631-a246-df1e310f86b5"). InnerVolumeSpecName "kube-api-access-b5kk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:13:42.855835 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:42.852342    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f7a7ba9-516a-4631-a246-df1e310f86b5-console-oauth-config\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:13:42.855835 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:42.852381    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f7a7ba9-516a-4631-a246-df1e310f86b5-oauth-serving-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:13:42.855835 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:42.852405    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f7a7ba9-516a-4631-a246-df1e310f86b5-console-serving-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:13:42.855835 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:42.852421    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7a7ba9-516a-4631-a246-df1e310f86b5-service-ca\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:13:42.855835 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:42.852434    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f7a7ba9-516a-4631-a246-df1e310f86b5-console-config\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:13:42.855835 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:42.852446    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5kk8\" (UniqueName: \"kubernetes.io/projected/8f7a7ba9-516a-4631-a246-df1e310f86b5-kube-api-access-b5kk8\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:13:43.375036 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:43.374998    2581 generic.go:356] "Generic (PLEG): container finished" podID="8f7a7ba9-516a-4631-a246-df1e310f86b5" containerID="2f660cb52c41106d0ab333eb476bf9473a399799c40f144deac67178554905e6" exitCode=0
Sep 09 20:13:43.375419 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:43.375063    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85bff69fd5-9bj2w"
Sep 09 20:13:43.375419 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:43.375068    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85bff69fd5-9bj2w" event={"ID":"8f7a7ba9-516a-4631-a246-df1e310f86b5","Type":"ContainerDied","Data":"2f660cb52c41106d0ab333eb476bf9473a399799c40f144deac67178554905e6"}
Sep 09 20:13:43.375419 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:43.375156    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85bff69fd5-9bj2w" event={"ID":"8f7a7ba9-516a-4631-a246-df1e310f86b5","Type":"ContainerDied","Data":"ad6e1a4490f930540f2092ee23145ee3b2e23a04a5887ea716bd7a21f30f4b62"}
Sep 09 20:13:43.375419 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:43.375186    2581 scope.go:117] "RemoveContainer" containerID="2f660cb52c41106d0ab333eb476bf9473a399799c40f144deac67178554905e6"
Sep 09 20:13:43.383912 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:43.383891    2581 scope.go:117] "RemoveContainer" containerID="2f660cb52c41106d0ab333eb476bf9473a399799c40f144deac67178554905e6"
Sep 09 20:13:43.384163 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:13:43.384143    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f660cb52c41106d0ab333eb476bf9473a399799c40f144deac67178554905e6\": container with ID starting with 2f660cb52c41106d0ab333eb476bf9473a399799c40f144deac67178554905e6 not found: ID does not exist" containerID="2f660cb52c41106d0ab333eb476bf9473a399799c40f144deac67178554905e6"
Sep 09 20:13:43.384231 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:43.384170    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f660cb52c41106d0ab333eb476bf9473a399799c40f144deac67178554905e6"} err="failed to get container status \"2f660cb52c41106d0ab333eb476bf9473a399799c40f144deac67178554905e6\": rpc error: code = NotFound desc = could not find container \"2f660cb52c41106d0ab333eb476bf9473a399799c40f144deac67178554905e6\": container with ID starting with 2f660cb52c41106d0ab333eb476bf9473a399799c40f144deac67178554905e6 not found: ID does not exist"
Sep 09 20:13:43.396208 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:43.396186    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85bff69fd5-9bj2w"]
Sep 09 20:13:43.398081 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:43.398062    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85bff69fd5-9bj2w"]
Sep 09 20:13:44.633391 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:13:44.633361    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f7a7ba9-516a-4631-a246-df1e310f86b5" path="/var/lib/kubelet/pods/8f7a7ba9-516a-4631-a246-df1e310f86b5/volumes"
Sep 09 20:14:14.463105 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.463057    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68568fb545-2gjzn"]
Sep 09 20:14:14.463667 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.463427    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f7a7ba9-516a-4631-a246-df1e310f86b5" containerName="console"
Sep 09 20:14:14.463667 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.463444    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f7a7ba9-516a-4631-a246-df1e310f86b5" containerName="console"
Sep 09 20:14:14.463667 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.463529    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f7a7ba9-516a-4631-a246-df1e310f86b5" containerName="console"
Sep 09 20:14:14.467649 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.467628    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:14.479978 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.479955    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68568fb545-2gjzn"]
Sep 09 20:14:14.578403 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.578356    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cznkt\" (UniqueName: \"kubernetes.io/projected/644a3f32-10dc-40c6-94c4-e359dc2d9993-kube-api-access-cznkt\") pod \"console-68568fb545-2gjzn\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") " pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:14.578403 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.578408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/644a3f32-10dc-40c6-94c4-e359dc2d9993-console-serving-cert\") pod \"console-68568fb545-2gjzn\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") " pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:14.578662 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.578435    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/644a3f32-10dc-40c6-94c4-e359dc2d9993-oauth-serving-cert\") pod \"console-68568fb545-2gjzn\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") " pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:14.578662 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.578501    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/644a3f32-10dc-40c6-94c4-e359dc2d9993-service-ca\") pod \"console-68568fb545-2gjzn\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") " pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:14.578662 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.578533    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/644a3f32-10dc-40c6-94c4-e359dc2d9993-console-config\") pod \"console-68568fb545-2gjzn\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") " pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:14.578662 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.578568    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/644a3f32-10dc-40c6-94c4-e359dc2d9993-trusted-ca-bundle\") pod \"console-68568fb545-2gjzn\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") " pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:14.578662 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.578597    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/644a3f32-10dc-40c6-94c4-e359dc2d9993-console-oauth-config\") pod \"console-68568fb545-2gjzn\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") " pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:14.679080 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.679040    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cznkt\" (UniqueName: \"kubernetes.io/projected/644a3f32-10dc-40c6-94c4-e359dc2d9993-kube-api-access-cznkt\") pod \"console-68568fb545-2gjzn\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") " pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:14.679080 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.679083    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/644a3f32-10dc-40c6-94c4-e359dc2d9993-console-serving-cert\") pod \"console-68568fb545-2gjzn\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") " pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:14.679278 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.679103    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/644a3f32-10dc-40c6-94c4-e359dc2d9993-oauth-serving-cert\") pod \"console-68568fb545-2gjzn\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") " pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:14.679278 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.679152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/644a3f32-10dc-40c6-94c4-e359dc2d9993-service-ca\") pod \"console-68568fb545-2gjzn\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") " pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:14.679278 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.679183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/644a3f32-10dc-40c6-94c4-e359dc2d9993-console-config\") pod \"console-68568fb545-2gjzn\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") " pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:14.679278 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.679217    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/644a3f32-10dc-40c6-94c4-e359dc2d9993-trusted-ca-bundle\") pod \"console-68568fb545-2gjzn\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") " pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:14.679278 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.679251    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/644a3f32-10dc-40c6-94c4-e359dc2d9993-console-oauth-config\") pod \"console-68568fb545-2gjzn\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") " pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:14.679915 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.679893    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/644a3f32-10dc-40c6-94c4-e359dc2d9993-oauth-serving-cert\") pod \"console-68568fb545-2gjzn\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") " pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:14.680034 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.679989    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/644a3f32-10dc-40c6-94c4-e359dc2d9993-console-config\") pod \"console-68568fb545-2gjzn\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") " pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:14.680092 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.680061    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/644a3f32-10dc-40c6-94c4-e359dc2d9993-service-ca\") pod \"console-68568fb545-2gjzn\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") " pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:14.680195 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.680172    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/644a3f32-10dc-40c6-94c4-e359dc2d9993-trusted-ca-bundle\") pod \"console-68568fb545-2gjzn\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") " pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:14.681588 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.681569    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/644a3f32-10dc-40c6-94c4-e359dc2d9993-console-oauth-config\") pod \"console-68568fb545-2gjzn\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") " pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:14.681713 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.681696    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/644a3f32-10dc-40c6-94c4-e359dc2d9993-console-serving-cert\") pod \"console-68568fb545-2gjzn\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") " pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:14.687467 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.687431    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cznkt\" (UniqueName: \"kubernetes.io/projected/644a3f32-10dc-40c6-94c4-e359dc2d9993-kube-api-access-cznkt\") pod \"console-68568fb545-2gjzn\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") " pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:14.776933 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:14.776902    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:15.053760 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:15.053684    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68568fb545-2gjzn"]
Sep 09 20:14:15.056899 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:14:15.056873    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644a3f32_10dc_40c6_94c4_e359dc2d9993.slice/crio-7b4b58503cfaa025e0054546b08dc318eea53c986247432119eb7cc9ec9093eb WatchSource:0}: Error finding container 7b4b58503cfaa025e0054546b08dc318eea53c986247432119eb7cc9ec9093eb: Status 404 returned error can't find the container with id 7b4b58503cfaa025e0054546b08dc318eea53c986247432119eb7cc9ec9093eb
Sep 09 20:14:15.484953 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:15.484921    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68568fb545-2gjzn" event={"ID":"644a3f32-10dc-40c6-94c4-e359dc2d9993","Type":"ContainerStarted","Data":"6c9cd0d62dcb6dfa86cdbc5543c0dc885443f0d061b9f1d832d6f85dae888534"}
Sep 09 20:14:15.485316 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:15.484969    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68568fb545-2gjzn" event={"ID":"644a3f32-10dc-40c6-94c4-e359dc2d9993","Type":"ContainerStarted","Data":"7b4b58503cfaa025e0054546b08dc318eea53c986247432119eb7cc9ec9093eb"}
Sep 09 20:14:15.507243 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:15.507200    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68568fb545-2gjzn" podStartSLOduration=1.507184917 podStartE2EDuration="1.507184917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:14:15.50568128 +0000 UTC m=+197.510858276" watchObservedRunningTime="2026-09-09 20:14:15.507184917 +0000 UTC m=+197.512361912"
Sep 09 20:14:24.777851 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:24.777806    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:24.778245 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:24.777879    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:24.782521 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:24.782500    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:25.522033 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:25.522005    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:14:25.571826 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:25.571796    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66d5c49969-szhzv"]
Sep 09 20:14:50.597647 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:50.597585    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66d5c49969-szhzv" podUID="87569eba-fd45-4826-80b5-d0201e63129d" containerName="console" containerID="cri-o://e6d8c399841f3540f27032a4a4b88005ba987883e2ceff0f6e70c20fac238687" gracePeriod=15
Sep 09 20:14:50.828125 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:50.828103    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:14:50.941852 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:50.941774    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87569eba-fd45-4826-80b5-d0201e63129d-trusted-ca-bundle\") pod \"87569eba-fd45-4826-80b5-d0201e63129d\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") "
Sep 09 20:14:50.941852 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:50.941817    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5dvt\" (UniqueName: \"kubernetes.io/projected/87569eba-fd45-4826-80b5-d0201e63129d-kube-api-access-x5dvt\") pod \"87569eba-fd45-4826-80b5-d0201e63129d\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") "
Sep 09 20:14:50.941852 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:50.941840    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87569eba-fd45-4826-80b5-d0201e63129d-console-config\") pod \"87569eba-fd45-4826-80b5-d0201e63129d\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") "
Sep 09 20:14:50.942092 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:50.941883    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87569eba-fd45-4826-80b5-d0201e63129d-oauth-serving-cert\") pod \"87569eba-fd45-4826-80b5-d0201e63129d\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") "
Sep 09 20:14:50.942092 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:50.941913    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87569eba-fd45-4826-80b5-d0201e63129d-console-oauth-config\") pod \"87569eba-fd45-4826-80b5-d0201e63129d\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") "
Sep 09 20:14:50.942092 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:50.941939    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87569eba-fd45-4826-80b5-d0201e63129d-service-ca\") pod \"87569eba-fd45-4826-80b5-d0201e63129d\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") "
Sep 09 20:14:50.942092 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:50.941981    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87569eba-fd45-4826-80b5-d0201e63129d-console-serving-cert\") pod \"87569eba-fd45-4826-80b5-d0201e63129d\" (UID: \"87569eba-fd45-4826-80b5-d0201e63129d\") "
Sep 09 20:14:50.942322 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:50.942294    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87569eba-fd45-4826-80b5-d0201e63129d-console-config" (OuterVolumeSpecName: "console-config") pod "87569eba-fd45-4826-80b5-d0201e63129d" (UID: "87569eba-fd45-4826-80b5-d0201e63129d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:14:50.942393 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:50.942328    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87569eba-fd45-4826-80b5-d0201e63129d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "87569eba-fd45-4826-80b5-d0201e63129d" (UID: "87569eba-fd45-4826-80b5-d0201e63129d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:14:50.942393 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:50.942379    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87569eba-fd45-4826-80b5-d0201e63129d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "87569eba-fd45-4826-80b5-d0201e63129d" (UID: "87569eba-fd45-4826-80b5-d0201e63129d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:14:50.942512 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:50.942442    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87569eba-fd45-4826-80b5-d0201e63129d-service-ca" (OuterVolumeSpecName: "service-ca") pod "87569eba-fd45-4826-80b5-d0201e63129d" (UID: "87569eba-fd45-4826-80b5-d0201e63129d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:14:50.944214 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:50.944183    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87569eba-fd45-4826-80b5-d0201e63129d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "87569eba-fd45-4826-80b5-d0201e63129d" (UID: "87569eba-fd45-4826-80b5-d0201e63129d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:14:50.944214 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:50.944206    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87569eba-fd45-4826-80b5-d0201e63129d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "87569eba-fd45-4826-80b5-d0201e63129d" (UID: "87569eba-fd45-4826-80b5-d0201e63129d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:14:50.944353 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:50.944306    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87569eba-fd45-4826-80b5-d0201e63129d-kube-api-access-x5dvt" (OuterVolumeSpecName: "kube-api-access-x5dvt") pod "87569eba-fd45-4826-80b5-d0201e63129d" (UID: "87569eba-fd45-4826-80b5-d0201e63129d"). InnerVolumeSpecName "kube-api-access-x5dvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:14:51.043338 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:51.043303    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87569eba-fd45-4826-80b5-d0201e63129d-trusted-ca-bundle\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:14:51.043338 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:51.043332    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5dvt\" (UniqueName: \"kubernetes.io/projected/87569eba-fd45-4826-80b5-d0201e63129d-kube-api-access-x5dvt\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:14:51.043566 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:51.043346    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87569eba-fd45-4826-80b5-d0201e63129d-console-config\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:14:51.043566 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:51.043358    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87569eba-fd45-4826-80b5-d0201e63129d-oauth-serving-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:14:51.043566 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:51.043369    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87569eba-fd45-4826-80b5-d0201e63129d-console-oauth-config\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:14:51.043566 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:51.043380    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87569eba-fd45-4826-80b5-d0201e63129d-service-ca\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:14:51.043566 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:51.043390    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87569eba-fd45-4826-80b5-d0201e63129d-console-serving-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:14:51.606982 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:51.606944    2581 generic.go:356] "Generic (PLEG): container finished" podID="87569eba-fd45-4826-80b5-d0201e63129d" containerID="e6d8c399841f3540f27032a4a4b88005ba987883e2ceff0f6e70c20fac238687" exitCode=0
Sep 09 20:14:51.606982 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:51.606987    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66d5c49969-szhzv" event={"ID":"87569eba-fd45-4826-80b5-d0201e63129d","Type":"ContainerDied","Data":"e6d8c399841f3540f27032a4a4b88005ba987883e2ceff0f6e70c20fac238687"}
Sep 09 20:14:51.607548 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:51.607007    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66d5c49969-szhzv"
Sep 09 20:14:51.607548 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:51.607029    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66d5c49969-szhzv" event={"ID":"87569eba-fd45-4826-80b5-d0201e63129d","Type":"ContainerDied","Data":"81190eb1dbfe6f1b380f93e9f66cc7a896eaaca9d5fa24d9076b81bf40fd0cf4"}
Sep 09 20:14:51.607548 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:51.607048    2581 scope.go:117] "RemoveContainer" containerID="e6d8c399841f3540f27032a4a4b88005ba987883e2ceff0f6e70c20fac238687"
Sep 09 20:14:51.615965 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:51.615948    2581 scope.go:117] "RemoveContainer" containerID="e6d8c399841f3540f27032a4a4b88005ba987883e2ceff0f6e70c20fac238687"
Sep 09 20:14:51.616259 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:14:51.616241    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6d8c399841f3540f27032a4a4b88005ba987883e2ceff0f6e70c20fac238687\": container with ID starting with e6d8c399841f3540f27032a4a4b88005ba987883e2ceff0f6e70c20fac238687 not found: ID does not exist" containerID="e6d8c399841f3540f27032a4a4b88005ba987883e2ceff0f6e70c20fac238687"
Sep 09 20:14:51.616315 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:51.616269    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d8c399841f3540f27032a4a4b88005ba987883e2ceff0f6e70c20fac238687"} err="failed to get container status \"e6d8c399841f3540f27032a4a4b88005ba987883e2ceff0f6e70c20fac238687\": rpc error: code = NotFound desc = could not find container \"e6d8c399841f3540f27032a4a4b88005ba987883e2ceff0f6e70c20fac238687\": container with ID starting with e6d8c399841f3540f27032a4a4b88005ba987883e2ceff0f6e70c20fac238687 not found: ID does not exist"
Sep 09 20:14:51.628838 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:51.628817    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66d5c49969-szhzv"]
Sep 09 20:14:51.636010 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:51.634139    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66d5c49969-szhzv"]
Sep 09 20:14:52.633441 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:14:52.633406    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87569eba-fd45-4826-80b5-d0201e63129d" path="/var/lib/kubelet/pods/87569eba-fd45-4826-80b5-d0201e63129d/volumes"
Sep 09 20:15:24.829261 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:24.829187    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-678bcff8bd-7dczz"]
Sep 09 20:15:24.829672 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:24.829440    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87569eba-fd45-4826-80b5-d0201e63129d" containerName="console"
Sep 09 20:15:24.829672 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:24.829469    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="87569eba-fd45-4826-80b5-d0201e63129d" containerName="console"
Sep 09 20:15:24.829672 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:24.829527    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="87569eba-fd45-4826-80b5-d0201e63129d" containerName="console"
Sep 09 20:15:24.831782 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:24.831758    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:24.842976 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:24.842928    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-678bcff8bd-7dczz"]
Sep 09 20:15:24.981071 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:24.981034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38fd331d-779a-4ac5-980d-1aaa79599361-console-config\") pod \"console-678bcff8bd-7dczz\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") " pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:24.981235 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:24.981078    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38fd331d-779a-4ac5-980d-1aaa79599361-trusted-ca-bundle\") pod \"console-678bcff8bd-7dczz\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") " pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:24.981235 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:24.981100    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38fd331d-779a-4ac5-980d-1aaa79599361-console-oauth-config\") pod \"console-678bcff8bd-7dczz\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") " pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:24.981235 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:24.981137    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38fd331d-779a-4ac5-980d-1aaa79599361-service-ca\") pod \"console-678bcff8bd-7dczz\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") " pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:24.981235 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:24.981197    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38fd331d-779a-4ac5-980d-1aaa79599361-oauth-serving-cert\") pod \"console-678bcff8bd-7dczz\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") " pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:24.981386 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:24.981249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmsbh\" (UniqueName: \"kubernetes.io/projected/38fd331d-779a-4ac5-980d-1aaa79599361-kube-api-access-vmsbh\") pod \"console-678bcff8bd-7dczz\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") " pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:24.981386 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:24.981268    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38fd331d-779a-4ac5-980d-1aaa79599361-console-serving-cert\") pod \"console-678bcff8bd-7dczz\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") " pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:25.082214 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:25.082128    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38fd331d-779a-4ac5-980d-1aaa79599361-trusted-ca-bundle\") pod \"console-678bcff8bd-7dczz\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") " pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:25.082214 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:25.082175    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38fd331d-779a-4ac5-980d-1aaa79599361-console-oauth-config\") pod \"console-678bcff8bd-7dczz\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") " pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:25.082214 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:25.082190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38fd331d-779a-4ac5-980d-1aaa79599361-service-ca\") pod \"console-678bcff8bd-7dczz\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") " pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:25.082214 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:25.082217    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38fd331d-779a-4ac5-980d-1aaa79599361-oauth-serving-cert\") pod \"console-678bcff8bd-7dczz\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") " pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:25.082542 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:25.082249    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmsbh\" (UniqueName: \"kubernetes.io/projected/38fd331d-779a-4ac5-980d-1aaa79599361-kube-api-access-vmsbh\") pod \"console-678bcff8bd-7dczz\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") " pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:25.082542 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:25.082269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38fd331d-779a-4ac5-980d-1aaa79599361-console-serving-cert\") pod \"console-678bcff8bd-7dczz\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") " pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:25.082542 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:25.082298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38fd331d-779a-4ac5-980d-1aaa79599361-console-config\") pod \"console-678bcff8bd-7dczz\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") " pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:25.083020 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:25.082995    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38fd331d-779a-4ac5-980d-1aaa79599361-console-config\") pod \"console-678bcff8bd-7dczz\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") " pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:25.083138 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:25.083091    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38fd331d-779a-4ac5-980d-1aaa79599361-oauth-serving-cert\") pod \"console-678bcff8bd-7dczz\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") " pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:25.083138 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:25.083125    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38fd331d-779a-4ac5-980d-1aaa79599361-trusted-ca-bundle\") pod \"console-678bcff8bd-7dczz\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") " pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:25.083215 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:25.083141    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38fd331d-779a-4ac5-980d-1aaa79599361-service-ca\") pod \"console-678bcff8bd-7dczz\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") " pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:25.084723 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:25.084704    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38fd331d-779a-4ac5-980d-1aaa79599361-console-oauth-config\") pod \"console-678bcff8bd-7dczz\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") " pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:25.084825 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:25.084809    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38fd331d-779a-4ac5-980d-1aaa79599361-console-serving-cert\") pod \"console-678bcff8bd-7dczz\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") " pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:25.091283 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:25.091262    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmsbh\" (UniqueName: \"kubernetes.io/projected/38fd331d-779a-4ac5-980d-1aaa79599361-kube-api-access-vmsbh\") pod \"console-678bcff8bd-7dczz\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") " pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:25.140857 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:25.140833    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:25.410095 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:25.410070    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-678bcff8bd-7dczz"]
Sep 09 20:15:25.412439 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:15:25.412404    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38fd331d_779a_4ac5_980d_1aaa79599361.slice/crio-024b58ff508f07967ed8e33fe6483ecfdc798472a50093b02b609087325e27e0 WatchSource:0}: Error finding container 024b58ff508f07967ed8e33fe6483ecfdc798472a50093b02b609087325e27e0: Status 404 returned error can't find the container with id 024b58ff508f07967ed8e33fe6483ecfdc798472a50093b02b609087325e27e0
Sep 09 20:15:25.720454 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:25.720376    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678bcff8bd-7dczz" event={"ID":"38fd331d-779a-4ac5-980d-1aaa79599361","Type":"ContainerStarted","Data":"bebb275e06dc9b584bf057543a87cad3bfee02ad0fef6ca492bbfffe4aae9105"}
Sep 09 20:15:25.720454 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:25.720423    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678bcff8bd-7dczz" event={"ID":"38fd331d-779a-4ac5-980d-1aaa79599361","Type":"ContainerStarted","Data":"024b58ff508f07967ed8e33fe6483ecfdc798472a50093b02b609087325e27e0"}
Sep 09 20:15:25.737902 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:25.737857    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-678bcff8bd-7dczz" podStartSLOduration=1.737842696 podStartE2EDuration="1.737842696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:15:25.736568573 +0000 UTC m=+267.741745571" watchObservedRunningTime="2026-09-09 20:15:25.737842696 +0000 UTC m=+267.743019696"
Sep 09 20:15:35.141267 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:35.141226    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:35.141812 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:35.141288    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:35.146120 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:35.146099    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:35.758244 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:35.758219    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:15:35.805895 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:35.805862    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68568fb545-2gjzn"]
Sep 09 20:15:58.386264 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:58.386237    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 20:15:58.389183 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:58.389162    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 20:15:58.391524 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:58.391507    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 20:15:58.394391 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:58.394374    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 20:15:58.396034 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:15:58.396016    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:16:00.825760 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:00.825699    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68568fb545-2gjzn" podUID="644a3f32-10dc-40c6-94c4-e359dc2d9993" containerName="console" containerID="cri-o://6c9cd0d62dcb6dfa86cdbc5543c0dc885443f0d061b9f1d832d6f85dae888534" gracePeriod=15
Sep 09 20:16:01.063574 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.063553    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:16:01.133100 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.133021    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/644a3f32-10dc-40c6-94c4-e359dc2d9993-console-oauth-config\") pod \"644a3f32-10dc-40c6-94c4-e359dc2d9993\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") "
Sep 09 20:16:01.133100 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.133077    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/644a3f32-10dc-40c6-94c4-e359dc2d9993-service-ca\") pod \"644a3f32-10dc-40c6-94c4-e359dc2d9993\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") "
Sep 09 20:16:01.133100 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.133099    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/644a3f32-10dc-40c6-94c4-e359dc2d9993-console-config\") pod \"644a3f32-10dc-40c6-94c4-e359dc2d9993\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") "
Sep 09 20:16:01.133410 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.133125    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/644a3f32-10dc-40c6-94c4-e359dc2d9993-console-serving-cert\") pod \"644a3f32-10dc-40c6-94c4-e359dc2d9993\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") "
Sep 09 20:16:01.133410 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.133147    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/644a3f32-10dc-40c6-94c4-e359dc2d9993-oauth-serving-cert\") pod \"644a3f32-10dc-40c6-94c4-e359dc2d9993\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") "
Sep 09 20:16:01.133410 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.133181    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cznkt\" (UniqueName: \"kubernetes.io/projected/644a3f32-10dc-40c6-94c4-e359dc2d9993-kube-api-access-cznkt\") pod \"644a3f32-10dc-40c6-94c4-e359dc2d9993\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") "
Sep 09 20:16:01.133410 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.133213    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/644a3f32-10dc-40c6-94c4-e359dc2d9993-trusted-ca-bundle\") pod \"644a3f32-10dc-40c6-94c4-e359dc2d9993\" (UID: \"644a3f32-10dc-40c6-94c4-e359dc2d9993\") "
Sep 09 20:16:01.133657 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.133610    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/644a3f32-10dc-40c6-94c4-e359dc2d9993-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "644a3f32-10dc-40c6-94c4-e359dc2d9993" (UID: "644a3f32-10dc-40c6-94c4-e359dc2d9993"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:16:01.133657 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.133627    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/644a3f32-10dc-40c6-94c4-e359dc2d9993-console-config" (OuterVolumeSpecName: "console-config") pod "644a3f32-10dc-40c6-94c4-e359dc2d9993" (UID: "644a3f32-10dc-40c6-94c4-e359dc2d9993"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:16:01.133657 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.133634    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/644a3f32-10dc-40c6-94c4-e359dc2d9993-service-ca" (OuterVolumeSpecName: "service-ca") pod "644a3f32-10dc-40c6-94c4-e359dc2d9993" (UID: "644a3f32-10dc-40c6-94c4-e359dc2d9993"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:16:01.133820 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.133731    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/644a3f32-10dc-40c6-94c4-e359dc2d9993-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "644a3f32-10dc-40c6-94c4-e359dc2d9993" (UID: "644a3f32-10dc-40c6-94c4-e359dc2d9993"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:16:01.135234 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.135211    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/644a3f32-10dc-40c6-94c4-e359dc2d9993-kube-api-access-cznkt" (OuterVolumeSpecName: "kube-api-access-cznkt") pod "644a3f32-10dc-40c6-94c4-e359dc2d9993" (UID: "644a3f32-10dc-40c6-94c4-e359dc2d9993"). InnerVolumeSpecName "kube-api-access-cznkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:16:01.135351 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.135232    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/644a3f32-10dc-40c6-94c4-e359dc2d9993-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "644a3f32-10dc-40c6-94c4-e359dc2d9993" (UID: "644a3f32-10dc-40c6-94c4-e359dc2d9993"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:16:01.135351 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.135278    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/644a3f32-10dc-40c6-94c4-e359dc2d9993-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "644a3f32-10dc-40c6-94c4-e359dc2d9993" (UID: "644a3f32-10dc-40c6-94c4-e359dc2d9993"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:16:01.233939 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.233905    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/644a3f32-10dc-40c6-94c4-e359dc2d9993-service-ca\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:16:01.233939 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.233934    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/644a3f32-10dc-40c6-94c4-e359dc2d9993-console-config\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:16:01.233939 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.233943    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/644a3f32-10dc-40c6-94c4-e359dc2d9993-console-serving-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:16:01.233939 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.233952    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/644a3f32-10dc-40c6-94c4-e359dc2d9993-oauth-serving-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:16:01.234193 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.233961    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cznkt\" (UniqueName: \"kubernetes.io/projected/644a3f32-10dc-40c6-94c4-e359dc2d9993-kube-api-access-cznkt\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:16:01.234193 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.233970    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/644a3f32-10dc-40c6-94c4-e359dc2d9993-trusted-ca-bundle\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:16:01.234193 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.233978    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/644a3f32-10dc-40c6-94c4-e359dc2d9993-console-oauth-config\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:16:01.842785 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.842746    2581 generic.go:356] "Generic (PLEG): container finished" podID="644a3f32-10dc-40c6-94c4-e359dc2d9993" containerID="6c9cd0d62dcb6dfa86cdbc5543c0dc885443f0d061b9f1d832d6f85dae888534" exitCode=0
Sep 09 20:16:01.843258 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.842790    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68568fb545-2gjzn"
Sep 09 20:16:01.843258 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.842829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68568fb545-2gjzn" event={"ID":"644a3f32-10dc-40c6-94c4-e359dc2d9993","Type":"ContainerDied","Data":"6c9cd0d62dcb6dfa86cdbc5543c0dc885443f0d061b9f1d832d6f85dae888534"}
Sep 09 20:16:01.843258 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.842876    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68568fb545-2gjzn" event={"ID":"644a3f32-10dc-40c6-94c4-e359dc2d9993","Type":"ContainerDied","Data":"7b4b58503cfaa025e0054546b08dc318eea53c986247432119eb7cc9ec9093eb"}
Sep 09 20:16:01.843258 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.842900    2581 scope.go:117] "RemoveContainer" containerID="6c9cd0d62dcb6dfa86cdbc5543c0dc885443f0d061b9f1d832d6f85dae888534"
Sep 09 20:16:01.856234 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.856212    2581 scope.go:117] "RemoveContainer" containerID="6c9cd0d62dcb6dfa86cdbc5543c0dc885443f0d061b9f1d832d6f85dae888534"
Sep 09 20:16:01.856548 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:16:01.856526    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c9cd0d62dcb6dfa86cdbc5543c0dc885443f0d061b9f1d832d6f85dae888534\": container with ID starting with 6c9cd0d62dcb6dfa86cdbc5543c0dc885443f0d061b9f1d832d6f85dae888534 not found: ID does not exist" containerID="6c9cd0d62dcb6dfa86cdbc5543c0dc885443f0d061b9f1d832d6f85dae888534"
Sep 09 20:16:01.856594 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.856556    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9cd0d62dcb6dfa86cdbc5543c0dc885443f0d061b9f1d832d6f85dae888534"} err="failed to get container status \"6c9cd0d62dcb6dfa86cdbc5543c0dc885443f0d061b9f1d832d6f85dae888534\": rpc error: code = NotFound desc = could not find container \"6c9cd0d62dcb6dfa86cdbc5543c0dc885443f0d061b9f1d832d6f85dae888534\": container with ID starting with 6c9cd0d62dcb6dfa86cdbc5543c0dc885443f0d061b9f1d832d6f85dae888534 not found: ID does not exist"
Sep 09 20:16:01.867061 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.867036    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68568fb545-2gjzn"]
Sep 09 20:16:01.873315 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:01.873294    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68568fb545-2gjzn"]
Sep 09 20:16:02.633904 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:16:02.633869    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="644a3f32-10dc-40c6-94c4-e359dc2d9993" path="/var/lib/kubelet/pods/644a3f32-10dc-40c6-94c4-e359dc2d9993/volumes"
Sep 09 20:17:01.264175 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:01.264142    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc"]
Sep 09 20:17:01.264575 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:01.264371    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="644a3f32-10dc-40c6-94c4-e359dc2d9993" containerName="console"
Sep 09 20:17:01.264575 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:01.264383    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="644a3f32-10dc-40c6-94c4-e359dc2d9993" containerName="console"
Sep 09 20:17:01.264575 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:01.264430    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="644a3f32-10dc-40c6-94c4-e359dc2d9993" containerName="console"
Sep 09 20:17:01.267520 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:01.267500    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc"
Sep 09 20:17:01.269964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:01.269942    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-npsx8\""
Sep 09 20:17:01.270084 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:01.269959    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 20:17:01.270926 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:01.270911    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 20:17:01.275698 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:01.275674    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc"]
Sep 09 20:17:01.358351 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:01.358310    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hk2d\" (UniqueName: \"kubernetes.io/projected/5574d811-1ccb-47ff-a85f-15b4649474a2-kube-api-access-8hk2d\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc\" (UID: \"5574d811-1ccb-47ff-a85f-15b4649474a2\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc"
Sep 09 20:17:01.358351 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:01.358358    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5574d811-1ccb-47ff-a85f-15b4649474a2-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc\" (UID: \"5574d811-1ccb-47ff-a85f-15b4649474a2\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc"
Sep 09 20:17:01.358606 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:01.358423    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5574d811-1ccb-47ff-a85f-15b4649474a2-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc\" (UID: \"5574d811-1ccb-47ff-a85f-15b4649474a2\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc"
Sep 09 20:17:01.459350 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:01.459313    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5574d811-1ccb-47ff-a85f-15b4649474a2-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc\" (UID: \"5574d811-1ccb-47ff-a85f-15b4649474a2\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc"
Sep 09 20:17:01.459537 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:01.459359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hk2d\" (UniqueName: \"kubernetes.io/projected/5574d811-1ccb-47ff-a85f-15b4649474a2-kube-api-access-8hk2d\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc\" (UID: \"5574d811-1ccb-47ff-a85f-15b4649474a2\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc"
Sep 09 20:17:01.459537 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:01.459383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5574d811-1ccb-47ff-a85f-15b4649474a2-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc\" (UID: \"5574d811-1ccb-47ff-a85f-15b4649474a2\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc"
Sep 09 20:17:01.459712 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:01.459697    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5574d811-1ccb-47ff-a85f-15b4649474a2-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc\" (UID: \"5574d811-1ccb-47ff-a85f-15b4649474a2\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc"
Sep 09 20:17:01.459754 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:01.459736    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5574d811-1ccb-47ff-a85f-15b4649474a2-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc\" (UID: \"5574d811-1ccb-47ff-a85f-15b4649474a2\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc"
Sep 09 20:17:01.468319 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:01.468295    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hk2d\" (UniqueName: \"kubernetes.io/projected/5574d811-1ccb-47ff-a85f-15b4649474a2-kube-api-access-8hk2d\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc\" (UID: \"5574d811-1ccb-47ff-a85f-15b4649474a2\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc"
Sep 09 20:17:01.577157 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:01.577071    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc"
Sep 09 20:17:01.844529 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:01.844506    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc"]
Sep 09 20:17:01.846503 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:17:01.846474    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5574d811_1ccb_47ff_a85f_15b4649474a2.slice/crio-8639d24f1a226ec3f942c2b61896962c8626c99ddb5bedc30b35f79e23494ac0 WatchSource:0}: Error finding container 8639d24f1a226ec3f942c2b61896962c8626c99ddb5bedc30b35f79e23494ac0: Status 404 returned error can't find the container with id 8639d24f1a226ec3f942c2b61896962c8626c99ddb5bedc30b35f79e23494ac0
Sep 09 20:17:01.848643 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:01.848623    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:17:02.047691 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:02.047658    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc" event={"ID":"5574d811-1ccb-47ff-a85f-15b4649474a2","Type":"ContainerStarted","Data":"8639d24f1a226ec3f942c2b61896962c8626c99ddb5bedc30b35f79e23494ac0"}
Sep 09 20:17:09.076525 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:09.076488    2581 generic.go:356] "Generic (PLEG): container finished" podID="5574d811-1ccb-47ff-a85f-15b4649474a2" containerID="345fff3c8a1f8c6bf353e08131f2a2fb1a74b8fd9721053e55e56688875d542d" exitCode=0
Sep 09 20:17:09.077092 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:09.076568    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc" event={"ID":"5574d811-1ccb-47ff-a85f-15b4649474a2","Type":"ContainerDied","Data":"345fff3c8a1f8c6bf353e08131f2a2fb1a74b8fd9721053e55e56688875d542d"}
Sep 09 20:17:12.092427 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:12.092393    2581 generic.go:356] "Generic (PLEG): container finished" podID="5574d811-1ccb-47ff-a85f-15b4649474a2" containerID="eb113d17c6302bc51a2ee39b01ffa7674beb1834d170087d3217d3d4b13ca99b" exitCode=0
Sep 09 20:17:12.092427 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:12.092432    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc" event={"ID":"5574d811-1ccb-47ff-a85f-15b4649474a2","Type":"ContainerDied","Data":"eb113d17c6302bc51a2ee39b01ffa7674beb1834d170087d3217d3d4b13ca99b"}
Sep 09 20:17:20.125811 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:20.125778    2581 generic.go:356] "Generic (PLEG): container finished" podID="5574d811-1ccb-47ff-a85f-15b4649474a2" containerID="c858b616e9bcaf521ea17c5fd84fcdf51121f562270af9f72a84c77f0c5a25f8" exitCode=0
Sep 09 20:17:20.125811 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:20.125817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc" event={"ID":"5574d811-1ccb-47ff-a85f-15b4649474a2","Type":"ContainerDied","Data":"c858b616e9bcaf521ea17c5fd84fcdf51121f562270af9f72a84c77f0c5a25f8"}
Sep 09 20:17:21.245584 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:21.245562    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc"
Sep 09 20:17:21.303761 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:21.303732    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hk2d\" (UniqueName: \"kubernetes.io/projected/5574d811-1ccb-47ff-a85f-15b4649474a2-kube-api-access-8hk2d\") pod \"5574d811-1ccb-47ff-a85f-15b4649474a2\" (UID: \"5574d811-1ccb-47ff-a85f-15b4649474a2\") "
Sep 09 20:17:21.303889 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:21.303787    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5574d811-1ccb-47ff-a85f-15b4649474a2-util\") pod \"5574d811-1ccb-47ff-a85f-15b4649474a2\" (UID: \"5574d811-1ccb-47ff-a85f-15b4649474a2\") "
Sep 09 20:17:21.303889 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:21.303827    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5574d811-1ccb-47ff-a85f-15b4649474a2-bundle\") pod \"5574d811-1ccb-47ff-a85f-15b4649474a2\" (UID: \"5574d811-1ccb-47ff-a85f-15b4649474a2\") "
Sep 09 20:17:21.304518 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:21.304448    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5574d811-1ccb-47ff-a85f-15b4649474a2-bundle" (OuterVolumeSpecName: "bundle") pod "5574d811-1ccb-47ff-a85f-15b4649474a2" (UID: "5574d811-1ccb-47ff-a85f-15b4649474a2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:17:21.305829 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:21.305800    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5574d811-1ccb-47ff-a85f-15b4649474a2-kube-api-access-8hk2d" (OuterVolumeSpecName: "kube-api-access-8hk2d") pod "5574d811-1ccb-47ff-a85f-15b4649474a2" (UID: "5574d811-1ccb-47ff-a85f-15b4649474a2"). InnerVolumeSpecName "kube-api-access-8hk2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:17:21.308783 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:21.308764    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5574d811-1ccb-47ff-a85f-15b4649474a2-util" (OuterVolumeSpecName: "util") pod "5574d811-1ccb-47ff-a85f-15b4649474a2" (UID: "5574d811-1ccb-47ff-a85f-15b4649474a2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:17:21.405118 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:21.405056    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5574d811-1ccb-47ff-a85f-15b4649474a2-util\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:17:21.405118 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:21.405079    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5574d811-1ccb-47ff-a85f-15b4649474a2-bundle\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:17:21.405118 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:21.405087    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hk2d\" (UniqueName: \"kubernetes.io/projected/5574d811-1ccb-47ff-a85f-15b4649474a2-kube-api-access-8hk2d\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:17:22.133956 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:22.133930    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc"
Sep 09 20:17:22.134110 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:22.133930    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1an8wtc" event={"ID":"5574d811-1ccb-47ff-a85f-15b4649474a2","Type":"ContainerDied","Data":"8639d24f1a226ec3f942c2b61896962c8626c99ddb5bedc30b35f79e23494ac0"}
Sep 09 20:17:22.134110 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:22.134045    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8639d24f1a226ec3f942c2b61896962c8626c99ddb5bedc30b35f79e23494ac0"
Sep 09 20:17:28.614699 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:28.614661    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ppbr2"]
Sep 09 20:17:28.615148 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:28.615005    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5574d811-1ccb-47ff-a85f-15b4649474a2" containerName="extract"
Sep 09 20:17:28.615148 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:28.615022    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5574d811-1ccb-47ff-a85f-15b4649474a2" containerName="extract"
Sep 09 20:17:28.615148 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:28.615037    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5574d811-1ccb-47ff-a85f-15b4649474a2" containerName="pull"
Sep 09 20:17:28.615148 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:28.615045    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5574d811-1ccb-47ff-a85f-15b4649474a2" containerName="pull"
Sep 09 20:17:28.615148 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:28.615072    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5574d811-1ccb-47ff-a85f-15b4649474a2" containerName="util"
Sep 09 20:17:28.615148 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:28.615079    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5574d811-1ccb-47ff-a85f-15b4649474a2" containerName="util"
Sep 09 20:17:28.615148 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:28.615138    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5574d811-1ccb-47ff-a85f-15b4649474a2" containerName="extract"
Sep 09 20:17:28.661417 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:28.661382    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ppbr2"
Sep 09 20:17:28.664084 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:28.664060    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 09 20:17:28.664215 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:28.664170    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-65t2f\""
Sep 09 20:17:28.664273 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:28.664220    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 09 20:17:28.664359 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:28.664336    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 09 20:17:28.666161 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:28.666141    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ppbr2"]
Sep 09 20:17:28.756326 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:28.756300    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftnhl\" (UniqueName: \"kubernetes.io/projected/8d2cd865-9719-4cb0-a3bb-2ba79881d9b5-kube-api-access-ftnhl\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-ppbr2\" (UID: \"8d2cd865-9719-4cb0-a3bb-2ba79881d9b5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ppbr2"
Sep 09 20:17:28.756449 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:28.756343    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/8d2cd865-9719-4cb0-a3bb-2ba79881d9b5-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-ppbr2\" (UID: \"8d2cd865-9719-4cb0-a3bb-2ba79881d9b5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ppbr2"
Sep 09 20:17:28.857610 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:28.857585    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftnhl\" (UniqueName: \"kubernetes.io/projected/8d2cd865-9719-4cb0-a3bb-2ba79881d9b5-kube-api-access-ftnhl\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-ppbr2\" (UID: \"8d2cd865-9719-4cb0-a3bb-2ba79881d9b5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ppbr2"
Sep 09 20:17:28.857728 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:28.857626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/8d2cd865-9719-4cb0-a3bb-2ba79881d9b5-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-ppbr2\" (UID: \"8d2cd865-9719-4cb0-a3bb-2ba79881d9b5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ppbr2"
Sep 09 20:17:28.859886 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:28.859866    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/8d2cd865-9719-4cb0-a3bb-2ba79881d9b5-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-ppbr2\" (UID: \"8d2cd865-9719-4cb0-a3bb-2ba79881d9b5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ppbr2"
Sep 09 20:17:28.875667 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:28.875617    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftnhl\" (UniqueName: \"kubernetes.io/projected/8d2cd865-9719-4cb0-a3bb-2ba79881d9b5-kube-api-access-ftnhl\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-ppbr2\" (UID: \"8d2cd865-9719-4cb0-a3bb-2ba79881d9b5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ppbr2"
Sep 09 20:17:28.971348 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:28.971315    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ppbr2"
Sep 09 20:17:29.258762 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:29.258734    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ppbr2"]
Sep 09 20:17:29.261186 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:17:29.261159    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2cd865_9719_4cb0_a3bb_2ba79881d9b5.slice/crio-663b5b4c4fe409b1d61d2f5a0f4aa306428c0fbe71a3ac34743d627861279948 WatchSource:0}: Error finding container 663b5b4c4fe409b1d61d2f5a0f4aa306428c0fbe71a3ac34743d627861279948: Status 404 returned error can't find the container with id 663b5b4c4fe409b1d61d2f5a0f4aa306428c0fbe71a3ac34743d627861279948
Sep 09 20:17:30.168343 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:30.168303    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ppbr2" event={"ID":"8d2cd865-9719-4cb0-a3bb-2ba79881d9b5","Type":"ContainerStarted","Data":"663b5b4c4fe409b1d61d2f5a0f4aa306428c0fbe71a3ac34743d627861279948"}
Sep 09 20:17:33.181814 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:33.181782    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ppbr2" event={"ID":"8d2cd865-9719-4cb0-a3bb-2ba79881d9b5","Type":"ContainerStarted","Data":"420333c30dcb73598380de304196e6e4b47c55b5697784eab0a30b845f4b3490"}
Sep 09 20:17:33.182210 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:33.181899    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ppbr2"
Sep 09 20:17:33.204939 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:33.204911    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-6l6gj"]
Sep 09 20:17:33.206844 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:33.206825    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-6l6gj"
Sep 09 20:17:33.208498 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:33.208444    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ppbr2" podStartSLOduration=2.175975601 podStartE2EDuration="5.208432171s" totalImagesPullingTime="3.03245657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:33.206753633 +0000 UTC m=+395.211930628" watchObservedRunningTime="2026-09-09 20:17:33.208432171 +0000 UTC m=+395.213609165"
Sep 09 20:17:33.209192 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:33.209165    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 09 20:17:33.209192 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:33.209177    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-hwkjt\""
Sep 09 20:17:33.209192 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:33.209169    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 09 20:17:33.217770 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:33.217749    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-6l6gj"]
Sep 09 20:17:33.288681 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:33.288657    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/2249b1c5-e912-466b-9706-4a93f4729d08-cabundle0\") pod \"keda-operator-579c8958f7-6l6gj\" (UID: \"2249b1c5-e912-466b-9706-4a93f4729d08\") " pod="openshift-keda/keda-operator-579c8958f7-6l6gj"
Sep 09 20:17:33.288824 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:33.288690    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4kdh\" (UniqueName: \"kubernetes.io/projected/2249b1c5-e912-466b-9706-4a93f4729d08-kube-api-access-c4kdh\") pod \"keda-operator-579c8958f7-6l6gj\" (UID: \"2249b1c5-e912-466b-9706-4a93f4729d08\") " pod="openshift-keda/keda-operator-579c8958f7-6l6gj"
Sep 09 20:17:33.288824 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:33.288749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2249b1c5-e912-466b-9706-4a93f4729d08-certificates\") pod \"keda-operator-579c8958f7-6l6gj\" (UID: \"2249b1c5-e912-466b-9706-4a93f4729d08\") " pod="openshift-keda/keda-operator-579c8958f7-6l6gj"
Sep 09 20:17:33.389684 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:33.389652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4kdh\" (UniqueName: \"kubernetes.io/projected/2249b1c5-e912-466b-9706-4a93f4729d08-kube-api-access-c4kdh\") pod \"keda-operator-579c8958f7-6l6gj\" (UID: \"2249b1c5-e912-466b-9706-4a93f4729d08\") " pod="openshift-keda/keda-operator-579c8958f7-6l6gj"
Sep 09 20:17:33.389860 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:33.389741    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2249b1c5-e912-466b-9706-4a93f4729d08-certificates\") pod \"keda-operator-579c8958f7-6l6gj\" (UID: \"2249b1c5-e912-466b-9706-4a93f4729d08\") " pod="openshift-keda/keda-operator-579c8958f7-6l6gj"
Sep 09 20:17:33.389860 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:33.389780    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/2249b1c5-e912-466b-9706-4a93f4729d08-cabundle0\") pod \"keda-operator-579c8958f7-6l6gj\" (UID: \"2249b1c5-e912-466b-9706-4a93f4729d08\") " pod="openshift-keda/keda-operator-579c8958f7-6l6gj"
Sep 09 20:17:33.389981 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:17:33.389877    2581 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 09 20:17:33.389981 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:17:33.389897    2581 secret.go:281] references non-existent secret key: ca.crt
Sep 09 20:17:33.389981 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:17:33.389907    2581 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 09 20:17:33.389981 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:17:33.389921    2581 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-6l6gj: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 09 20:17:33.389981 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:17:33.389973    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2249b1c5-e912-466b-9706-4a93f4729d08-certificates podName:2249b1c5-e912-466b-9706-4a93f4729d08 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:33.889960384 +0000 UTC m=+395.895137361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/2249b1c5-e912-466b-9706-4a93f4729d08-certificates") pod "keda-operator-579c8958f7-6l6gj" (UID: "2249b1c5-e912-466b-9706-4a93f4729d08") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 09 20:17:33.390474 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:33.390437    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/2249b1c5-e912-466b-9706-4a93f4729d08-cabundle0\") pod \"keda-operator-579c8958f7-6l6gj\" (UID: \"2249b1c5-e912-466b-9706-4a93f4729d08\") " pod="openshift-keda/keda-operator-579c8958f7-6l6gj"
Sep 09 20:17:33.400124 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:33.400101    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4kdh\" (UniqueName: \"kubernetes.io/projected/2249b1c5-e912-466b-9706-4a93f4729d08-kube-api-access-c4kdh\") pod \"keda-operator-579c8958f7-6l6gj\" (UID: \"2249b1c5-e912-466b-9706-4a93f4729d08\") " pod="openshift-keda/keda-operator-579c8958f7-6l6gj"
Sep 09 20:17:33.892934 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:33.892899    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2249b1c5-e912-466b-9706-4a93f4729d08-certificates\") pod \"keda-operator-579c8958f7-6l6gj\" (UID: \"2249b1c5-e912-466b-9706-4a93f4729d08\") " pod="openshift-keda/keda-operator-579c8958f7-6l6gj"
Sep 09 20:17:33.893093 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:17:33.893046    2581 secret.go:281] references non-existent secret key: ca.crt
Sep 09 20:17:33.893093 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:17:33.893065    2581 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 09 20:17:33.893093 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:17:33.893075    2581 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-6l6gj: references non-existent secret key: ca.crt
Sep 09 20:17:33.893197 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:17:33.893125    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2249b1c5-e912-466b-9706-4a93f4729d08-certificates podName:2249b1c5-e912-466b-9706-4a93f4729d08 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:34.893111685 +0000 UTC m=+396.898288658 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/2249b1c5-e912-466b-9706-4a93f4729d08-certificates") pod "keda-operator-579c8958f7-6l6gj" (UID: "2249b1c5-e912-466b-9706-4a93f4729d08") : references non-existent secret key: ca.crt
Sep 09 20:17:34.006651 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:34.006615    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-5lwjt"]
Sep 09 20:17:34.008854 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:34.008834    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-5lwjt"
Sep 09 20:17:34.011275 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:34.011255    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 09 20:17:34.018322 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:34.018293    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-5lwjt"]
Sep 09 20:17:34.095018 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:34.094986    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skgzb\" (UniqueName: \"kubernetes.io/projected/3a302159-35b9-4299-8930-22dd1f144b40-kube-api-access-skgzb\") pod \"keda-admission-6c4d864b86-5lwjt\" (UID: \"3a302159-35b9-4299-8930-22dd1f144b40\") " pod="openshift-keda/keda-admission-6c4d864b86-5lwjt"
Sep 09 20:17:34.095163 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:34.095026    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3a302159-35b9-4299-8930-22dd1f144b40-certificates\") pod \"keda-admission-6c4d864b86-5lwjt\" (UID: \"3a302159-35b9-4299-8930-22dd1f144b40\") " pod="openshift-keda/keda-admission-6c4d864b86-5lwjt"
Sep 09 20:17:34.195951 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:34.195876    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skgzb\" (UniqueName: \"kubernetes.io/projected/3a302159-35b9-4299-8930-22dd1f144b40-kube-api-access-skgzb\") pod \"keda-admission-6c4d864b86-5lwjt\" (UID: \"3a302159-35b9-4299-8930-22dd1f144b40\") " pod="openshift-keda/keda-admission-6c4d864b86-5lwjt"
Sep 09 20:17:34.195951 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:34.195913    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3a302159-35b9-4299-8930-22dd1f144b40-certificates\") pod \"keda-admission-6c4d864b86-5lwjt\" (UID: \"3a302159-35b9-4299-8930-22dd1f144b40\") " pod="openshift-keda/keda-admission-6c4d864b86-5lwjt"
Sep 09 20:17:34.198315 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:34.198297    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3a302159-35b9-4299-8930-22dd1f144b40-certificates\") pod \"keda-admission-6c4d864b86-5lwjt\" (UID: \"3a302159-35b9-4299-8930-22dd1f144b40\") " pod="openshift-keda/keda-admission-6c4d864b86-5lwjt"
Sep 09 20:17:34.206522 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:34.206498    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skgzb\" (UniqueName: \"kubernetes.io/projected/3a302159-35b9-4299-8930-22dd1f144b40-kube-api-access-skgzb\") pod \"keda-admission-6c4d864b86-5lwjt\" (UID: \"3a302159-35b9-4299-8930-22dd1f144b40\") " pod="openshift-keda/keda-admission-6c4d864b86-5lwjt"
Sep 09 20:17:34.319786 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:34.319751    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-5lwjt"
Sep 09 20:17:34.588251 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:34.588224    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-5lwjt"]
Sep 09 20:17:34.590836 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:17:34.590806    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a302159_35b9_4299_8930_22dd1f144b40.slice/crio-a8ce87dce71e2e7455b478bdb7afe129d651e45b57e50e84e3e5599d78d4eb34 WatchSource:0}: Error finding container a8ce87dce71e2e7455b478bdb7afe129d651e45b57e50e84e3e5599d78d4eb34: Status 404 returned error can't find the container with id a8ce87dce71e2e7455b478bdb7afe129d651e45b57e50e84e3e5599d78d4eb34
Sep 09 20:17:34.900158 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:34.900074    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2249b1c5-e912-466b-9706-4a93f4729d08-certificates\") pod \"keda-operator-579c8958f7-6l6gj\" (UID: \"2249b1c5-e912-466b-9706-4a93f4729d08\") " pod="openshift-keda/keda-operator-579c8958f7-6l6gj"
Sep 09 20:17:34.902443 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:34.902405    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2249b1c5-e912-466b-9706-4a93f4729d08-certificates\") pod \"keda-operator-579c8958f7-6l6gj\" (UID: \"2249b1c5-e912-466b-9706-4a93f4729d08\") " pod="openshift-keda/keda-operator-579c8958f7-6l6gj"
Sep 09 20:17:35.016683 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:35.016646    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-6l6gj"
Sep 09 20:17:35.191138 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:35.191050    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-5lwjt" event={"ID":"3a302159-35b9-4299-8930-22dd1f144b40","Type":"ContainerStarted","Data":"a8ce87dce71e2e7455b478bdb7afe129d651e45b57e50e84e3e5599d78d4eb34"}
Sep 09 20:17:35.286446 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:35.286378    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-6l6gj"]
Sep 09 20:17:35.288659 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:17:35.288619    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2249b1c5_e912_466b_9706_4a93f4729d08.slice/crio-0e2006d04b54e4f9c56a4fe32953e924c7cc75336b17666489ff5aed2a836914 WatchSource:0}: Error finding container 0e2006d04b54e4f9c56a4fe32953e924c7cc75336b17666489ff5aed2a836914: Status 404 returned error can't find the container with id 0e2006d04b54e4f9c56a4fe32953e924c7cc75336b17666489ff5aed2a836914
Sep 09 20:17:36.196853 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:36.196766    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-6l6gj" event={"ID":"2249b1c5-e912-466b-9706-4a93f4729d08","Type":"ContainerStarted","Data":"0e2006d04b54e4f9c56a4fe32953e924c7cc75336b17666489ff5aed2a836914"}
Sep 09 20:17:36.198545 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:36.198519    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-5lwjt" event={"ID":"3a302159-35b9-4299-8930-22dd1f144b40","Type":"ContainerStarted","Data":"e168b39de72a6a6584213843801f58c3cdde871eb3c09e71e2c25a2b333c52bb"}
Sep 09 20:17:36.198685 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:36.198667    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-5lwjt"
Sep 09 20:17:36.217075 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:36.217027    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-5lwjt" podStartSLOduration=1.961329118 podStartE2EDuration="3.217014066s" totalImagesPullingTime="1.255684948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:36.214335772 +0000 UTC m=+398.219512766" watchObservedRunningTime="2026-09-09 20:17:36.217014066 +0000 UTC m=+398.222191052"
Sep 09 20:17:39.212771 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:39.212738    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-6l6gj" event={"ID":"2249b1c5-e912-466b-9706-4a93f4729d08","Type":"ContainerStarted","Data":"2511f87893b820d36db3feaf05f813ad221163aada462c6baa579b93aa2cf3ff"}
Sep 09 20:17:39.213107 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:39.212881    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-6l6gj"
Sep 09 20:17:39.229882 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:39.229835    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-6l6gj" podStartSLOduration=3.065220949 podStartE2EDuration="6.229821614s" totalImagesPullingTime="3.164600665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:39.228544687 +0000 UTC m=+401.233721683" watchObservedRunningTime="2026-09-09 20:17:39.229821614 +0000 UTC m=+401.234998609"
Sep 09 20:17:54.187608 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:54.187573    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ppbr2"
Sep 09 20:17:57.205806 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:17:57.205777    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-5lwjt"
Sep 09 20:18:00.219194 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:00.219161    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-6l6gj"
Sep 09 20:18:01.251099 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.251046    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56fd7586dc-nmwz6"]
Sep 09 20:18:01.253374 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.253349    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:01.264338 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.264314    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56fd7586dc-nmwz6"]
Sep 09 20:18:01.386247 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.386215    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6494ca8-56c5-478b-9871-edddb8a79b97-trusted-ca-bundle\") pod \"console-56fd7586dc-nmwz6\" (UID: \"b6494ca8-56c5-478b-9871-edddb8a79b97\") " pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:01.386247 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.386250    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6494ca8-56c5-478b-9871-edddb8a79b97-console-config\") pod \"console-56fd7586dc-nmwz6\" (UID: \"b6494ca8-56c5-478b-9871-edddb8a79b97\") " pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:01.386453 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.386271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6494ca8-56c5-478b-9871-edddb8a79b97-console-serving-cert\") pod \"console-56fd7586dc-nmwz6\" (UID: \"b6494ca8-56c5-478b-9871-edddb8a79b97\") " pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:01.386453 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.386301    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfspf\" (UniqueName: \"kubernetes.io/projected/b6494ca8-56c5-478b-9871-edddb8a79b97-kube-api-access-xfspf\") pod \"console-56fd7586dc-nmwz6\" (UID: \"b6494ca8-56c5-478b-9871-edddb8a79b97\") " pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:01.386453 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.386339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6494ca8-56c5-478b-9871-edddb8a79b97-service-ca\") pod \"console-56fd7586dc-nmwz6\" (UID: \"b6494ca8-56c5-478b-9871-edddb8a79b97\") " pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:01.386453 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.386355    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6494ca8-56c5-478b-9871-edddb8a79b97-console-oauth-config\") pod \"console-56fd7586dc-nmwz6\" (UID: \"b6494ca8-56c5-478b-9871-edddb8a79b97\") " pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:01.386453 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.386369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6494ca8-56c5-478b-9871-edddb8a79b97-oauth-serving-cert\") pod \"console-56fd7586dc-nmwz6\" (UID: \"b6494ca8-56c5-478b-9871-edddb8a79b97\") " pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:01.486886 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.486856    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6494ca8-56c5-478b-9871-edddb8a79b97-console-serving-cert\") pod \"console-56fd7586dc-nmwz6\" (UID: \"b6494ca8-56c5-478b-9871-edddb8a79b97\") " pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:01.486971 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.486897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfspf\" (UniqueName: \"kubernetes.io/projected/b6494ca8-56c5-478b-9871-edddb8a79b97-kube-api-access-xfspf\") pod \"console-56fd7586dc-nmwz6\" (UID: \"b6494ca8-56c5-478b-9871-edddb8a79b97\") " pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:01.487175 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.487152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6494ca8-56c5-478b-9871-edddb8a79b97-service-ca\") pod \"console-56fd7586dc-nmwz6\" (UID: \"b6494ca8-56c5-478b-9871-edddb8a79b97\") " pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:01.487224 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.487202    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6494ca8-56c5-478b-9871-edddb8a79b97-console-oauth-config\") pod \"console-56fd7586dc-nmwz6\" (UID: \"b6494ca8-56c5-478b-9871-edddb8a79b97\") " pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:01.487272 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.487226    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6494ca8-56c5-478b-9871-edddb8a79b97-oauth-serving-cert\") pod \"console-56fd7586dc-nmwz6\" (UID: \"b6494ca8-56c5-478b-9871-edddb8a79b97\") " pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:01.487321 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.487298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6494ca8-56c5-478b-9871-edddb8a79b97-trusted-ca-bundle\") pod \"console-56fd7586dc-nmwz6\" (UID: \"b6494ca8-56c5-478b-9871-edddb8a79b97\") " pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:01.487370 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.487325    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6494ca8-56c5-478b-9871-edddb8a79b97-console-config\") pod \"console-56fd7586dc-nmwz6\" (UID: \"b6494ca8-56c5-478b-9871-edddb8a79b97\") " pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:01.487881 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.487857    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6494ca8-56c5-478b-9871-edddb8a79b97-service-ca\") pod \"console-56fd7586dc-nmwz6\" (UID: \"b6494ca8-56c5-478b-9871-edddb8a79b97\") " pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:01.487975 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.487927    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6494ca8-56c5-478b-9871-edddb8a79b97-console-config\") pod \"console-56fd7586dc-nmwz6\" (UID: \"b6494ca8-56c5-478b-9871-edddb8a79b97\") " pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:01.488023 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.488001    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6494ca8-56c5-478b-9871-edddb8a79b97-oauth-serving-cert\") pod \"console-56fd7586dc-nmwz6\" (UID: \"b6494ca8-56c5-478b-9871-edddb8a79b97\") " pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:01.488184 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.488167    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6494ca8-56c5-478b-9871-edddb8a79b97-trusted-ca-bundle\") pod \"console-56fd7586dc-nmwz6\" (UID: \"b6494ca8-56c5-478b-9871-edddb8a79b97\") " pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:01.489951 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.489921    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6494ca8-56c5-478b-9871-edddb8a79b97-console-oauth-config\") pod \"console-56fd7586dc-nmwz6\" (UID: \"b6494ca8-56c5-478b-9871-edddb8a79b97\") " pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:01.490034 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.489929    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6494ca8-56c5-478b-9871-edddb8a79b97-console-serving-cert\") pod \"console-56fd7586dc-nmwz6\" (UID: \"b6494ca8-56c5-478b-9871-edddb8a79b97\") " pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:01.497188 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.497163    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfspf\" (UniqueName: \"kubernetes.io/projected/b6494ca8-56c5-478b-9871-edddb8a79b97-kube-api-access-xfspf\") pod \"console-56fd7586dc-nmwz6\" (UID: \"b6494ca8-56c5-478b-9871-edddb8a79b97\") " pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:01.563605 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.563531    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:01.834470 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:01.834434    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56fd7586dc-nmwz6"]
Sep 09 20:18:01.836385 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:18:01.836361    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6494ca8_56c5_478b_9871_edddb8a79b97.slice/crio-dd9140525ec64d2a199df526a9186d9f70f283970be95479d22735cf3432abe4 WatchSource:0}: Error finding container dd9140525ec64d2a199df526a9186d9f70f283970be95479d22735cf3432abe4: Status 404 returned error can't find the container with id dd9140525ec64d2a199df526a9186d9f70f283970be95479d22735cf3432abe4
Sep 09 20:18:02.296837 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:02.296800    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56fd7586dc-nmwz6" event={"ID":"b6494ca8-56c5-478b-9871-edddb8a79b97","Type":"ContainerStarted","Data":"0d65525fa7ace90dceb0de26722541a2b5d76f5138227715f642b996f98411ef"}
Sep 09 20:18:02.297233 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:02.296851    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56fd7586dc-nmwz6" event={"ID":"b6494ca8-56c5-478b-9871-edddb8a79b97","Type":"ContainerStarted","Data":"dd9140525ec64d2a199df526a9186d9f70f283970be95479d22735cf3432abe4"}
Sep 09 20:18:02.315474 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:02.315407    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56fd7586dc-nmwz6" podStartSLOduration=1.315389057 podStartE2EDuration="1.315389057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:18:02.313927968 +0000 UTC m=+424.319104963" watchObservedRunningTime="2026-09-09 20:18:02.315389057 +0000 UTC m=+424.320566052"
Sep 09 20:18:11.564648 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:11.564612    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:11.565034 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:11.564663    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:11.569270 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:11.569247    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:12.340229 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:12.340199    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56fd7586dc-nmwz6"
Sep 09 20:18:12.386519 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:12.386492    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-678bcff8bd-7dczz"]
Sep 09 20:18:26.598828 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:26.598750    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh"]
Sep 09 20:18:26.600911 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:26.600895    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh"
Sep 09 20:18:26.603269 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:26.603230    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-npsx8\""
Sep 09 20:18:26.603421 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:26.603403    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 20:18:26.604147 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:26.604133    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 20:18:26.610453 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:26.610430    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh"]
Sep 09 20:18:26.677700 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:26.677675    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7078f03-504f-4d30-847a-026995aeed92-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh\" (UID: \"c7078f03-504f-4d30-847a-026995aeed92\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh"
Sep 09 20:18:26.677846 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:26.677780    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7078f03-504f-4d30-847a-026995aeed92-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh\" (UID: \"c7078f03-504f-4d30-847a-026995aeed92\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh"
Sep 09 20:18:26.677846 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:26.677822    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlscg\" (UniqueName: \"kubernetes.io/projected/c7078f03-504f-4d30-847a-026995aeed92-kube-api-access-zlscg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh\" (UID: \"c7078f03-504f-4d30-847a-026995aeed92\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh"
Sep 09 20:18:26.778185 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:26.778147    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7078f03-504f-4d30-847a-026995aeed92-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh\" (UID: \"c7078f03-504f-4d30-847a-026995aeed92\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh"
Sep 09 20:18:26.778350 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:26.778235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7078f03-504f-4d30-847a-026995aeed92-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh\" (UID: \"c7078f03-504f-4d30-847a-026995aeed92\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh"
Sep 09 20:18:26.778350 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:26.778261    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlscg\" (UniqueName: \"kubernetes.io/projected/c7078f03-504f-4d30-847a-026995aeed92-kube-api-access-zlscg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh\" (UID: \"c7078f03-504f-4d30-847a-026995aeed92\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh"
Sep 09 20:18:26.778559 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:26.778538    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7078f03-504f-4d30-847a-026995aeed92-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh\" (UID: \"c7078f03-504f-4d30-847a-026995aeed92\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh"
Sep 09 20:18:26.778628 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:26.778596    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7078f03-504f-4d30-847a-026995aeed92-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh\" (UID: \"c7078f03-504f-4d30-847a-026995aeed92\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh"
Sep 09 20:18:26.786797 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:26.786773    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlscg\" (UniqueName: \"kubernetes.io/projected/c7078f03-504f-4d30-847a-026995aeed92-kube-api-access-zlscg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh\" (UID: \"c7078f03-504f-4d30-847a-026995aeed92\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh"
Sep 09 20:18:26.910338 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:26.910263    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh"
Sep 09 20:18:27.177588 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:27.177495    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh"]
Sep 09 20:18:27.181531 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:18:27.181500    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7078f03_504f_4d30_847a_026995aeed92.slice/crio-798a4342e283431d3c7f5f7a5b8a23571dcd214bdf4ae5b97ca2f00aef54180e WatchSource:0}: Error finding container 798a4342e283431d3c7f5f7a5b8a23571dcd214bdf4ae5b97ca2f00aef54180e: Status 404 returned error can't find the container with id 798a4342e283431d3c7f5f7a5b8a23571dcd214bdf4ae5b97ca2f00aef54180e
Sep 09 20:18:27.395776 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:27.395748    2581 generic.go:356] "Generic (PLEG): container finished" podID="c7078f03-504f-4d30-847a-026995aeed92" containerID="f946825e30e0bd1812691e46331bdce12d50820595b4dc2b151dd42c0367bcf5" exitCode=0
Sep 09 20:18:27.395943 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:27.395835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh" event={"ID":"c7078f03-504f-4d30-847a-026995aeed92","Type":"ContainerDied","Data":"f946825e30e0bd1812691e46331bdce12d50820595b4dc2b151dd42c0367bcf5"}
Sep 09 20:18:27.395943 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:27.395878    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh" event={"ID":"c7078f03-504f-4d30-847a-026995aeed92","Type":"ContainerStarted","Data":"798a4342e283431d3c7f5f7a5b8a23571dcd214bdf4ae5b97ca2f00aef54180e"}
Sep 09 20:18:37.411752 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.411656    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-678bcff8bd-7dczz" podUID="38fd331d-779a-4ac5-980d-1aaa79599361" containerName="console" containerID="cri-o://bebb275e06dc9b584bf057543a87cad3bfee02ad0fef6ca492bbfffe4aae9105" gracePeriod=15
Sep 09 20:18:37.441608 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.441580    2581 generic.go:356] "Generic (PLEG): container finished" podID="c7078f03-504f-4d30-847a-026995aeed92" containerID="c99915786039f7f2b28207339740410528a067deeb2a9aced68dec4ce6ef2749" exitCode=0
Sep 09 20:18:37.441703 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.441636    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh" event={"ID":"c7078f03-504f-4d30-847a-026995aeed92","Type":"ContainerDied","Data":"c99915786039f7f2b28207339740410528a067deeb2a9aced68dec4ce6ef2749"}
Sep 09 20:18:37.645539 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.645515    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:18:37.763149 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.763119    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmsbh\" (UniqueName: \"kubernetes.io/projected/38fd331d-779a-4ac5-980d-1aaa79599361-kube-api-access-vmsbh\") pod \"38fd331d-779a-4ac5-980d-1aaa79599361\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") "
Sep 09 20:18:37.763149 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.763154    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38fd331d-779a-4ac5-980d-1aaa79599361-oauth-serving-cert\") pod \"38fd331d-779a-4ac5-980d-1aaa79599361\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") "
Sep 09 20:18:37.763338 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.763175    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38fd331d-779a-4ac5-980d-1aaa79599361-service-ca\") pod \"38fd331d-779a-4ac5-980d-1aaa79599361\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") "
Sep 09 20:18:37.763338 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.763212    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38fd331d-779a-4ac5-980d-1aaa79599361-console-config\") pod \"38fd331d-779a-4ac5-980d-1aaa79599361\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") "
Sep 09 20:18:37.763338 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.763238    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38fd331d-779a-4ac5-980d-1aaa79599361-console-oauth-config\") pod \"38fd331d-779a-4ac5-980d-1aaa79599361\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") "
Sep 09 20:18:37.763338 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.763263    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38fd331d-779a-4ac5-980d-1aaa79599361-console-serving-cert\") pod \"38fd331d-779a-4ac5-980d-1aaa79599361\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") "
Sep 09 20:18:37.763338 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.763312    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38fd331d-779a-4ac5-980d-1aaa79599361-trusted-ca-bundle\") pod \"38fd331d-779a-4ac5-980d-1aaa79599361\" (UID: \"38fd331d-779a-4ac5-980d-1aaa79599361\") "
Sep 09 20:18:37.763612 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.763570    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38fd331d-779a-4ac5-980d-1aaa79599361-service-ca" (OuterVolumeSpecName: "service-ca") pod "38fd331d-779a-4ac5-980d-1aaa79599361" (UID: "38fd331d-779a-4ac5-980d-1aaa79599361"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:37.764074 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.763657    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38fd331d-779a-4ac5-980d-1aaa79599361-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "38fd331d-779a-4ac5-980d-1aaa79599361" (UID: "38fd331d-779a-4ac5-980d-1aaa79599361"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:37.765788 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.764227    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38fd331d-779a-4ac5-980d-1aaa79599361-console-config" (OuterVolumeSpecName: "console-config") pod "38fd331d-779a-4ac5-980d-1aaa79599361" (UID: "38fd331d-779a-4ac5-980d-1aaa79599361"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:37.765788 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.764318    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38fd331d-779a-4ac5-980d-1aaa79599361-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "38fd331d-779a-4ac5-980d-1aaa79599361" (UID: "38fd331d-779a-4ac5-980d-1aaa79599361"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:37.769409 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.766048    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38fd331d-779a-4ac5-980d-1aaa79599361-kube-api-access-vmsbh" (OuterVolumeSpecName: "kube-api-access-vmsbh") pod "38fd331d-779a-4ac5-980d-1aaa79599361" (UID: "38fd331d-779a-4ac5-980d-1aaa79599361"). InnerVolumeSpecName "kube-api-access-vmsbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:18:37.769409 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.766286    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38fd331d-779a-4ac5-980d-1aaa79599361-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "38fd331d-779a-4ac5-980d-1aaa79599361" (UID: "38fd331d-779a-4ac5-980d-1aaa79599361"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:18:37.769723 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.769699    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38fd331d-779a-4ac5-980d-1aaa79599361-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "38fd331d-779a-4ac5-980d-1aaa79599361" (UID: "38fd331d-779a-4ac5-980d-1aaa79599361"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:18:37.864476 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.864433    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmsbh\" (UniqueName: \"kubernetes.io/projected/38fd331d-779a-4ac5-980d-1aaa79599361-kube-api-access-vmsbh\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:18:37.864476 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.864478    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38fd331d-779a-4ac5-980d-1aaa79599361-oauth-serving-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:18:37.864667 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.864494    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38fd331d-779a-4ac5-980d-1aaa79599361-service-ca\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:18:37.864667 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.864507    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38fd331d-779a-4ac5-980d-1aaa79599361-console-config\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:18:37.864667 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.864517    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38fd331d-779a-4ac5-980d-1aaa79599361-console-oauth-config\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:18:37.864667 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.864529    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38fd331d-779a-4ac5-980d-1aaa79599361-console-serving-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:18:37.864667 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:37.864540    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38fd331d-779a-4ac5-980d-1aaa79599361-trusted-ca-bundle\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:18:38.447582 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:38.447499    2581 generic.go:356] "Generic (PLEG): container finished" podID="38fd331d-779a-4ac5-980d-1aaa79599361" containerID="bebb275e06dc9b584bf057543a87cad3bfee02ad0fef6ca492bbfffe4aae9105" exitCode=0
Sep 09 20:18:38.447582 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:38.447555    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-678bcff8bd-7dczz"
Sep 09 20:18:38.448049 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:38.447582    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678bcff8bd-7dczz" event={"ID":"38fd331d-779a-4ac5-980d-1aaa79599361","Type":"ContainerDied","Data":"bebb275e06dc9b584bf057543a87cad3bfee02ad0fef6ca492bbfffe4aae9105"}
Sep 09 20:18:38.448049 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:38.447663    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-678bcff8bd-7dczz" event={"ID":"38fd331d-779a-4ac5-980d-1aaa79599361","Type":"ContainerDied","Data":"024b58ff508f07967ed8e33fe6483ecfdc798472a50093b02b609087325e27e0"}
Sep 09 20:18:38.448049 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:38.447689    2581 scope.go:117] "RemoveContainer" containerID="bebb275e06dc9b584bf057543a87cad3bfee02ad0fef6ca492bbfffe4aae9105"
Sep 09 20:18:38.449812 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:38.449789    2581 generic.go:356] "Generic (PLEG): container finished" podID="c7078f03-504f-4d30-847a-026995aeed92" containerID="9cfe966b12ced6449a2866f230f040e04ea4fe7952c80a711529be6cdc33031c" exitCode=0
Sep 09 20:18:38.449920 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:38.449847    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh" event={"ID":"c7078f03-504f-4d30-847a-026995aeed92","Type":"ContainerDied","Data":"9cfe966b12ced6449a2866f230f040e04ea4fe7952c80a711529be6cdc33031c"}
Sep 09 20:18:38.457747 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:38.457730    2581 scope.go:117] "RemoveContainer" containerID="bebb275e06dc9b584bf057543a87cad3bfee02ad0fef6ca492bbfffe4aae9105"
Sep 09 20:18:38.458007 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:18:38.457992    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bebb275e06dc9b584bf057543a87cad3bfee02ad0fef6ca492bbfffe4aae9105\": container with ID starting with bebb275e06dc9b584bf057543a87cad3bfee02ad0fef6ca492bbfffe4aae9105 not found: ID does not exist" containerID="bebb275e06dc9b584bf057543a87cad3bfee02ad0fef6ca492bbfffe4aae9105"
Sep 09 20:18:38.458048 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:38.458014    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bebb275e06dc9b584bf057543a87cad3bfee02ad0fef6ca492bbfffe4aae9105"} err="failed to get container status \"bebb275e06dc9b584bf057543a87cad3bfee02ad0fef6ca492bbfffe4aae9105\": rpc error: code = NotFound desc = could not find container \"bebb275e06dc9b584bf057543a87cad3bfee02ad0fef6ca492bbfffe4aae9105\": container with ID starting with bebb275e06dc9b584bf057543a87cad3bfee02ad0fef6ca492bbfffe4aae9105 not found: ID does not exist"
Sep 09 20:18:38.480600 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:38.480578    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-678bcff8bd-7dczz"]
Sep 09 20:18:38.485011 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:38.484992    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-678bcff8bd-7dczz"]
Sep 09 20:18:38.634364 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:38.634331    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38fd331d-779a-4ac5-980d-1aaa79599361" path="/var/lib/kubelet/pods/38fd331d-779a-4ac5-980d-1aaa79599361/volumes"
Sep 09 20:18:39.572678 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:39.572658    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh"
Sep 09 20:18:39.679435 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:39.679400    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlscg\" (UniqueName: \"kubernetes.io/projected/c7078f03-504f-4d30-847a-026995aeed92-kube-api-access-zlscg\") pod \"c7078f03-504f-4d30-847a-026995aeed92\" (UID: \"c7078f03-504f-4d30-847a-026995aeed92\") "
Sep 09 20:18:39.679610 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:39.679450    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7078f03-504f-4d30-847a-026995aeed92-util\") pod \"c7078f03-504f-4d30-847a-026995aeed92\" (UID: \"c7078f03-504f-4d30-847a-026995aeed92\") "
Sep 09 20:18:39.679610 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:39.679525    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7078f03-504f-4d30-847a-026995aeed92-bundle\") pod \"c7078f03-504f-4d30-847a-026995aeed92\" (UID: \"c7078f03-504f-4d30-847a-026995aeed92\") "
Sep 09 20:18:39.680391 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:39.680359    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7078f03-504f-4d30-847a-026995aeed92-bundle" (OuterVolumeSpecName: "bundle") pod "c7078f03-504f-4d30-847a-026995aeed92" (UID: "c7078f03-504f-4d30-847a-026995aeed92"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:18:39.681575 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:39.681550    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7078f03-504f-4d30-847a-026995aeed92-kube-api-access-zlscg" (OuterVolumeSpecName: "kube-api-access-zlscg") pod "c7078f03-504f-4d30-847a-026995aeed92" (UID: "c7078f03-504f-4d30-847a-026995aeed92"). InnerVolumeSpecName "kube-api-access-zlscg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:18:39.684305 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:39.684284    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7078f03-504f-4d30-847a-026995aeed92-util" (OuterVolumeSpecName: "util") pod "c7078f03-504f-4d30-847a-026995aeed92" (UID: "c7078f03-504f-4d30-847a-026995aeed92"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:18:39.780304 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:39.780279    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlscg\" (UniqueName: \"kubernetes.io/projected/c7078f03-504f-4d30-847a-026995aeed92-kube-api-access-zlscg\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:18:39.780304 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:39.780300    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7078f03-504f-4d30-847a-026995aeed92-util\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:18:39.780453 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:39.780311    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7078f03-504f-4d30-847a-026995aeed92-bundle\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:18:40.459888 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:40.459857    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh" event={"ID":"c7078f03-504f-4d30-847a-026995aeed92","Type":"ContainerDied","Data":"798a4342e283431d3c7f5f7a5b8a23571dcd214bdf4ae5b97ca2f00aef54180e"}
Sep 09 20:18:40.460046 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:40.459903    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="798a4342e283431d3c7f5f7a5b8a23571dcd214bdf4ae5b97ca2f00aef54180e"
Sep 09 20:18:40.460046 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:40.459863    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s4nxh"
Sep 09 20:18:53.763283 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.763247    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p"]
Sep 09 20:18:53.763660 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.763533    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38fd331d-779a-4ac5-980d-1aaa79599361" containerName="console"
Sep 09 20:18:53.763660 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.763546    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="38fd331d-779a-4ac5-980d-1aaa79599361" containerName="console"
Sep 09 20:18:53.763660 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.763560    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7078f03-504f-4d30-847a-026995aeed92" containerName="util"
Sep 09 20:18:53.763660 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.763565    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7078f03-504f-4d30-847a-026995aeed92" containerName="util"
Sep 09 20:18:53.763660 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.763577    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7078f03-504f-4d30-847a-026995aeed92" containerName="pull"
Sep 09 20:18:53.763660 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.763585    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7078f03-504f-4d30-847a-026995aeed92" containerName="pull"
Sep 09 20:18:53.763660 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.763594    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7078f03-504f-4d30-847a-026995aeed92" containerName="extract"
Sep 09 20:18:53.763660 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.763602    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7078f03-504f-4d30-847a-026995aeed92" containerName="extract"
Sep 09 20:18:53.763660 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.763643    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7078f03-504f-4d30-847a-026995aeed92" containerName="extract"
Sep 09 20:18:53.763660 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.763650    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="38fd331d-779a-4ac5-980d-1aaa79599361" containerName="console"
Sep 09 20:18:53.766522 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.766506    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p"
Sep 09 20:18:53.769083 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.769059    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 20:18:53.769927 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.769911    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-npsx8\""
Sep 09 20:18:53.769979 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.769961    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 20:18:53.777227 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.777205    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p"]
Sep 09 20:18:53.878482 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.878435    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fde1518-e687-4b30-817f-2d0d59a75ce4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p\" (UID: \"5fde1518-e687-4b30-817f-2d0d59a75ce4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p"
Sep 09 20:18:53.878644 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.878503    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2ffx\" (UniqueName: \"kubernetes.io/projected/5fde1518-e687-4b30-817f-2d0d59a75ce4-kube-api-access-h2ffx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p\" (UID: \"5fde1518-e687-4b30-817f-2d0d59a75ce4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p"
Sep 09 20:18:53.878644 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.878558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fde1518-e687-4b30-817f-2d0d59a75ce4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p\" (UID: \"5fde1518-e687-4b30-817f-2d0d59a75ce4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p"
Sep 09 20:18:53.979278 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.979238    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fde1518-e687-4b30-817f-2d0d59a75ce4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p\" (UID: \"5fde1518-e687-4b30-817f-2d0d59a75ce4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p"
Sep 09 20:18:53.979278 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.979285    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2ffx\" (UniqueName: \"kubernetes.io/projected/5fde1518-e687-4b30-817f-2d0d59a75ce4-kube-api-access-h2ffx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p\" (UID: \"5fde1518-e687-4b30-817f-2d0d59a75ce4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p"
Sep 09 20:18:53.979489 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.979327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fde1518-e687-4b30-817f-2d0d59a75ce4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p\" (UID: \"5fde1518-e687-4b30-817f-2d0d59a75ce4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p"
Sep 09 20:18:53.979688 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.979673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fde1518-e687-4b30-817f-2d0d59a75ce4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p\" (UID: \"5fde1518-e687-4b30-817f-2d0d59a75ce4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p"
Sep 09 20:18:53.979751 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.979729    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fde1518-e687-4b30-817f-2d0d59a75ce4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p\" (UID: \"5fde1518-e687-4b30-817f-2d0d59a75ce4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p"
Sep 09 20:18:53.990111 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:53.990087    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2ffx\" (UniqueName: \"kubernetes.io/projected/5fde1518-e687-4b30-817f-2d0d59a75ce4-kube-api-access-h2ffx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p\" (UID: \"5fde1518-e687-4b30-817f-2d0d59a75ce4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p"
Sep 09 20:18:54.075009 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:54.074928    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p"
Sep 09 20:18:54.344873 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:54.344843    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p"]
Sep 09 20:18:54.347352 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:18:54.347324    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fde1518_e687_4b30_817f_2d0d59a75ce4.slice/crio-94354b2844fb6bfddaba95930690172cb523958dbe6c1c1ecc26b70231627ccf WatchSource:0}: Error finding container 94354b2844fb6bfddaba95930690172cb523958dbe6c1c1ecc26b70231627ccf: Status 404 returned error can't find the container with id 94354b2844fb6bfddaba95930690172cb523958dbe6c1c1ecc26b70231627ccf
Sep 09 20:18:54.522774 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:54.522743    2581 generic.go:356] "Generic (PLEG): container finished" podID="5fde1518-e687-4b30-817f-2d0d59a75ce4" containerID="8d0350df4e847531adeec590575bfff1941ee78b141cf8a5e07720354c8a494b" exitCode=0
Sep 09 20:18:54.522909 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:54.522802    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p" event={"ID":"5fde1518-e687-4b30-817f-2d0d59a75ce4","Type":"ContainerDied","Data":"8d0350df4e847531adeec590575bfff1941ee78b141cf8a5e07720354c8a494b"}
Sep 09 20:18:54.522909 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:54.522840    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p" event={"ID":"5fde1518-e687-4b30-817f-2d0d59a75ce4","Type":"ContainerStarted","Data":"94354b2844fb6bfddaba95930690172cb523958dbe6c1c1ecc26b70231627ccf"}
Sep 09 20:18:59.545276 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:59.545241    2581 generic.go:356] "Generic (PLEG): container finished" podID="5fde1518-e687-4b30-817f-2d0d59a75ce4" containerID="79eaf6fa4195a56beabc6f74b343e2854ec6ad971bd303ceb33cba3d40b619ee" exitCode=0
Sep 09 20:18:59.545696 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:18:59.545325    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p" event={"ID":"5fde1518-e687-4b30-817f-2d0d59a75ce4","Type":"ContainerDied","Data":"79eaf6fa4195a56beabc6f74b343e2854ec6ad971bd303ceb33cba3d40b619ee"}
Sep 09 20:19:00.551647 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:00.551613    2581 generic.go:356] "Generic (PLEG): container finished" podID="5fde1518-e687-4b30-817f-2d0d59a75ce4" containerID="1f80f17c3de98fac38fa551d2058abe3114b37c1e587faeee79e8147df548181" exitCode=0
Sep 09 20:19:00.552027 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:00.551697    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p" event={"ID":"5fde1518-e687-4b30-817f-2d0d59a75ce4","Type":"ContainerDied","Data":"1f80f17c3de98fac38fa551d2058abe3114b37c1e587faeee79e8147df548181"}
Sep 09 20:19:01.678624 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:01.678597    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p"
Sep 09 20:19:01.736602 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:01.736564    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2ffx\" (UniqueName: \"kubernetes.io/projected/5fde1518-e687-4b30-817f-2d0d59a75ce4-kube-api-access-h2ffx\") pod \"5fde1518-e687-4b30-817f-2d0d59a75ce4\" (UID: \"5fde1518-e687-4b30-817f-2d0d59a75ce4\") "
Sep 09 20:19:01.736602 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:01.736609    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fde1518-e687-4b30-817f-2d0d59a75ce4-bundle\") pod \"5fde1518-e687-4b30-817f-2d0d59a75ce4\" (UID: \"5fde1518-e687-4b30-817f-2d0d59a75ce4\") "
Sep 09 20:19:01.736806 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:01.736691    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fde1518-e687-4b30-817f-2d0d59a75ce4-util\") pod \"5fde1518-e687-4b30-817f-2d0d59a75ce4\" (UID: \"5fde1518-e687-4b30-817f-2d0d59a75ce4\") "
Sep 09 20:19:01.737098 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:01.737065    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fde1518-e687-4b30-817f-2d0d59a75ce4-bundle" (OuterVolumeSpecName: "bundle") pod "5fde1518-e687-4b30-817f-2d0d59a75ce4" (UID: "5fde1518-e687-4b30-817f-2d0d59a75ce4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:19:01.739155 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:01.739132    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fde1518-e687-4b30-817f-2d0d59a75ce4-kube-api-access-h2ffx" (OuterVolumeSpecName: "kube-api-access-h2ffx") pod "5fde1518-e687-4b30-817f-2d0d59a75ce4" (UID: "5fde1518-e687-4b30-817f-2d0d59a75ce4"). InnerVolumeSpecName "kube-api-access-h2ffx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:19:01.742237 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:01.742213    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fde1518-e687-4b30-817f-2d0d59a75ce4-util" (OuterVolumeSpecName: "util") pod "5fde1518-e687-4b30-817f-2d0d59a75ce4" (UID: "5fde1518-e687-4b30-817f-2d0d59a75ce4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:19:01.837808 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:01.837721    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fde1518-e687-4b30-817f-2d0d59a75ce4-util\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:19:01.837808 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:01.837753    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2ffx\" (UniqueName: \"kubernetes.io/projected/5fde1518-e687-4b30-817f-2d0d59a75ce4-kube-api-access-h2ffx\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:19:01.837808 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:01.837767    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fde1518-e687-4b30-817f-2d0d59a75ce4-bundle\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:19:02.561404 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:02.561373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p" event={"ID":"5fde1518-e687-4b30-817f-2d0d59a75ce4","Type":"ContainerDied","Data":"94354b2844fb6bfddaba95930690172cb523958dbe6c1c1ecc26b70231627ccf"}
Sep 09 20:19:02.561601 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:02.561421    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftw66p"
Sep 09 20:19:02.561601 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:02.561423    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94354b2844fb6bfddaba95930690172cb523958dbe6c1c1ecc26b70231627ccf"
Sep 09 20:19:26.280862 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.280825    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"]
Sep 09 20:19:26.281300 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.281083    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fde1518-e687-4b30-817f-2d0d59a75ce4" containerName="pull"
Sep 09 20:19:26.281300 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.281094    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fde1518-e687-4b30-817f-2d0d59a75ce4" containerName="pull"
Sep 09 20:19:26.281300 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.281105    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fde1518-e687-4b30-817f-2d0d59a75ce4" containerName="extract"
Sep 09 20:19:26.281300 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.281111    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fde1518-e687-4b30-817f-2d0d59a75ce4" containerName="extract"
Sep 09 20:19:26.281300 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.281124    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fde1518-e687-4b30-817f-2d0d59a75ce4" containerName="util"
Sep 09 20:19:26.281300 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.281129    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fde1518-e687-4b30-817f-2d0d59a75ce4" containerName="util"
Sep 09 20:19:26.281300 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.281171    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fde1518-e687-4b30-817f-2d0d59a75ce4" containerName="extract"
Sep 09 20:19:26.283986 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.283968    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:26.286549 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.286521    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 20:19:26.286549 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.286534    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 09 20:19:26.286753 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.286537    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 09 20:19:26.286753 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.286581    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-f8fcf\""
Sep 09 20:19:26.286753 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.286652    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 20:19:26.286753 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.286652    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 20:19:26.286926 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.286884    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 09 20:19:26.295121 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.295101    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"]
Sep 09 20:19:26.314452 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.314426    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpcqt\" (UniqueName: \"kubernetes.io/projected/3bc5f46a-e750-4641-81cf-2808d7fe37a3-kube-api-access-rpcqt\") pod \"istiod-openshift-gateway-7b7995d5cd-mjfrf\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:26.314579 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.314476    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3bc5f46a-e750-4641-81cf-2808d7fe37a3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-mjfrf\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:26.314579 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.314517    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3bc5f46a-e750-4641-81cf-2808d7fe37a3-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-mjfrf\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:26.314579 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.314557    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc5f46a-e750-4641-81cf-2808d7fe37a3-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-mjfrf\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:26.314683 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.314604    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3bc5f46a-e750-4641-81cf-2808d7fe37a3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-mjfrf\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:26.314683 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.314655    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc5f46a-e750-4641-81cf-2808d7fe37a3-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-mjfrf\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:26.314745 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.314687    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3bc5f46a-e750-4641-81cf-2808d7fe37a3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-mjfrf\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:26.415502 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.415473    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc5f46a-e750-4641-81cf-2808d7fe37a3-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-mjfrf\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:26.415619 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.415509    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3bc5f46a-e750-4641-81cf-2808d7fe37a3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-mjfrf\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:26.415619 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.415530    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpcqt\" (UniqueName: \"kubernetes.io/projected/3bc5f46a-e750-4641-81cf-2808d7fe37a3-kube-api-access-rpcqt\") pod \"istiod-openshift-gateway-7b7995d5cd-mjfrf\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:26.415619 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.415554    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3bc5f46a-e750-4641-81cf-2808d7fe37a3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-mjfrf\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:26.415619 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.415574    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3bc5f46a-e750-4641-81cf-2808d7fe37a3-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-mjfrf\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:26.415619 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.415604    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc5f46a-e750-4641-81cf-2808d7fe37a3-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-mjfrf\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:26.415839 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.415757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3bc5f46a-e750-4641-81cf-2808d7fe37a3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-mjfrf\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:26.416321 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.416292    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3bc5f46a-e750-4641-81cf-2808d7fe37a3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-mjfrf\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:26.417961 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.417941    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3bc5f46a-e750-4641-81cf-2808d7fe37a3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-mjfrf\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:26.418050 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.417995    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc5f46a-e750-4641-81cf-2808d7fe37a3-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-mjfrf\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:26.418050 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.418011    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3bc5f46a-e750-4641-81cf-2808d7fe37a3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-mjfrf\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:26.418050 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.418020    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3bc5f46a-e750-4641-81cf-2808d7fe37a3-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-mjfrf\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:26.424884 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.424860    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc5f46a-e750-4641-81cf-2808d7fe37a3-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-mjfrf\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:26.425566 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.425548    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpcqt\" (UniqueName: \"kubernetes.io/projected/3bc5f46a-e750-4641-81cf-2808d7fe37a3-kube-api-access-rpcqt\") pod \"istiod-openshift-gateway-7b7995d5cd-mjfrf\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:26.593518 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.593411    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:26.866934 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:26.866905    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"]
Sep 09 20:19:26.869252 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:19:26.869224    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc5f46a_e750_4641_81cf_2808d7fe37a3.slice/crio-c10dca4933feb871a6ae58f315f7481392165157b2510919bcba088591f61999 WatchSource:0}: Error finding container c10dca4933feb871a6ae58f315f7481392165157b2510919bcba088591f61999: Status 404 returned error can't find the container with id c10dca4933feb871a6ae58f315f7481392165157b2510919bcba088591f61999
Sep 09 20:19:27.661581 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:27.661544    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf" event={"ID":"3bc5f46a-e750-4641-81cf-2808d7fe37a3","Type":"ContainerStarted","Data":"c10dca4933feb871a6ae58f315f7481392165157b2510919bcba088591f61999"}
Sep 09 20:19:29.635220 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:29.635190    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:19:29.635482 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:29.635251    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:19:30.675792 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:30.675758    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf" event={"ID":"3bc5f46a-e750-4641-81cf-2808d7fe37a3","Type":"ContainerStarted","Data":"919a4964d55f4648b5f92ffd3fd6dbd54d5d34a67e10dc3cd4e336e8706c86fb"}
Sep 09 20:19:30.676143 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:30.675962    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:30.677503 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:30.677477    2581 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-mjfrf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 20:19:30.677626 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:30.677521    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf" podUID="3bc5f46a-e750-4641-81cf-2808d7fe37a3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:19:30.696230 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:30.696189    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf" podStartSLOduration=1.9323109889999999 podStartE2EDuration="4.696174578s" totalImagesPullingTime="2.763863589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:19:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:19:30.696146176 +0000 UTC m=+512.701323184" watchObservedRunningTime="2026-09-09 20:19:30.696174578 +0000 UTC m=+512.701351573"
Sep 09 20:19:31.681099 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:31.681073    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:19:33.031567 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.031538    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"]
Sep 09 20:19:33.036516 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.036494    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.039022 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.039003    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-zwsqk\""
Sep 09 20:19:33.047764 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.047743    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"]
Sep 09 20:19:33.170847 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.170806    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e089356-1131-499c-a8e5-afe9c2c9b0c6-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.171040 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.170851    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snk2n\" (UniqueName: \"kubernetes.io/projected/4e089356-1131-499c-a8e5-afe9c2c9b0c6-kube-api-access-snk2n\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.171040 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.170879    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e089356-1131-499c-a8e5-afe9c2c9b0c6-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.171040 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.170900    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e089356-1131-499c-a8e5-afe9c2c9b0c6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.171040 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.170940    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e089356-1131-499c-a8e5-afe9c2c9b0c6-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.171040 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.171000    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e089356-1131-499c-a8e5-afe9c2c9b0c6-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.171040 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.171029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e089356-1131-499c-a8e5-afe9c2c9b0c6-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.171252 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.171053    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e089356-1131-499c-a8e5-afe9c2c9b0c6-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.171252 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.171122    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e089356-1131-499c-a8e5-afe9c2c9b0c6-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.272151 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.272108    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e089356-1131-499c-a8e5-afe9c2c9b0c6-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.272151 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.272151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e089356-1131-499c-a8e5-afe9c2c9b0c6-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.272393 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.272168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e089356-1131-499c-a8e5-afe9c2c9b0c6-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.272393 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.272195    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e089356-1131-499c-a8e5-afe9c2c9b0c6-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.272393 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.272242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e089356-1131-499c-a8e5-afe9c2c9b0c6-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.272393 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.272271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snk2n\" (UniqueName: \"kubernetes.io/projected/4e089356-1131-499c-a8e5-afe9c2c9b0c6-kube-api-access-snk2n\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.272393 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.272363    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e089356-1131-499c-a8e5-afe9c2c9b0c6-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.272668 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.272412    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e089356-1131-499c-a8e5-afe9c2c9b0c6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.272668 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.272483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e089356-1131-499c-a8e5-afe9c2c9b0c6-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.272668 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.272552    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e089356-1131-499c-a8e5-afe9c2c9b0c6-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.272961 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.272939    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e089356-1131-499c-a8e5-afe9c2c9b0c6-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.273039 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.272963    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e089356-1131-499c-a8e5-afe9c2c9b0c6-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.273076 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.273040    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e089356-1131-499c-a8e5-afe9c2c9b0c6-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.273076 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.273058    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e089356-1131-499c-a8e5-afe9c2c9b0c6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.274710 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.274687    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e089356-1131-499c-a8e5-afe9c2c9b0c6-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.274811 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.274798    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e089356-1131-499c-a8e5-afe9c2c9b0c6-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.281214 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.281190    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e089356-1131-499c-a8e5-afe9c2c9b0c6-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.281343 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.281327    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snk2n\" (UniqueName: \"kubernetes.io/projected/4e089356-1131-499c-a8e5-afe9c2c9b0c6-kube-api-access-snk2n\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4\" (UID: \"4e089356-1131-499c-a8e5-afe9c2c9b0c6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.349213 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.349139    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:33.623508 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.623482    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"]
Sep 09 20:19:33.625630 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:19:33.625600    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e089356_1131_499c_a8e5_afe9c2c9b0c6.slice/crio-1aa310bac288a5956d209bc95f9d2a5152fb9b25ad021de8d97bc21624a8a223 WatchSource:0}: Error finding container 1aa310bac288a5956d209bc95f9d2a5152fb9b25ad021de8d97bc21624a8a223: Status 404 returned error can't find the container with id 1aa310bac288a5956d209bc95f9d2a5152fb9b25ad021de8d97bc21624a8a223
Sep 09 20:19:33.691221 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:33.691180    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4" event={"ID":"4e089356-1131-499c-a8e5-afe9c2c9b0c6","Type":"ContainerStarted","Data":"1aa310bac288a5956d209bc95f9d2a5152fb9b25ad021de8d97bc21624a8a223"}
Sep 09 20:19:36.217281 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:36.217250    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:19:36.217548 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:36.217317    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:19:36.217548 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:36.217367    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:19:36.706275 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:36.706241    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4" event={"ID":"4e089356-1131-499c-a8e5-afe9c2c9b0c6","Type":"ContainerStarted","Data":"b4d08c412670293248497e6838e25fead94426d227a4082f8b0534290deb0c79"}
Sep 09 20:19:36.726453 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:36.726399    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4" podStartSLOduration=1.137013658 podStartE2EDuration="3.726383284s" totalImagesPullingTime="2.589369626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:19:36.724601993 +0000 UTC m=+518.729778988" watchObservedRunningTime="2026-09-09 20:19:36.726383284 +0000 UTC m=+518.731560279"
Sep 09 20:19:37.350227 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:37.350189    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:37.354848 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:37.354824    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:37.710019 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:37.709944    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:37.711098 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:37.711079    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:19:41.915607 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:41.915572    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf"]
Sep 09 20:19:41.918849 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:41.918832    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf"
Sep 09 20:19:41.922080 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:41.922057    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 20:19:41.922325 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:41.922311    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 20:19:41.923186 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:41.923173    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-npsx8\""
Sep 09 20:19:41.932969 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:41.932945    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf"]
Sep 09 20:19:42.016488 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.016445    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7"]
Sep 09 20:19:42.019684 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.019668    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7"
Sep 09 20:19:42.034529 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.034499    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7"]
Sep 09 20:19:42.046121 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.046098    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fb7ee8d-dd7a-4507-a16a-dea0b16007d4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf\" (UID: \"2fb7ee8d-dd7a-4507-a16a-dea0b16007d4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf"
Sep 09 20:19:42.046247 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.046137    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fb7ee8d-dd7a-4507-a16a-dea0b16007d4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf\" (UID: \"2fb7ee8d-dd7a-4507-a16a-dea0b16007d4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf"
Sep 09 20:19:42.046247 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.046220    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98pk4\" (UniqueName: \"kubernetes.io/projected/2fb7ee8d-dd7a-4507-a16a-dea0b16007d4-kube-api-access-98pk4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf\" (UID: \"2fb7ee8d-dd7a-4507-a16a-dea0b16007d4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf"
Sep 09 20:19:42.111501 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.111473    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp"]
Sep 09 20:19:42.114790 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.114775    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp"
Sep 09 20:19:42.128127 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.128101    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp"]
Sep 09 20:19:42.147544 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.147523    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98pk4\" (UniqueName: \"kubernetes.io/projected/2fb7ee8d-dd7a-4507-a16a-dea0b16007d4-kube-api-access-98pk4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf\" (UID: \"2fb7ee8d-dd7a-4507-a16a-dea0b16007d4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf"
Sep 09 20:19:42.147634 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.147566    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d51fee9-f492-4995-b40d-fa243258e96a-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7\" (UID: \"2d51fee9-f492-4995-b40d-fa243258e96a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7"
Sep 09 20:19:42.147634 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.147592    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tprr9\" (UniqueName: \"kubernetes.io/projected/2d51fee9-f492-4995-b40d-fa243258e96a-kube-api-access-tprr9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7\" (UID: \"2d51fee9-f492-4995-b40d-fa243258e96a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7"
Sep 09 20:19:42.147708 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.147662    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fb7ee8d-dd7a-4507-a16a-dea0b16007d4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf\" (UID: \"2fb7ee8d-dd7a-4507-a16a-dea0b16007d4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf"
Sep 09 20:19:42.147743 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.147703    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d51fee9-f492-4995-b40d-fa243258e96a-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7\" (UID: \"2d51fee9-f492-4995-b40d-fa243258e96a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7"
Sep 09 20:19:42.147776 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.147745    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fb7ee8d-dd7a-4507-a16a-dea0b16007d4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf\" (UID: \"2fb7ee8d-dd7a-4507-a16a-dea0b16007d4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf"
Sep 09 20:19:42.147968 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.147951    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fb7ee8d-dd7a-4507-a16a-dea0b16007d4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf\" (UID: \"2fb7ee8d-dd7a-4507-a16a-dea0b16007d4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf"
Sep 09 20:19:42.148007 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.147999    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fb7ee8d-dd7a-4507-a16a-dea0b16007d4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf\" (UID: \"2fb7ee8d-dd7a-4507-a16a-dea0b16007d4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf"
Sep 09 20:19:42.159932 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.159902    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98pk4\" (UniqueName: \"kubernetes.io/projected/2fb7ee8d-dd7a-4507-a16a-dea0b16007d4-kube-api-access-98pk4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf\" (UID: \"2fb7ee8d-dd7a-4507-a16a-dea0b16007d4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf"
Sep 09 20:19:42.200173 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.200119    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25"]
Sep 09 20:19:42.203734 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.203720    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25"
Sep 09 20:19:42.216180 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.216157    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25"]
Sep 09 20:19:42.227548 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.227523    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf"
Sep 09 20:19:42.248846 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.248821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d51fee9-f492-4995-b40d-fa243258e96a-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7\" (UID: \"2d51fee9-f492-4995-b40d-fa243258e96a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7"
Sep 09 20:19:42.248972 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.248884    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bab3ff7-f099-41b7-baac-42c013cc42aa-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp\" (UID: \"4bab3ff7-f099-41b7-baac-42c013cc42aa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp"
Sep 09 20:19:42.248972 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.248909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bab3ff7-f099-41b7-baac-42c013cc42aa-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp\" (UID: \"4bab3ff7-f099-41b7-baac-42c013cc42aa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp"
Sep 09 20:19:42.249075 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.249001    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d51fee9-f492-4995-b40d-fa243258e96a-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7\" (UID: \"2d51fee9-f492-4995-b40d-fa243258e96a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7"
Sep 09 20:19:42.249075 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.249047    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tprr9\" (UniqueName: \"kubernetes.io/projected/2d51fee9-f492-4995-b40d-fa243258e96a-kube-api-access-tprr9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7\" (UID: \"2d51fee9-f492-4995-b40d-fa243258e96a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7"
Sep 09 20:19:42.249176 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.249081    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z57qp\" (UniqueName: \"kubernetes.io/projected/4bab3ff7-f099-41b7-baac-42c013cc42aa-kube-api-access-z57qp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp\" (UID: \"4bab3ff7-f099-41b7-baac-42c013cc42aa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp"
Sep 09 20:19:42.249284 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.249261    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d51fee9-f492-4995-b40d-fa243258e96a-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7\" (UID: \"2d51fee9-f492-4995-b40d-fa243258e96a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7"
Sep 09 20:19:42.249365 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.249346    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d51fee9-f492-4995-b40d-fa243258e96a-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7\" (UID: \"2d51fee9-f492-4995-b40d-fa243258e96a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7"
Sep 09 20:19:42.261096 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.261063    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tprr9\" (UniqueName: \"kubernetes.io/projected/2d51fee9-f492-4995-b40d-fa243258e96a-kube-api-access-tprr9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7\" (UID: \"2d51fee9-f492-4995-b40d-fa243258e96a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7"
Sep 09 20:19:42.330994 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.330945    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7"
Sep 09 20:19:42.350276 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.350247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z57qp\" (UniqueName: \"kubernetes.io/projected/4bab3ff7-f099-41b7-baac-42c013cc42aa-kube-api-access-z57qp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp\" (UID: \"4bab3ff7-f099-41b7-baac-42c013cc42aa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp"
Sep 09 20:19:42.350400 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.350324    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bab3ff7-f099-41b7-baac-42c013cc42aa-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp\" (UID: \"4bab3ff7-f099-41b7-baac-42c013cc42aa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp"
Sep 09 20:19:42.350400 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.350349    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bab3ff7-f099-41b7-baac-42c013cc42aa-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp\" (UID: \"4bab3ff7-f099-41b7-baac-42c013cc42aa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp"
Sep 09 20:19:42.350536 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.350397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fdeeba50-d9b9-4558-bbdb-def597b9c2c4-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25\" (UID: \"fdeeba50-d9b9-4558-bbdb-def597b9c2c4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25"
Sep 09 20:19:42.350536 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.350450    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcqv7\" (UniqueName: \"kubernetes.io/projected/fdeeba50-d9b9-4558-bbdb-def597b9c2c4-kube-api-access-qcqv7\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25\" (UID: \"fdeeba50-d9b9-4558-bbdb-def597b9c2c4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25"
Sep 09 20:19:42.350624 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.350572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdeeba50-d9b9-4558-bbdb-def597b9c2c4-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25\" (UID: \"fdeeba50-d9b9-4558-bbdb-def597b9c2c4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25"
Sep 09 20:19:42.350757 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.350741    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bab3ff7-f099-41b7-baac-42c013cc42aa-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp\" (UID: \"4bab3ff7-f099-41b7-baac-42c013cc42aa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp"
Sep 09 20:19:42.350833 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.350811    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bab3ff7-f099-41b7-baac-42c013cc42aa-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp\" (UID: \"4bab3ff7-f099-41b7-baac-42c013cc42aa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp"
Sep 09 20:19:42.359934 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.359911    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z57qp\" (UniqueName: \"kubernetes.io/projected/4bab3ff7-f099-41b7-baac-42c013cc42aa-kube-api-access-z57qp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp\" (UID: \"4bab3ff7-f099-41b7-baac-42c013cc42aa\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp"
Sep 09 20:19:42.423863 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.423831    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp"
Sep 09 20:19:42.451260 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.451187    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fdeeba50-d9b9-4558-bbdb-def597b9c2c4-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25\" (UID: \"fdeeba50-d9b9-4558-bbdb-def597b9c2c4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25"
Sep 09 20:19:42.451260 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.451233    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcqv7\" (UniqueName: \"kubernetes.io/projected/fdeeba50-d9b9-4558-bbdb-def597b9c2c4-kube-api-access-qcqv7\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25\" (UID: \"fdeeba50-d9b9-4558-bbdb-def597b9c2c4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25"
Sep 09 20:19:42.451446 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.451361    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdeeba50-d9b9-4558-bbdb-def597b9c2c4-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25\" (UID: \"fdeeba50-d9b9-4558-bbdb-def597b9c2c4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25"
Sep 09 20:19:42.451608 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.451586    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fdeeba50-d9b9-4558-bbdb-def597b9c2c4-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25\" (UID: \"fdeeba50-d9b9-4558-bbdb-def597b9c2c4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25"
Sep 09 20:19:42.451665 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.451630    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdeeba50-d9b9-4558-bbdb-def597b9c2c4-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25\" (UID: \"fdeeba50-d9b9-4558-bbdb-def597b9c2c4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25"
Sep 09 20:19:42.461091 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.461066    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcqv7\" (UniqueName: \"kubernetes.io/projected/fdeeba50-d9b9-4558-bbdb-def597b9c2c4-kube-api-access-qcqv7\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25\" (UID: \"fdeeba50-d9b9-4558-bbdb-def597b9c2c4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25"
Sep 09 20:19:42.502494 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.502448    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf"]
Sep 09 20:19:42.505451 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:19:42.505422    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb7ee8d_dd7a_4507_a16a_dea0b16007d4.slice/crio-697f40902da2faf96461f26d8f6cb16f15c1f4241e2c0b7815167c9a8932c8ad WatchSource:0}: Error finding container 697f40902da2faf96461f26d8f6cb16f15c1f4241e2c0b7815167c9a8932c8ad: Status 404 returned error can't find the container with id 697f40902da2faf96461f26d8f6cb16f15c1f4241e2c0b7815167c9a8932c8ad
Sep 09 20:19:42.512307 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.512290    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25"
Sep 09 20:19:42.604912 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.604885    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7"]
Sep 09 20:19:42.606705 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:19:42.606674    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d51fee9_f492_4995_b40d_fa243258e96a.slice/crio-73b32a81e41b37f46c31e57e063033e1981a1ddd8325231357d55df53eb11adf WatchSource:0}: Error finding container 73b32a81e41b37f46c31e57e063033e1981a1ddd8325231357d55df53eb11adf: Status 404 returned error can't find the container with id 73b32a81e41b37f46c31e57e063033e1981a1ddd8325231357d55df53eb11adf
Sep 09 20:19:42.698685 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.698623    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp"]
Sep 09 20:19:42.700638 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:19:42.700615    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bab3ff7_f099_41b7_baac_42c013cc42aa.slice/crio-57424c82f8b017f0f4dc6f66d6091d47645e7e361e2986d2312109277c0dabeb WatchSource:0}: Error finding container 57424c82f8b017f0f4dc6f66d6091d47645e7e361e2986d2312109277c0dabeb: Status 404 returned error can't find the container with id 57424c82f8b017f0f4dc6f66d6091d47645e7e361e2986d2312109277c0dabeb
Sep 09 20:19:42.734756 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.734724    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7" event={"ID":"2d51fee9-f492-4995-b40d-fa243258e96a","Type":"ContainerStarted","Data":"9cbc92b21c3e5c45492faf4da65771779a6f848894057e0f50953ee018b44a98"}
Sep 09 20:19:42.734941 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.734782    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7" event={"ID":"2d51fee9-f492-4995-b40d-fa243258e96a","Type":"ContainerStarted","Data":"73b32a81e41b37f46c31e57e063033e1981a1ddd8325231357d55df53eb11adf"}
Sep 09 20:19:42.736579 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.736553    2581 generic.go:356] "Generic (PLEG): container finished" podID="2fb7ee8d-dd7a-4507-a16a-dea0b16007d4" containerID="5b2762d9d1e41ecf7f55b0f71a42a56b9aa7957bdd17fe12ae83c9520f882228" exitCode=0
Sep 09 20:19:42.736696 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.736633    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf" event={"ID":"2fb7ee8d-dd7a-4507-a16a-dea0b16007d4","Type":"ContainerDied","Data":"5b2762d9d1e41ecf7f55b0f71a42a56b9aa7957bdd17fe12ae83c9520f882228"}
Sep 09 20:19:42.736696 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.736681    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf" event={"ID":"2fb7ee8d-dd7a-4507-a16a-dea0b16007d4","Type":"ContainerStarted","Data":"697f40902da2faf96461f26d8f6cb16f15c1f4241e2c0b7815167c9a8932c8ad"}
Sep 09 20:19:42.738219 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.738177    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp" event={"ID":"4bab3ff7-f099-41b7-baac-42c013cc42aa","Type":"ContainerStarted","Data":"57424c82f8b017f0f4dc6f66d6091d47645e7e361e2986d2312109277c0dabeb"}
Sep 09 20:19:42.796982 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:42.796950    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25"]
Sep 09 20:19:42.812330 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:19:42.812295    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdeeba50_d9b9_4558_bbdb_def597b9c2c4.slice/crio-948cbcd3dcc9c0f6ebd10d4ee95f40e34bf56a4e582704176e5850bdfa422af2 WatchSource:0}: Error finding container 948cbcd3dcc9c0f6ebd10d4ee95f40e34bf56a4e582704176e5850bdfa422af2: Status 404 returned error can't find the container with id 948cbcd3dcc9c0f6ebd10d4ee95f40e34bf56a4e582704176e5850bdfa422af2
Sep 09 20:19:43.743722 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:43.743689    2581 generic.go:356] "Generic (PLEG): container finished" podID="4bab3ff7-f099-41b7-baac-42c013cc42aa" containerID="e3ccaba409d0152906f53c1aec033bad35a56ddd9862740261e84005c5389235" exitCode=0
Sep 09 20:19:43.744165 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:43.743773    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp" event={"ID":"4bab3ff7-f099-41b7-baac-42c013cc42aa","Type":"ContainerDied","Data":"e3ccaba409d0152906f53c1aec033bad35a56ddd9862740261e84005c5389235"}
Sep 09 20:19:43.745369 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:43.745350    2581 generic.go:356] "Generic (PLEG): container finished" podID="2d51fee9-f492-4995-b40d-fa243258e96a" containerID="9cbc92b21c3e5c45492faf4da65771779a6f848894057e0f50953ee018b44a98" exitCode=0
Sep 09 20:19:43.745450 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:43.745374    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7" event={"ID":"2d51fee9-f492-4995-b40d-fa243258e96a","Type":"ContainerDied","Data":"9cbc92b21c3e5c45492faf4da65771779a6f848894057e0f50953ee018b44a98"}
Sep 09 20:19:43.746998 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:43.746970    2581 generic.go:356] "Generic (PLEG): container finished" podID="fdeeba50-d9b9-4558-bbdb-def597b9c2c4" containerID="79b5dd0e900fb33d20c155af9527631ede581f871aae3672b9dbae59601e9a8a" exitCode=0
Sep 09 20:19:43.747084 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:43.747025    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25" event={"ID":"fdeeba50-d9b9-4558-bbdb-def597b9c2c4","Type":"ContainerDied","Data":"79b5dd0e900fb33d20c155af9527631ede581f871aae3672b9dbae59601e9a8a"}
Sep 09 20:19:43.747084 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:43.747072    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25" event={"ID":"fdeeba50-d9b9-4558-bbdb-def597b9c2c4","Type":"ContainerStarted","Data":"948cbcd3dcc9c0f6ebd10d4ee95f40e34bf56a4e582704176e5850bdfa422af2"}
Sep 09 20:19:44.754737 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:44.754700    2581 generic.go:356] "Generic (PLEG): container finished" podID="2fb7ee8d-dd7a-4507-a16a-dea0b16007d4" containerID="fb4459dd0657f497653e075397f26765bb2081c8224c8da1ab3dcc8ebfedfb4c" exitCode=0
Sep 09 20:19:44.755079 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:44.754745    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf" event={"ID":"2fb7ee8d-dd7a-4507-a16a-dea0b16007d4","Type":"ContainerDied","Data":"fb4459dd0657f497653e075397f26765bb2081c8224c8da1ab3dcc8ebfedfb4c"}
Sep 09 20:19:45.760574 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:45.760543    2581 generic.go:356] "Generic (PLEG): container finished" podID="4bab3ff7-f099-41b7-baac-42c013cc42aa" containerID="6b596f3e786803f0ce9d933a0a2bd1433c79f1a474d11875208d612643498ab3" exitCode=0
Sep 09 20:19:45.761015 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:45.760635    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp" event={"ID":"4bab3ff7-f099-41b7-baac-42c013cc42aa","Type":"ContainerDied","Data":"6b596f3e786803f0ce9d933a0a2bd1433c79f1a474d11875208d612643498ab3"}
Sep 09 20:19:45.762601 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:45.762575    2581 generic.go:356] "Generic (PLEG): container finished" podID="2d51fee9-f492-4995-b40d-fa243258e96a" containerID="c51f668b91bf5ee31581b8e1d1e753ef0ed1820a40c01b977b00dc76f21bba2c" exitCode=0
Sep 09 20:19:45.762703 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:45.762658    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7" event={"ID":"2d51fee9-f492-4995-b40d-fa243258e96a","Type":"ContainerDied","Data":"c51f668b91bf5ee31581b8e1d1e753ef0ed1820a40c01b977b00dc76f21bba2c"}
Sep 09 20:19:45.764831 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:45.764812    2581 generic.go:356] "Generic (PLEG): container finished" podID="2fb7ee8d-dd7a-4507-a16a-dea0b16007d4" containerID="600fd2c7a316245587c4c6f352c789bb87cd1dd7b65405ca626187db717b3327" exitCode=0
Sep 09 20:19:45.764916 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:45.764844    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf" event={"ID":"2fb7ee8d-dd7a-4507-a16a-dea0b16007d4","Type":"ContainerDied","Data":"600fd2c7a316245587c4c6f352c789bb87cd1dd7b65405ca626187db717b3327"}
Sep 09 20:19:46.771824 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:46.771787    2581 generic.go:356] "Generic (PLEG): container finished" podID="4bab3ff7-f099-41b7-baac-42c013cc42aa" containerID="7af1aa3245688e9b57f7d495879ed0c2739f1a49c7fee4d5a841bc80a21e17ac" exitCode=0
Sep 09 20:19:46.772207 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:46.771866    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp" event={"ID":"4bab3ff7-f099-41b7-baac-42c013cc42aa","Type":"ContainerDied","Data":"7af1aa3245688e9b57f7d495879ed0c2739f1a49c7fee4d5a841bc80a21e17ac"}
Sep 09 20:19:46.773877 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:46.773854    2581 generic.go:356] "Generic (PLEG): container finished" podID="2d51fee9-f492-4995-b40d-fa243258e96a" containerID="316a2b8d1c0ce76c969f9096af6ec17b6daeb718c0a2b0a9568536b70a11416b" exitCode=0
Sep 09 20:19:46.774017 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:46.773936    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7" event={"ID":"2d51fee9-f492-4995-b40d-fa243258e96a","Type":"ContainerDied","Data":"316a2b8d1c0ce76c969f9096af6ec17b6daeb718c0a2b0a9568536b70a11416b"}
Sep 09 20:19:46.898753 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:46.898729    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf"
Sep 09 20:19:46.989418 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:46.989385    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fb7ee8d-dd7a-4507-a16a-dea0b16007d4-bundle\") pod \"2fb7ee8d-dd7a-4507-a16a-dea0b16007d4\" (UID: \"2fb7ee8d-dd7a-4507-a16a-dea0b16007d4\") "
Sep 09 20:19:46.989574 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:46.989435    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fb7ee8d-dd7a-4507-a16a-dea0b16007d4-util\") pod \"2fb7ee8d-dd7a-4507-a16a-dea0b16007d4\" (UID: \"2fb7ee8d-dd7a-4507-a16a-dea0b16007d4\") "
Sep 09 20:19:46.989574 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:46.989491    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98pk4\" (UniqueName: \"kubernetes.io/projected/2fb7ee8d-dd7a-4507-a16a-dea0b16007d4-kube-api-access-98pk4\") pod \"2fb7ee8d-dd7a-4507-a16a-dea0b16007d4\" (UID: \"2fb7ee8d-dd7a-4507-a16a-dea0b16007d4\") "
Sep 09 20:19:46.989915 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:46.989888    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb7ee8d-dd7a-4507-a16a-dea0b16007d4-bundle" (OuterVolumeSpecName: "bundle") pod "2fb7ee8d-dd7a-4507-a16a-dea0b16007d4" (UID: "2fb7ee8d-dd7a-4507-a16a-dea0b16007d4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:19:46.991654 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:46.991622    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb7ee8d-dd7a-4507-a16a-dea0b16007d4-kube-api-access-98pk4" (OuterVolumeSpecName: "kube-api-access-98pk4") pod "2fb7ee8d-dd7a-4507-a16a-dea0b16007d4" (UID: "2fb7ee8d-dd7a-4507-a16a-dea0b16007d4"). InnerVolumeSpecName "kube-api-access-98pk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:19:46.994772 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:46.994737    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb7ee8d-dd7a-4507-a16a-dea0b16007d4-util" (OuterVolumeSpecName: "util") pod "2fb7ee8d-dd7a-4507-a16a-dea0b16007d4" (UID: "2fb7ee8d-dd7a-4507-a16a-dea0b16007d4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:19:47.091010 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:47.090911    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fb7ee8d-dd7a-4507-a16a-dea0b16007d4-util\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:19:47.091010 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:47.090951    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98pk4\" (UniqueName: \"kubernetes.io/projected/2fb7ee8d-dd7a-4507-a16a-dea0b16007d4-kube-api-access-98pk4\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:19:47.091010 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:47.090962    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fb7ee8d-dd7a-4507-a16a-dea0b16007d4-bundle\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:19:47.780874 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:47.780837    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf" event={"ID":"2fb7ee8d-dd7a-4507-a16a-dea0b16007d4","Type":"ContainerDied","Data":"697f40902da2faf96461f26d8f6cb16f15c1f4241e2c0b7815167c9a8932c8ad"}
Sep 09 20:19:47.780874 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:47.780867    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jp5sf"
Sep 09 20:19:47.781370 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:47.780897    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="697f40902da2faf96461f26d8f6cb16f15c1f4241e2c0b7815167c9a8932c8ad"
Sep 09 20:19:47.910774 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:47.910749    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp"
Sep 09 20:19:47.937323 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:47.937297    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7"
Sep 09 20:19:47.997017 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:47.996986    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bab3ff7-f099-41b7-baac-42c013cc42aa-bundle\") pod \"4bab3ff7-f099-41b7-baac-42c013cc42aa\" (UID: \"4bab3ff7-f099-41b7-baac-42c013cc42aa\") "
Sep 09 20:19:47.997171 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:47.997035    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bab3ff7-f099-41b7-baac-42c013cc42aa-util\") pod \"4bab3ff7-f099-41b7-baac-42c013cc42aa\" (UID: \"4bab3ff7-f099-41b7-baac-42c013cc42aa\") "
Sep 09 20:19:47.997171 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:47.997072    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z57qp\" (UniqueName: \"kubernetes.io/projected/4bab3ff7-f099-41b7-baac-42c013cc42aa-kube-api-access-z57qp\") pod \"4bab3ff7-f099-41b7-baac-42c013cc42aa\" (UID: \"4bab3ff7-f099-41b7-baac-42c013cc42aa\") "
Sep 09 20:19:47.997479 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:47.997430    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bab3ff7-f099-41b7-baac-42c013cc42aa-bundle" (OuterVolumeSpecName: "bundle") pod "4bab3ff7-f099-41b7-baac-42c013cc42aa" (UID: "4bab3ff7-f099-41b7-baac-42c013cc42aa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:19:47.999158 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:47.999130    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bab3ff7-f099-41b7-baac-42c013cc42aa-kube-api-access-z57qp" (OuterVolumeSpecName: "kube-api-access-z57qp") pod "4bab3ff7-f099-41b7-baac-42c013cc42aa" (UID: "4bab3ff7-f099-41b7-baac-42c013cc42aa"). InnerVolumeSpecName "kube-api-access-z57qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:19:48.054246 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:48.054163    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bab3ff7-f099-41b7-baac-42c013cc42aa-util" (OuterVolumeSpecName: "util") pod "4bab3ff7-f099-41b7-baac-42c013cc42aa" (UID: "4bab3ff7-f099-41b7-baac-42c013cc42aa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:19:48.098263 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:48.098230    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tprr9\" (UniqueName: \"kubernetes.io/projected/2d51fee9-f492-4995-b40d-fa243258e96a-kube-api-access-tprr9\") pod \"2d51fee9-f492-4995-b40d-fa243258e96a\" (UID: \"2d51fee9-f492-4995-b40d-fa243258e96a\") "
Sep 09 20:19:48.098416 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:48.098280    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d51fee9-f492-4995-b40d-fa243258e96a-util\") pod \"2d51fee9-f492-4995-b40d-fa243258e96a\" (UID: \"2d51fee9-f492-4995-b40d-fa243258e96a\") "
Sep 09 20:19:48.098416 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:48.098345    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d51fee9-f492-4995-b40d-fa243258e96a-bundle\") pod \"2d51fee9-f492-4995-b40d-fa243258e96a\" (UID: \"2d51fee9-f492-4995-b40d-fa243258e96a\") "
Sep 09 20:19:48.098568 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:48.098517    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z57qp\" (UniqueName: \"kubernetes.io/projected/4bab3ff7-f099-41b7-baac-42c013cc42aa-kube-api-access-z57qp\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:19:48.098568 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:48.098530    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4bab3ff7-f099-41b7-baac-42c013cc42aa-bundle\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:19:48.098568 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:48.098541    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4bab3ff7-f099-41b7-baac-42c013cc42aa-util\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:19:48.099062 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:48.099021    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d51fee9-f492-4995-b40d-fa243258e96a-bundle" (OuterVolumeSpecName: "bundle") pod "2d51fee9-f492-4995-b40d-fa243258e96a" (UID: "2d51fee9-f492-4995-b40d-fa243258e96a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:19:48.100301 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:48.100280    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d51fee9-f492-4995-b40d-fa243258e96a-kube-api-access-tprr9" (OuterVolumeSpecName: "kube-api-access-tprr9") pod "2d51fee9-f492-4995-b40d-fa243258e96a" (UID: "2d51fee9-f492-4995-b40d-fa243258e96a"). InnerVolumeSpecName "kube-api-access-tprr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:19:48.102743 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:48.102710    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d51fee9-f492-4995-b40d-fa243258e96a-util" (OuterVolumeSpecName: "util") pod "2d51fee9-f492-4995-b40d-fa243258e96a" (UID: "2d51fee9-f492-4995-b40d-fa243258e96a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:19:48.199470 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:48.199433    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tprr9\" (UniqueName: \"kubernetes.io/projected/2d51fee9-f492-4995-b40d-fa243258e96a-kube-api-access-tprr9\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:19:48.199577 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:48.199481    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d51fee9-f492-4995-b40d-fa243258e96a-util\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:19:48.199577 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:48.199495    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d51fee9-f492-4995-b40d-fa243258e96a-bundle\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:19:48.787472 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:48.787432    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp"
Sep 09 20:19:48.787472 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:48.787454    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76cdjp" event={"ID":"4bab3ff7-f099-41b7-baac-42c013cc42aa","Type":"ContainerDied","Data":"57424c82f8b017f0f4dc6f66d6091d47645e7e361e2986d2312109277c0dabeb"}
Sep 09 20:19:48.787964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:48.787507    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57424c82f8b017f0f4dc6f66d6091d47645e7e361e2986d2312109277c0dabeb"
Sep 09 20:19:48.789454 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:48.789430    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7"
Sep 09 20:19:48.789454 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:48.789440    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apkcr7" event={"ID":"2d51fee9-f492-4995-b40d-fa243258e96a","Type":"ContainerDied","Data":"73b32a81e41b37f46c31e57e063033e1981a1ddd8325231357d55df53eb11adf"}
Sep 09 20:19:48.789645 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:48.789500    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73b32a81e41b37f46c31e57e063033e1981a1ddd8325231357d55df53eb11adf"
Sep 09 20:19:48.791703 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:48.791679    2581 generic.go:356] "Generic (PLEG): container finished" podID="fdeeba50-d9b9-4558-bbdb-def597b9c2c4" containerID="4ffd17073c2f9cbcd1b77fe6c1da6290a43ca15a8fae4e05c95f5c58dfdef75f" exitCode=0
Sep 09 20:19:48.791784 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:48.791752    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25" event={"ID":"fdeeba50-d9b9-4558-bbdb-def597b9c2c4","Type":"ContainerDied","Data":"4ffd17073c2f9cbcd1b77fe6c1da6290a43ca15a8fae4e05c95f5c58dfdef75f"}
Sep 09 20:19:49.798210 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:49.798178    2581 generic.go:356] "Generic (PLEG): container finished" podID="fdeeba50-d9b9-4558-bbdb-def597b9c2c4" containerID="e5e934d392b7882bc533dfdd6c7de87f9f1bc0668fa1705dfca4031b20ebbbbe" exitCode=0
Sep 09 20:19:49.798576 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:49.798266    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25" event={"ID":"fdeeba50-d9b9-4558-bbdb-def597b9c2c4","Type":"ContainerDied","Data":"e5e934d392b7882bc533dfdd6c7de87f9f1bc0668fa1705dfca4031b20ebbbbe"}
Sep 09 20:19:50.926351 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:50.926325    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25"
Sep 09 20:19:51.023948 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:51.023913    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fdeeba50-d9b9-4558-bbdb-def597b9c2c4-util\") pod \"fdeeba50-d9b9-4558-bbdb-def597b9c2c4\" (UID: \"fdeeba50-d9b9-4558-bbdb-def597b9c2c4\") "
Sep 09 20:19:51.024120 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:51.023969    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcqv7\" (UniqueName: \"kubernetes.io/projected/fdeeba50-d9b9-4558-bbdb-def597b9c2c4-kube-api-access-qcqv7\") pod \"fdeeba50-d9b9-4558-bbdb-def597b9c2c4\" (UID: \"fdeeba50-d9b9-4558-bbdb-def597b9c2c4\") "
Sep 09 20:19:51.024120 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:51.024001    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdeeba50-d9b9-4558-bbdb-def597b9c2c4-bundle\") pod \"fdeeba50-d9b9-4558-bbdb-def597b9c2c4\" (UID: \"fdeeba50-d9b9-4558-bbdb-def597b9c2c4\") "
Sep 09 20:19:51.024623 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:51.024582    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdeeba50-d9b9-4558-bbdb-def597b9c2c4-bundle" (OuterVolumeSpecName: "bundle") pod "fdeeba50-d9b9-4558-bbdb-def597b9c2c4" (UID: "fdeeba50-d9b9-4558-bbdb-def597b9c2c4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:19:51.026100 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:51.026077    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdeeba50-d9b9-4558-bbdb-def597b9c2c4-kube-api-access-qcqv7" (OuterVolumeSpecName: "kube-api-access-qcqv7") pod "fdeeba50-d9b9-4558-bbdb-def597b9c2c4" (UID: "fdeeba50-d9b9-4558-bbdb-def597b9c2c4"). InnerVolumeSpecName "kube-api-access-qcqv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:19:51.028512 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:51.028487    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdeeba50-d9b9-4558-bbdb-def597b9c2c4-util" (OuterVolumeSpecName: "util") pod "fdeeba50-d9b9-4558-bbdb-def597b9c2c4" (UID: "fdeeba50-d9b9-4558-bbdb-def597b9c2c4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:19:51.124849 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:51.124755    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fdeeba50-d9b9-4558-bbdb-def597b9c2c4-util\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:19:51.124849 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:51.124787    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcqv7\" (UniqueName: \"kubernetes.io/projected/fdeeba50-d9b9-4558-bbdb-def597b9c2c4-kube-api-access-qcqv7\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:19:51.124849 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:51.124797    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdeeba50-d9b9-4558-bbdb-def597b9c2c4-bundle\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:19:51.808894 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:51.808860    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25"
Sep 09 20:19:51.809060 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:51.808866    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cllf25" event={"ID":"fdeeba50-d9b9-4558-bbdb-def597b9c2c4","Type":"ContainerDied","Data":"948cbcd3dcc9c0f6ebd10d4ee95f40e34bf56a4e582704176e5850bdfa422af2"}
Sep 09 20:19:51.809060 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:51.808982    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="948cbcd3dcc9c0f6ebd10d4ee95f40e34bf56a4e582704176e5850bdfa422af2"
Sep 09 20:19:59.708670 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.708638    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-dxwm2"]
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.708892    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d51fee9-f492-4995-b40d-fa243258e96a" containerName="extract"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.708903    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d51fee9-f492-4995-b40d-fa243258e96a" containerName="extract"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.708917    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdeeba50-d9b9-4558-bbdb-def597b9c2c4" containerName="util"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.708922    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdeeba50-d9b9-4558-bbdb-def597b9c2c4" containerName="util"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.708930    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fb7ee8d-dd7a-4507-a16a-dea0b16007d4" containerName="extract"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.708935    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fb7ee8d-dd7a-4507-a16a-dea0b16007d4" containerName="extract"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.708942    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d51fee9-f492-4995-b40d-fa243258e96a" containerName="util"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.708947    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d51fee9-f492-4995-b40d-fa243258e96a" containerName="util"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.708954    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bab3ff7-f099-41b7-baac-42c013cc42aa" containerName="util"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.708959    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bab3ff7-f099-41b7-baac-42c013cc42aa" containerName="util"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.708965    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bab3ff7-f099-41b7-baac-42c013cc42aa" containerName="extract"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.708970    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bab3ff7-f099-41b7-baac-42c013cc42aa" containerName="extract"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.708976    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdeeba50-d9b9-4558-bbdb-def597b9c2c4" containerName="extract"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.708981    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdeeba50-d9b9-4558-bbdb-def597b9c2c4" containerName="extract"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.708989    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fb7ee8d-dd7a-4507-a16a-dea0b16007d4" containerName="pull"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.708994    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fb7ee8d-dd7a-4507-a16a-dea0b16007d4" containerName="pull"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.709005    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d51fee9-f492-4995-b40d-fa243258e96a" containerName="pull"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.709009    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d51fee9-f492-4995-b40d-fa243258e96a" containerName="pull"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.709016    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bab3ff7-f099-41b7-baac-42c013cc42aa" containerName="pull"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.709020    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bab3ff7-f099-41b7-baac-42c013cc42aa" containerName="pull"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.709027    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fb7ee8d-dd7a-4507-a16a-dea0b16007d4" containerName="util"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.709031    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fb7ee8d-dd7a-4507-a16a-dea0b16007d4" containerName="util"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.709037    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdeeba50-d9b9-4558-bbdb-def597b9c2c4" containerName="pull"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.709041    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdeeba50-d9b9-4558-bbdb-def597b9c2c4" containerName="pull"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.709081    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdeeba50-d9b9-4558-bbdb-def597b9c2c4" containerName="extract"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.709087    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bab3ff7-f099-41b7-baac-42c013cc42aa" containerName="extract"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.709094    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d51fee9-f492-4995-b40d-fa243258e96a" containerName="extract"
Sep 09 20:19:59.709123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.709101    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fb7ee8d-dd7a-4507-a16a-dea0b16007d4" containerName="extract"
Sep 09 20:19:59.718436 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.718416    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-dxwm2"
Sep 09 20:19:59.720825 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.720803    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 20:19:59.720928 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.720804    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 09 20:19:59.720928 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.720878    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-6wl9j\""
Sep 09 20:19:59.722090 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.722063    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 20:19:59.724183 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.724159    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-dxwm2"]
Sep 09 20:19:59.898186 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.898156    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv58j\" (UniqueName: \"kubernetes.io/projected/7b459bd6-a3b5-45af-b1df-cc0517f020ff-kube-api-access-pv58j\") pod \"dns-operator-controller-manager-65dc8f76f5-dxwm2\" (UID: \"7b459bd6-a3b5-45af-b1df-cc0517f020ff\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-dxwm2"
Sep 09 20:19:59.999551 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:19:59.999520    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv58j\" (UniqueName: \"kubernetes.io/projected/7b459bd6-a3b5-45af-b1df-cc0517f020ff-kube-api-access-pv58j\") pod \"dns-operator-controller-manager-65dc8f76f5-dxwm2\" (UID: \"7b459bd6-a3b5-45af-b1df-cc0517f020ff\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-dxwm2"
Sep 09 20:20:00.010370 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:00.010337    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv58j\" (UniqueName: \"kubernetes.io/projected/7b459bd6-a3b5-45af-b1df-cc0517f020ff-kube-api-access-pv58j\") pod \"dns-operator-controller-manager-65dc8f76f5-dxwm2\" (UID: \"7b459bd6-a3b5-45af-b1df-cc0517f020ff\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-dxwm2"
Sep 09 20:20:00.028786 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:00.028757    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-dxwm2"
Sep 09 20:20:00.301378 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:00.301347    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-dxwm2"]
Sep 09 20:20:00.303531 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:20:00.303506    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b459bd6_a3b5_45af_b1df_cc0517f020ff.slice/crio-7b893e45b6f5f216911d09e012f921fb9334b2a152bead1c9860888169a88a2f WatchSource:0}: Error finding container 7b893e45b6f5f216911d09e012f921fb9334b2a152bead1c9860888169a88a2f: Status 404 returned error can't find the container with id 7b893e45b6f5f216911d09e012f921fb9334b2a152bead1c9860888169a88a2f
Sep 09 20:20:00.851792 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:00.851760    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-dxwm2" event={"ID":"7b459bd6-a3b5-45af-b1df-cc0517f020ff","Type":"ContainerStarted","Data":"7b893e45b6f5f216911d09e012f921fb9334b2a152bead1c9860888169a88a2f"}
Sep 09 20:20:01.709992 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:01.709952    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-j6nph"]
Sep 09 20:20:01.714595 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:01.714574    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-j6nph"
Sep 09 20:20:01.717055 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:01.717035    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-fcpv7\""
Sep 09 20:20:01.722813 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:01.722781    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-j6nph"]
Sep 09 20:20:01.813165 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:01.813124    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvdtc\" (UniqueName: \"kubernetes.io/projected/d21fa37d-4a14-42e2-b957-2d8a816294a1-kube-api-access-pvdtc\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-j6nph\" (UID: \"d21fa37d-4a14-42e2-b957-2d8a816294a1\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-j6nph"
Sep 09 20:20:01.913869 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:01.913832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvdtc\" (UniqueName: \"kubernetes.io/projected/d21fa37d-4a14-42e2-b957-2d8a816294a1-kube-api-access-pvdtc\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-j6nph\" (UID: \"d21fa37d-4a14-42e2-b957-2d8a816294a1\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-j6nph"
Sep 09 20:20:01.924552 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:01.924517    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvdtc\" (UniqueName: \"kubernetes.io/projected/d21fa37d-4a14-42e2-b957-2d8a816294a1-kube-api-access-pvdtc\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-j6nph\" (UID: \"d21fa37d-4a14-42e2-b957-2d8a816294a1\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-j6nph"
Sep 09 20:20:02.025901 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:02.025848    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-j6nph"
Sep 09 20:20:02.299251 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:02.299226    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-j6nph"]
Sep 09 20:20:02.301021 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:20:02.300995    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21fa37d_4a14_42e2_b957_2d8a816294a1.slice/crio-f342a19b0b0b8b66bf2faa142bee642630f5f979e630aab3f4e159e64db8ae99 WatchSource:0}: Error finding container f342a19b0b0b8b66bf2faa142bee642630f5f979e630aab3f4e159e64db8ae99: Status 404 returned error can't find the container with id f342a19b0b0b8b66bf2faa142bee642630f5f979e630aab3f4e159e64db8ae99
Sep 09 20:20:02.862342 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:02.862307    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-j6nph" event={"ID":"d21fa37d-4a14-42e2-b957-2d8a816294a1","Type":"ContainerStarted","Data":"f342a19b0b0b8b66bf2faa142bee642630f5f979e630aab3f4e159e64db8ae99"}
Sep 09 20:20:06.127247 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:06.127206    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-ns7nk"]
Sep 09 20:20:06.130414 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:06.130388    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-ns7nk"
Sep 09 20:20:06.133097 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:06.133071    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-7zdhx\""
Sep 09 20:20:06.142375 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:06.142347    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fd7138a-f0ac-4adf-8206-c2a1e273cc00-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-ns7nk\" (UID: \"9fd7138a-f0ac-4adf-8206-c2a1e273cc00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-ns7nk"
Sep 09 20:20:06.142516 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:06.142394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpxqj\" (UniqueName: \"kubernetes.io/projected/9fd7138a-f0ac-4adf-8206-c2a1e273cc00-kube-api-access-fpxqj\") pod \"kuadrant-operator-controller-manager-6d4cb79c-ns7nk\" (UID: \"9fd7138a-f0ac-4adf-8206-c2a1e273cc00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-ns7nk"
Sep 09 20:20:06.142723 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:06.142705    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-ns7nk"]
Sep 09 20:20:06.243201 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:06.243169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpxqj\" (UniqueName: \"kubernetes.io/projected/9fd7138a-f0ac-4adf-8206-c2a1e273cc00-kube-api-access-fpxqj\") pod \"kuadrant-operator-controller-manager-6d4cb79c-ns7nk\" (UID: \"9fd7138a-f0ac-4adf-8206-c2a1e273cc00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-ns7nk"
Sep 09 20:20:06.243367 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:06.243258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fd7138a-f0ac-4adf-8206-c2a1e273cc00-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-ns7nk\" (UID: \"9fd7138a-f0ac-4adf-8206-c2a1e273cc00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-ns7nk"
Sep 09 20:20:06.243706 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:06.243688    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fd7138a-f0ac-4adf-8206-c2a1e273cc00-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-ns7nk\" (UID: \"9fd7138a-f0ac-4adf-8206-c2a1e273cc00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-ns7nk"
Sep 09 20:20:06.254768 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:06.254742    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpxqj\" (UniqueName: \"kubernetes.io/projected/9fd7138a-f0ac-4adf-8206-c2a1e273cc00-kube-api-access-fpxqj\") pod \"kuadrant-operator-controller-manager-6d4cb79c-ns7nk\" (UID: \"9fd7138a-f0ac-4adf-8206-c2a1e273cc00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-ns7nk"
Sep 09 20:20:06.441551 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:06.441450    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-ns7nk"
Sep 09 20:20:06.717730 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:06.717699    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-ns7nk"]
Sep 09 20:20:06.719908 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:20:06.719877    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd7138a_f0ac_4adf_8206_c2a1e273cc00.slice/crio-d73e388b3059ae95c589a71a1db1ffa46ff086a9960fcc2c411fc6b9600f3dcf WatchSource:0}: Error finding container d73e388b3059ae95c589a71a1db1ffa46ff086a9960fcc2c411fc6b9600f3dcf: Status 404 returned error can't find the container with id d73e388b3059ae95c589a71a1db1ffa46ff086a9960fcc2c411fc6b9600f3dcf
Sep 09 20:20:06.882881 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:06.882840    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-j6nph" event={"ID":"d21fa37d-4a14-42e2-b957-2d8a816294a1","Type":"ContainerStarted","Data":"f6c622a3867403726945e738aa6df84a9343736a545d424c9cf3fd657b30830d"}
Sep 09 20:20:06.883100 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:06.883015    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-j6nph"
Sep 09 20:20:06.884173 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:06.884153    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-ns7nk" event={"ID":"9fd7138a-f0ac-4adf-8206-c2a1e273cc00","Type":"ContainerStarted","Data":"d73e388b3059ae95c589a71a1db1ffa46ff086a9960fcc2c411fc6b9600f3dcf"}
Sep 09 20:20:06.901475 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:06.901424    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-j6nph" podStartSLOduration=2.161489992 podStartE2EDuration="5.901411791s" totalImagesPullingTime="3.739921799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:20:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:20:06.900109355 +0000 UTC m=+548.905286349" watchObservedRunningTime="2026-09-09 20:20:06.901411791 +0000 UTC m=+548.906588786"
Sep 09 20:20:10.909620 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:10.909539    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-ns7nk" event={"ID":"9fd7138a-f0ac-4adf-8206-c2a1e273cc00","Type":"ContainerStarted","Data":"fed9f6293411f557a6c371cab4a747970760a1bf411283559419b13fcfaf95d3"}
Sep 09 20:20:10.909972 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:10.909676    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-ns7nk"
Sep 09 20:20:10.933998 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:10.933946    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-ns7nk" podStartSLOduration=1.045048952 podStartE2EDuration="4.933931742s" totalImagesPullingTime="3.88888279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:20:10.930868412 +0000 UTC m=+552.936045407" watchObservedRunningTime="2026-09-09 20:20:10.933931742 +0000 UTC m=+552.939108738"
Sep 09 20:20:16.938931 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:16.938900    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-dxwm2" event={"ID":"7b459bd6-a3b5-45af-b1df-cc0517f020ff","Type":"ContainerStarted","Data":"db7c7e4d8e8ff9e8e92913ad6148091c24af6524082cfe3199afc1fb7fc126f4"}
Sep 09 20:20:16.939416 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:16.939025    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-dxwm2"
Sep 09 20:20:16.960706 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:16.960660    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-dxwm2" podStartSLOduration=1.4680306779999999 podStartE2EDuration="17.960636167s" totalImagesPullingTime="16.492605489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:20:16.958786981 +0000 UTC m=+558.963963976" watchObservedRunningTime="2026-09-09 20:20:16.960636167 +0000 UTC m=+558.965813161"
Sep 09 20:20:17.891787 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:17.891755    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-j6nph"
Sep 09 20:20:21.917293 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:21.917261    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-ns7nk"
Sep 09 20:20:27.946867 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:27.946836    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-dxwm2"
Sep 09 20:20:56.151787 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:56.151752    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-k9jgt"]
Sep 09 20:20:56.153852 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:56.153830    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-k9jgt"
Sep 09 20:20:56.156254 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:56.156222    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 20:20:56.156354 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:56.156314    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9mpkd\""
Sep 09 20:20:56.162748 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:56.162725    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-k9jgt"]
Sep 09 20:20:56.200379 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:56.200352    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/817eda65-cdf2-418b-a684-b6e51c955198-config-file\") pod \"limitador-limitador-8588749977-k9jgt\" (UID: \"817eda65-cdf2-418b-a684-b6e51c955198\") " pod="kuadrant-system/limitador-limitador-8588749977-k9jgt"
Sep 09 20:20:56.200521 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:56.200431    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72bh6\" (UniqueName: \"kubernetes.io/projected/817eda65-cdf2-418b-a684-b6e51c955198-kube-api-access-72bh6\") pod \"limitador-limitador-8588749977-k9jgt\" (UID: \"817eda65-cdf2-418b-a684-b6e51c955198\") " pod="kuadrant-system/limitador-limitador-8588749977-k9jgt"
Sep 09 20:20:56.250628 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:56.250592    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-k9jgt"]
Sep 09 20:20:56.301500 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:56.301448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72bh6\" (UniqueName: \"kubernetes.io/projected/817eda65-cdf2-418b-a684-b6e51c955198-kube-api-access-72bh6\") pod \"limitador-limitador-8588749977-k9jgt\" (UID: \"817eda65-cdf2-418b-a684-b6e51c955198\") " pod="kuadrant-system/limitador-limitador-8588749977-k9jgt"
Sep 09 20:20:56.301670 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:56.301525    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/817eda65-cdf2-418b-a684-b6e51c955198-config-file\") pod \"limitador-limitador-8588749977-k9jgt\" (UID: \"817eda65-cdf2-418b-a684-b6e51c955198\") " pod="kuadrant-system/limitador-limitador-8588749977-k9jgt"
Sep 09 20:20:56.302107 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:56.302086    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/817eda65-cdf2-418b-a684-b6e51c955198-config-file\") pod \"limitador-limitador-8588749977-k9jgt\" (UID: \"817eda65-cdf2-418b-a684-b6e51c955198\") " pod="kuadrant-system/limitador-limitador-8588749977-k9jgt"
Sep 09 20:20:56.309808 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:56.309782    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72bh6\" (UniqueName: \"kubernetes.io/projected/817eda65-cdf2-418b-a684-b6e51c955198-kube-api-access-72bh6\") pod \"limitador-limitador-8588749977-k9jgt\" (UID: \"817eda65-cdf2-418b-a684-b6e51c955198\") " pod="kuadrant-system/limitador-limitador-8588749977-k9jgt"
Sep 09 20:20:56.464716 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:56.464636    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-k9jgt"
Sep 09 20:20:56.734246 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:56.734222    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-k9jgt"]
Sep 09 20:20:56.736481 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:20:56.736442    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817eda65_cdf2_418b_a684_b6e51c955198.slice/crio-1e2b073f9ccb4d625c2047eac788e167428804fb3e2d3d277a831c1e86ce570f WatchSource:0}: Error finding container 1e2b073f9ccb4d625c2047eac788e167428804fb3e2d3d277a831c1e86ce570f: Status 404 returned error can't find the container with id 1e2b073f9ccb4d625c2047eac788e167428804fb3e2d3d277a831c1e86ce570f
Sep 09 20:20:57.125099 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:57.125016    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-k9jgt" event={"ID":"817eda65-cdf2-418b-a684-b6e51c955198","Type":"ContainerStarted","Data":"1e2b073f9ccb4d625c2047eac788e167428804fb3e2d3d277a831c1e86ce570f"}
Sep 09 20:20:58.131114 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:58.131087    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-k9jgt" event={"ID":"817eda65-cdf2-418b-a684-b6e51c955198","Type":"ContainerStarted","Data":"98c62fd77112ddda4de627644bc0eea11aee64dbca0d46d8530d0a1b6c1a7498"}
Sep 09 20:20:58.131418 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:58.131214    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-k9jgt"
Sep 09 20:20:58.147714 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:58.147674    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-k9jgt" podStartSLOduration=0.81955605 podStartE2EDuration="2.147659001s" totalImagesPullingTime="1.328102951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:20:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:20:58.146103905 +0000 UTC m=+600.151280901" watchObservedRunningTime="2026-09-09 20:20:58.147659001 +0000 UTC m=+600.152835995"
Sep 09 20:20:58.419482 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:58.419392    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 20:20:58.421872 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:58.421852    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 20:20:58.424086 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:58.424069    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 20:20:58.426666 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:20:58.426648    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 20:21:09.138175 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:09.138135    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-k9jgt"
Sep 09 20:21:10.243205 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:10.243170    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-k9jgt"]
Sep 09 20:21:10.243650 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:10.243403    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-k9jgt" podUID="817eda65-cdf2-418b-a684-b6e51c955198" containerName="limitador" containerID="cri-o://98c62fd77112ddda4de627644bc0eea11aee64dbca0d46d8530d0a1b6c1a7498" gracePeriod=30
Sep 09 20:21:11.174770 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:11.174749    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-k9jgt"
Sep 09 20:21:11.193622 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:11.193595    2581 generic.go:356] "Generic (PLEG): container finished" podID="817eda65-cdf2-418b-a684-b6e51c955198" containerID="98c62fd77112ddda4de627644bc0eea11aee64dbca0d46d8530d0a1b6c1a7498" exitCode=0
Sep 09 20:21:11.193780 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:11.193641    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-k9jgt"
Sep 09 20:21:11.193780 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:11.193683    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-k9jgt" event={"ID":"817eda65-cdf2-418b-a684-b6e51c955198","Type":"ContainerDied","Data":"98c62fd77112ddda4de627644bc0eea11aee64dbca0d46d8530d0a1b6c1a7498"}
Sep 09 20:21:11.193780 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:11.193744    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-k9jgt" event={"ID":"817eda65-cdf2-418b-a684-b6e51c955198","Type":"ContainerDied","Data":"1e2b073f9ccb4d625c2047eac788e167428804fb3e2d3d277a831c1e86ce570f"}
Sep 09 20:21:11.193780 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:11.193774    2581 scope.go:117] "RemoveContainer" containerID="98c62fd77112ddda4de627644bc0eea11aee64dbca0d46d8530d0a1b6c1a7498"
Sep 09 20:21:11.203367 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:11.203345    2581 scope.go:117] "RemoveContainer" containerID="98c62fd77112ddda4de627644bc0eea11aee64dbca0d46d8530d0a1b6c1a7498"
Sep 09 20:21:11.203694 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:21:11.203667    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98c62fd77112ddda4de627644bc0eea11aee64dbca0d46d8530d0a1b6c1a7498\": container with ID starting with 98c62fd77112ddda4de627644bc0eea11aee64dbca0d46d8530d0a1b6c1a7498 not found: ID does not exist" containerID="98c62fd77112ddda4de627644bc0eea11aee64dbca0d46d8530d0a1b6c1a7498"
Sep 09 20:21:11.203757 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:11.203699    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98c62fd77112ddda4de627644bc0eea11aee64dbca0d46d8530d0a1b6c1a7498"} err="failed to get container status \"98c62fd77112ddda4de627644bc0eea11aee64dbca0d46d8530d0a1b6c1a7498\": rpc error: code = NotFound desc = could not find container \"98c62fd77112ddda4de627644bc0eea11aee64dbca0d46d8530d0a1b6c1a7498\": container with ID starting with 98c62fd77112ddda4de627644bc0eea11aee64dbca0d46d8530d0a1b6c1a7498 not found: ID does not exist"
Sep 09 20:21:11.226984 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:11.226961    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72bh6\" (UniqueName: \"kubernetes.io/projected/817eda65-cdf2-418b-a684-b6e51c955198-kube-api-access-72bh6\") pod \"817eda65-cdf2-418b-a684-b6e51c955198\" (UID: \"817eda65-cdf2-418b-a684-b6e51c955198\") "
Sep 09 20:21:11.227082 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:11.227052    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/817eda65-cdf2-418b-a684-b6e51c955198-config-file\") pod \"817eda65-cdf2-418b-a684-b6e51c955198\" (UID: \"817eda65-cdf2-418b-a684-b6e51c955198\") "
Sep 09 20:21:11.227372 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:11.227353    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/817eda65-cdf2-418b-a684-b6e51c955198-config-file" (OuterVolumeSpecName: "config-file") pod "817eda65-cdf2-418b-a684-b6e51c955198" (UID: "817eda65-cdf2-418b-a684-b6e51c955198"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:21:11.229045 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:11.229021    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/817eda65-cdf2-418b-a684-b6e51c955198-kube-api-access-72bh6" (OuterVolumeSpecName: "kube-api-access-72bh6") pod "817eda65-cdf2-418b-a684-b6e51c955198" (UID: "817eda65-cdf2-418b-a684-b6e51c955198"). InnerVolumeSpecName "kube-api-access-72bh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:21:11.327893 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:11.327819    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72bh6\" (UniqueName: \"kubernetes.io/projected/817eda65-cdf2-418b-a684-b6e51c955198-kube-api-access-72bh6\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:21:11.327893 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:11.327845    2581 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/817eda65-cdf2-418b-a684-b6e51c955198-config-file\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:21:11.515814 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:11.515783    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-k9jgt"]
Sep 09 20:21:11.519863 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:11.519828    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-k9jgt"]
Sep 09 20:21:12.634480 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:12.634434    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="817eda65-cdf2-418b-a684-b6e51c955198" path="/var/lib/kubelet/pods/817eda65-cdf2-418b-a684-b6e51c955198/volumes"
Sep 09 20:21:39.218835 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.218798    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"]
Sep 09 20:21:39.219240 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.219110    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="817eda65-cdf2-418b-a684-b6e51c955198" containerName="limitador"
Sep 09 20:21:39.219240 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.219121    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="817eda65-cdf2-418b-a684-b6e51c955198" containerName="limitador"
Sep 09 20:21:39.219240 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.219182    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="817eda65-cdf2-418b-a684-b6e51c955198" containerName="limitador"
Sep 09 20:21:39.223273 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.223252    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:39.232922 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.232897    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"]
Sep 09 20:21:39.257811 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.257781    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e75de6e0-0c7d-4e09-8408-5378cb8294b1-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-46gml\" (UID: \"e75de6e0-0c7d-4e09-8408-5378cb8294b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:39.257811 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.257812    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e75de6e0-0c7d-4e09-8408-5378cb8294b1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-46gml\" (UID: \"e75de6e0-0c7d-4e09-8408-5378cb8294b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:39.258005 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.257828    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e75de6e0-0c7d-4e09-8408-5378cb8294b1-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-46gml\" (UID: \"e75de6e0-0c7d-4e09-8408-5378cb8294b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:39.258005 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.257883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e75de6e0-0c7d-4e09-8408-5378cb8294b1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-46gml\" (UID: \"e75de6e0-0c7d-4e09-8408-5378cb8294b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:39.258005 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.257924    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm5z4\" (UniqueName: \"kubernetes.io/projected/e75de6e0-0c7d-4e09-8408-5378cb8294b1-kube-api-access-hm5z4\") pod \"istiod-openshift-gateway-5677ccb49b-46gml\" (UID: \"e75de6e0-0c7d-4e09-8408-5378cb8294b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:39.258005 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.257974    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e75de6e0-0c7d-4e09-8408-5378cb8294b1-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-46gml\" (UID: \"e75de6e0-0c7d-4e09-8408-5378cb8294b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:39.258142 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.258005    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e75de6e0-0c7d-4e09-8408-5378cb8294b1-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-46gml\" (UID: \"e75de6e0-0c7d-4e09-8408-5378cb8294b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:39.358686 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.358654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e75de6e0-0c7d-4e09-8408-5378cb8294b1-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-46gml\" (UID: \"e75de6e0-0c7d-4e09-8408-5378cb8294b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:39.358856 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.358698    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e75de6e0-0c7d-4e09-8408-5378cb8294b1-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-46gml\" (UID: \"e75de6e0-0c7d-4e09-8408-5378cb8294b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:39.358856 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.358715    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e75de6e0-0c7d-4e09-8408-5378cb8294b1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-46gml\" (UID: \"e75de6e0-0c7d-4e09-8408-5378cb8294b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:39.358856 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.358830    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e75de6e0-0c7d-4e09-8408-5378cb8294b1-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-46gml\" (UID: \"e75de6e0-0c7d-4e09-8408-5378cb8294b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:39.359012 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.358891    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e75de6e0-0c7d-4e09-8408-5378cb8294b1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-46gml\" (UID: \"e75de6e0-0c7d-4e09-8408-5378cb8294b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:39.359012 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.358917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm5z4\" (UniqueName: \"kubernetes.io/projected/e75de6e0-0c7d-4e09-8408-5378cb8294b1-kube-api-access-hm5z4\") pod \"istiod-openshift-gateway-5677ccb49b-46gml\" (UID: \"e75de6e0-0c7d-4e09-8408-5378cb8294b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:39.359012 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.358953    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e75de6e0-0c7d-4e09-8408-5378cb8294b1-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-46gml\" (UID: \"e75de6e0-0c7d-4e09-8408-5378cb8294b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:39.359726 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.359691    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e75de6e0-0c7d-4e09-8408-5378cb8294b1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-46gml\" (UID: \"e75de6e0-0c7d-4e09-8408-5378cb8294b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:39.361176 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.361147    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e75de6e0-0c7d-4e09-8408-5378cb8294b1-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-46gml\" (UID: \"e75de6e0-0c7d-4e09-8408-5378cb8294b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:39.361295 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.361224    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e75de6e0-0c7d-4e09-8408-5378cb8294b1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-46gml\" (UID: \"e75de6e0-0c7d-4e09-8408-5378cb8294b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:39.361295 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.361245    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e75de6e0-0c7d-4e09-8408-5378cb8294b1-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-46gml\" (UID: \"e75de6e0-0c7d-4e09-8408-5378cb8294b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:39.361295 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.361277    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e75de6e0-0c7d-4e09-8408-5378cb8294b1-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-46gml\" (UID: \"e75de6e0-0c7d-4e09-8408-5378cb8294b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:39.367594 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.367569    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e75de6e0-0c7d-4e09-8408-5378cb8294b1-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-46gml\" (UID: \"e75de6e0-0c7d-4e09-8408-5378cb8294b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:39.368244 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.368224    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm5z4\" (UniqueName: \"kubernetes.io/projected/e75de6e0-0c7d-4e09-8408-5378cb8294b1-kube-api-access-hm5z4\") pod \"istiod-openshift-gateway-5677ccb49b-46gml\" (UID: \"e75de6e0-0c7d-4e09-8408-5378cb8294b1\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:39.532356 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.532323    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:39.806401 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.806308    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"]
Sep 09 20:21:39.808979 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:21:39.808950    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode75de6e0_0c7d_4e09_8408_5378cb8294b1.slice/crio-3cf80891a5f2473b5788e12cbdbfe60d7da57534e6b6cd4e9247f06d99b02afa WatchSource:0}: Error finding container 3cf80891a5f2473b5788e12cbdbfe60d7da57534e6b6cd4e9247f06d99b02afa: Status 404 returned error can't find the container with id 3cf80891a5f2473b5788e12cbdbfe60d7da57534e6b6cd4e9247f06d99b02afa
Sep 09 20:21:39.811113 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.811077    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:21:39.811180 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:39.811153    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:21:40.334770 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:40.334737    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml" event={"ID":"e75de6e0-0c7d-4e09-8408-5378cb8294b1","Type":"ContainerStarted","Data":"96ab4c10cecea3ef9714d89c77cb4cd6b21cdfaabf74038dcd7fbf5659f51e0f"}
Sep 09 20:21:40.335287 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:40.334794    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml" event={"ID":"e75de6e0-0c7d-4e09-8408-5378cb8294b1","Type":"ContainerStarted","Data":"3cf80891a5f2473b5788e12cbdbfe60d7da57534e6b6cd4e9247f06d99b02afa"}
Sep 09 20:21:40.335287 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:40.334914    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:40.336596 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:40.336566    2581 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-46gml container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 20:21:40.336698 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:40.336613    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml" podUID="e75de6e0-0c7d-4e09-8408-5378cb8294b1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:21:40.358603 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:40.358559    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml" podStartSLOduration=1.358546161 podStartE2EDuration="1.358546161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:21:40.355679786 +0000 UTC m=+642.360856781" watchObservedRunningTime="2026-09-09 20:21:40.358546161 +0000 UTC m=+642.363723155"
Sep 09 20:21:41.340388 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.340361    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-46gml"
Sep 09 20:21:41.395311 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.395280    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"]
Sep 09 20:21:41.395494 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.395469    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf" podUID="3bc5f46a-e750-4641-81cf-2808d7fe37a3" containerName="discovery" containerID="cri-o://919a4964d55f4648b5f92ffd3fd6dbd54d5d34a67e10dc3cd4e336e8706c86fb" gracePeriod=30
Sep 09 20:21:41.637973 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.637949    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:21:41.676587 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.676556    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3bc5f46a-e750-4641-81cf-2808d7fe37a3-istio-csr-dns-cert\") pod \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") "
Sep 09 20:21:41.676587 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.676587    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3bc5f46a-e750-4641-81cf-2808d7fe37a3-istio-kubeconfig\") pod \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") "
Sep 09 20:21:41.676783 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.676610    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3bc5f46a-e750-4641-81cf-2808d7fe37a3-cacerts\") pod \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") "
Sep 09 20:21:41.676783 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.676639    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc5f46a-e750-4641-81cf-2808d7fe37a3-local-certs\") pod \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") "
Sep 09 20:21:41.676783 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.676772    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3bc5f46a-e750-4641-81cf-2808d7fe37a3-istio-csr-ca-configmap\") pod \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") "
Sep 09 20:21:41.676937 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.676831    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc5f46a-e750-4641-81cf-2808d7fe37a3-istio-token\") pod \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") "
Sep 09 20:21:41.676937 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.676863    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpcqt\" (UniqueName: \"kubernetes.io/projected/3bc5f46a-e750-4641-81cf-2808d7fe37a3-kube-api-access-rpcqt\") pod \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\" (UID: \"3bc5f46a-e750-4641-81cf-2808d7fe37a3\") "
Sep 09 20:21:41.677319 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.677270    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc5f46a-e750-4641-81cf-2808d7fe37a3-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "3bc5f46a-e750-4641-81cf-2808d7fe37a3" (UID: "3bc5f46a-e750-4641-81cf-2808d7fe37a3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:21:41.679102 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.679070    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc5f46a-e750-4641-81cf-2808d7fe37a3-local-certs" (OuterVolumeSpecName: "local-certs") pod "3bc5f46a-e750-4641-81cf-2808d7fe37a3" (UID: "3bc5f46a-e750-4641-81cf-2808d7fe37a3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:21:41.679267 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.679138    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc5f46a-e750-4641-81cf-2808d7fe37a3-cacerts" (OuterVolumeSpecName: "cacerts") pod "3bc5f46a-e750-4641-81cf-2808d7fe37a3" (UID: "3bc5f46a-e750-4641-81cf-2808d7fe37a3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:21:41.679267 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.679241    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc5f46a-e750-4641-81cf-2808d7fe37a3-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "3bc5f46a-e750-4641-81cf-2808d7fe37a3" (UID: "3bc5f46a-e750-4641-81cf-2808d7fe37a3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:21:41.679267 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.679259    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc5f46a-e750-4641-81cf-2808d7fe37a3-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "3bc5f46a-e750-4641-81cf-2808d7fe37a3" (UID: "3bc5f46a-e750-4641-81cf-2808d7fe37a3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:21:41.679414 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.679289    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc5f46a-e750-4641-81cf-2808d7fe37a3-istio-token" (OuterVolumeSpecName: "istio-token") pod "3bc5f46a-e750-4641-81cf-2808d7fe37a3" (UID: "3bc5f46a-e750-4641-81cf-2808d7fe37a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:21:41.679450 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.679418    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc5f46a-e750-4641-81cf-2808d7fe37a3-kube-api-access-rpcqt" (OuterVolumeSpecName: "kube-api-access-rpcqt") pod "3bc5f46a-e750-4641-81cf-2808d7fe37a3" (UID: "3bc5f46a-e750-4641-81cf-2808d7fe37a3"). InnerVolumeSpecName "kube-api-access-rpcqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:21:41.777861 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.777825    2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3bc5f46a-e750-4641-81cf-2808d7fe37a3-istio-csr-dns-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:21:41.777861 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.777854    2581 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3bc5f46a-e750-4641-81cf-2808d7fe37a3-istio-kubeconfig\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:21:41.777861 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.777864    2581 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3bc5f46a-e750-4641-81cf-2808d7fe37a3-cacerts\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:21:41.778085 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.777872    2581 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc5f46a-e750-4641-81cf-2808d7fe37a3-local-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:21:41.778085 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.777880    2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3bc5f46a-e750-4641-81cf-2808d7fe37a3-istio-csr-ca-configmap\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:21:41.778085 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.777889    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc5f46a-e750-4641-81cf-2808d7fe37a3-istio-token\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:21:41.778085 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:41.777897    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpcqt\" (UniqueName: \"kubernetes.io/projected/3bc5f46a-e750-4641-81cf-2808d7fe37a3-kube-api-access-rpcqt\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:21:42.345318 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:42.345288    2581 generic.go:356] "Generic (PLEG): container finished" podID="3bc5f46a-e750-4641-81cf-2808d7fe37a3" containerID="919a4964d55f4648b5f92ffd3fd6dbd54d5d34a67e10dc3cd4e336e8706c86fb" exitCode=0
Sep 09 20:21:42.345807 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:42.345336    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"
Sep 09 20:21:42.345807 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:42.345372    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf" event={"ID":"3bc5f46a-e750-4641-81cf-2808d7fe37a3","Type":"ContainerDied","Data":"919a4964d55f4648b5f92ffd3fd6dbd54d5d34a67e10dc3cd4e336e8706c86fb"}
Sep 09 20:21:42.345807 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:42.345443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf" event={"ID":"3bc5f46a-e750-4641-81cf-2808d7fe37a3","Type":"ContainerDied","Data":"c10dca4933feb871a6ae58f315f7481392165157b2510919bcba088591f61999"}
Sep 09 20:21:42.345807 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:42.345482    2581 scope.go:117] "RemoveContainer" containerID="919a4964d55f4648b5f92ffd3fd6dbd54d5d34a67e10dc3cd4e336e8706c86fb"
Sep 09 20:21:42.354350 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:42.354332    2581 scope.go:117] "RemoveContainer" containerID="919a4964d55f4648b5f92ffd3fd6dbd54d5d34a67e10dc3cd4e336e8706c86fb"
Sep 09 20:21:42.354621 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:21:42.354604    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"919a4964d55f4648b5f92ffd3fd6dbd54d5d34a67e10dc3cd4e336e8706c86fb\": container with ID starting with 919a4964d55f4648b5f92ffd3fd6dbd54d5d34a67e10dc3cd4e336e8706c86fb not found: ID does not exist" containerID="919a4964d55f4648b5f92ffd3fd6dbd54d5d34a67e10dc3cd4e336e8706c86fb"
Sep 09 20:21:42.354685 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:42.354628    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"919a4964d55f4648b5f92ffd3fd6dbd54d5d34a67e10dc3cd4e336e8706c86fb"} err="failed to get container status \"919a4964d55f4648b5f92ffd3fd6dbd54d5d34a67e10dc3cd4e336e8706c86fb\": rpc error: code = NotFound desc = could not find container \"919a4964d55f4648b5f92ffd3fd6dbd54d5d34a67e10dc3cd4e336e8706c86fb\": container with ID starting with 919a4964d55f4648b5f92ffd3fd6dbd54d5d34a67e10dc3cd4e336e8706c86fb not found: ID does not exist"
Sep 09 20:21:42.369164 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:42.369143    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"]
Sep 09 20:21:42.374411 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:42.374392    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mjfrf"]
Sep 09 20:21:42.634979 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:42.634900    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bc5f46a-e750-4641-81cf-2808d7fe37a3" path="/var/lib/kubelet/pods/3bc5f46a-e750-4641-81cf-2808d7fe37a3/volumes"
Sep 09 20:21:50.571941 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.571911    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-55b9566966-jgss6"]
Sep 09 20:21:50.572300 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.572240    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bc5f46a-e750-4641-81cf-2808d7fe37a3" containerName="discovery"
Sep 09 20:21:50.572300 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.572252    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bc5f46a-e750-4641-81cf-2808d7fe37a3" containerName="discovery"
Sep 09 20:21:50.572300 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.572298    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bc5f46a-e750-4641-81cf-2808d7fe37a3" containerName="discovery"
Sep 09 20:21:50.576482 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.576449    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-55b9566966-jgss6"
Sep 09 20:21:50.578944 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.578925    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 09 20:21:50.579036 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.578966    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 09 20:21:50.579855 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.579831    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-vt28c\""
Sep 09 20:21:50.579958 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.579887    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 09 20:21:50.582507 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.582446    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-jgss6"]
Sep 09 20:21:50.612768 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.612742    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-jlltk"]
Sep 09 20:21:50.615958 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.615939    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-jlltk"
Sep 09 20:21:50.618181 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.618161    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 09 20:21:50.618281 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.618228    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-w62ns\""
Sep 09 20:21:50.627729 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.627707    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-jlltk"]
Sep 09 20:21:50.645046 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.645023    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a2ce1df6-13fd-4804-b7a3-16c520bb7898-data\") pod \"seaweedfs-64568bcd49-jlltk\" (UID: \"a2ce1df6-13fd-4804-b7a3-16c520bb7898\") " pod="kserve/seaweedfs-64568bcd49-jlltk"
Sep 09 20:21:50.645158 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.645055    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hqfk\" (UniqueName: \"kubernetes.io/projected/c6262414-82c7-4fa0-880c-bcd7cfb9ba50-kube-api-access-6hqfk\") pod \"kserve-controller-manager-55b9566966-jgss6\" (UID: \"c6262414-82c7-4fa0-880c-bcd7cfb9ba50\") " pod="kserve/kserve-controller-manager-55b9566966-jgss6"
Sep 09 20:21:50.645158 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.645134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4bbh\" (UniqueName: \"kubernetes.io/projected/a2ce1df6-13fd-4804-b7a3-16c520bb7898-kube-api-access-s4bbh\") pod \"seaweedfs-64568bcd49-jlltk\" (UID: \"a2ce1df6-13fd-4804-b7a3-16c520bb7898\") " pod="kserve/seaweedfs-64568bcd49-jlltk"
Sep 09 20:21:50.645225 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.645177    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6262414-82c7-4fa0-880c-bcd7cfb9ba50-cert\") pod \"kserve-controller-manager-55b9566966-jgss6\" (UID: \"c6262414-82c7-4fa0-880c-bcd7cfb9ba50\") " pod="kserve/kserve-controller-manager-55b9566966-jgss6"
Sep 09 20:21:50.746029 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.745995    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4bbh\" (UniqueName: \"kubernetes.io/projected/a2ce1df6-13fd-4804-b7a3-16c520bb7898-kube-api-access-s4bbh\") pod \"seaweedfs-64568bcd49-jlltk\" (UID: \"a2ce1df6-13fd-4804-b7a3-16c520bb7898\") " pod="kserve/seaweedfs-64568bcd49-jlltk"
Sep 09 20:21:50.746180 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.746045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6262414-82c7-4fa0-880c-bcd7cfb9ba50-cert\") pod \"kserve-controller-manager-55b9566966-jgss6\" (UID: \"c6262414-82c7-4fa0-880c-bcd7cfb9ba50\") " pod="kserve/kserve-controller-manager-55b9566966-jgss6"
Sep 09 20:21:50.746180 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.746077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a2ce1df6-13fd-4804-b7a3-16c520bb7898-data\") pod \"seaweedfs-64568bcd49-jlltk\" (UID: \"a2ce1df6-13fd-4804-b7a3-16c520bb7898\") " pod="kserve/seaweedfs-64568bcd49-jlltk"
Sep 09 20:21:50.746180 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.746103    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hqfk\" (UniqueName: \"kubernetes.io/projected/c6262414-82c7-4fa0-880c-bcd7cfb9ba50-kube-api-access-6hqfk\") pod \"kserve-controller-manager-55b9566966-jgss6\" (UID: \"c6262414-82c7-4fa0-880c-bcd7cfb9ba50\") " pod="kserve/kserve-controller-manager-55b9566966-jgss6"
Sep 09 20:21:50.746283 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:21:50.746206    2581 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 09 20:21:50.746314 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:21:50.746282    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6262414-82c7-4fa0-880c-bcd7cfb9ba50-cert podName:c6262414-82c7-4fa0-880c-bcd7cfb9ba50 nodeName:}" failed. No retries permitted until 2026-09-09 20:21:51.246265133 +0000 UTC m=+653.251442109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c6262414-82c7-4fa0-880c-bcd7cfb9ba50-cert") pod "kserve-controller-manager-55b9566966-jgss6" (UID: "c6262414-82c7-4fa0-880c-bcd7cfb9ba50") : secret "kserve-webhook-server-cert" not found
Sep 09 20:21:50.746502 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.746449    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a2ce1df6-13fd-4804-b7a3-16c520bb7898-data\") pod \"seaweedfs-64568bcd49-jlltk\" (UID: \"a2ce1df6-13fd-4804-b7a3-16c520bb7898\") " pod="kserve/seaweedfs-64568bcd49-jlltk"
Sep 09 20:21:50.757838 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.757812    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4bbh\" (UniqueName: \"kubernetes.io/projected/a2ce1df6-13fd-4804-b7a3-16c520bb7898-kube-api-access-s4bbh\") pod \"seaweedfs-64568bcd49-jlltk\" (UID: \"a2ce1df6-13fd-4804-b7a3-16c520bb7898\") " pod="kserve/seaweedfs-64568bcd49-jlltk"
Sep 09 20:21:50.757938 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.757815    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hqfk\" (UniqueName: \"kubernetes.io/projected/c6262414-82c7-4fa0-880c-bcd7cfb9ba50-kube-api-access-6hqfk\") pod \"kserve-controller-manager-55b9566966-jgss6\" (UID: \"c6262414-82c7-4fa0-880c-bcd7cfb9ba50\") " pod="kserve/kserve-controller-manager-55b9566966-jgss6"
Sep 09 20:21:50.924825 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:50.924743    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-jlltk"
Sep 09 20:21:51.197476 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:51.197437    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-jlltk"]
Sep 09 20:21:51.199359 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:21:51.199335    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ce1df6_13fd_4804_b7a3_16c520bb7898.slice/crio-ca2ca619377db6c00df96ac0e8438dbeaeb97f08f68999778be6f48fbba1d371 WatchSource:0}: Error finding container ca2ca619377db6c00df96ac0e8438dbeaeb97f08f68999778be6f48fbba1d371: Status 404 returned error can't find the container with id ca2ca619377db6c00df96ac0e8438dbeaeb97f08f68999778be6f48fbba1d371
Sep 09 20:21:51.251285 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:51.251253    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6262414-82c7-4fa0-880c-bcd7cfb9ba50-cert\") pod \"kserve-controller-manager-55b9566966-jgss6\" (UID: \"c6262414-82c7-4fa0-880c-bcd7cfb9ba50\") " pod="kserve/kserve-controller-manager-55b9566966-jgss6"
Sep 09 20:21:51.253569 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:51.253550    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6262414-82c7-4fa0-880c-bcd7cfb9ba50-cert\") pod \"kserve-controller-manager-55b9566966-jgss6\" (UID: \"c6262414-82c7-4fa0-880c-bcd7cfb9ba50\") " pod="kserve/kserve-controller-manager-55b9566966-jgss6"
Sep 09 20:21:51.389068 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:51.389036    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-jlltk" event={"ID":"a2ce1df6-13fd-4804-b7a3-16c520bb7898","Type":"ContainerStarted","Data":"ca2ca619377db6c00df96ac0e8438dbeaeb97f08f68999778be6f48fbba1d371"}
Sep 09 20:21:51.487549 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:51.487524    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-55b9566966-jgss6"
Sep 09 20:21:51.758906 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:51.758876    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-jgss6"]
Sep 09 20:21:51.760161 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:21:51.760134    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6262414_82c7_4fa0_880c_bcd7cfb9ba50.slice/crio-0dbc2208be71add3d743f0e436393d47356742c22c03c30d47dfcb289e176e31 WatchSource:0}: Error finding container 0dbc2208be71add3d743f0e436393d47356742c22c03c30d47dfcb289e176e31: Status 404 returned error can't find the container with id 0dbc2208be71add3d743f0e436393d47356742c22c03c30d47dfcb289e176e31
Sep 09 20:21:52.400085 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:52.400048    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-55b9566966-jgss6" event={"ID":"c6262414-82c7-4fa0-880c-bcd7cfb9ba50","Type":"ContainerStarted","Data":"0dbc2208be71add3d743f0e436393d47356742c22c03c30d47dfcb289e176e31"}
Sep 09 20:21:55.418881 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:55.418852    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-55b9566966-jgss6" event={"ID":"c6262414-82c7-4fa0-880c-bcd7cfb9ba50","Type":"ContainerStarted","Data":"40ae0f6eafc79da217a7d6f2089dec0039d8559b7e8c478c5e8f05f08832ad62"}
Sep 09 20:21:55.419193 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:55.418939    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-55b9566966-jgss6"
Sep 09 20:21:55.437734 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:55.437686    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-55b9566966-jgss6" podStartSLOduration=1.92531881 podStartE2EDuration="5.437670687s" totalImagesPullingTime="3.512351877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:21:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:21:55.435268464 +0000 UTC m=+657.440445462" watchObservedRunningTime="2026-09-09 20:21:55.437670687 +0000 UTC m=+657.442847682"
Sep 09 20:21:56.425803 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:56.425762    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-jlltk" event={"ID":"a2ce1df6-13fd-4804-b7a3-16c520bb7898","Type":"ContainerStarted","Data":"46522a2e21a0d612fc84a02e964b72968c9e78aadb8f8d69815fa0eddf76f8ad"}
Sep 09 20:21:56.444476 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:56.444408    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-jlltk" podStartSLOduration=2.28348867 podStartE2EDuration="6.444391554s" totalImagesPullingTime="4.160902884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:21:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:21:56.44270585 +0000 UTC m=+658.447882845" watchObservedRunningTime="2026-09-09 20:21:56.444391554 +0000 UTC m=+658.449568549"
Sep 09 20:21:57.430769 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:57.430737    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-jlltk"
Sep 09 20:21:58.438264 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:21:58.438236    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-jlltk"
Sep 09 20:22:26.431244 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:26.431207    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-55b9566966-jgss6"
Sep 09 20:22:32.430868 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:32.430810    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-jgss6"]
Sep 09 20:22:32.431328 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:32.431030    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-55b9566966-jgss6" podUID="c6262414-82c7-4fa0-880c-bcd7cfb9ba50" containerName="manager" containerID="cri-o://40ae0f6eafc79da217a7d6f2089dec0039d8559b7e8c478c5e8f05f08832ad62" gracePeriod=10
Sep 09 20:22:32.453822 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:32.453798    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-55b9566966-ch2xc"]
Sep 09 20:22:32.455703 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:32.455687    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-55b9566966-ch2xc"
Sep 09 20:22:32.467030 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:32.467005    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-ch2xc"]
Sep 09 20:22:32.510119 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:32.510092    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ceccaadd-9d57-4932-947c-534e33017cad-cert\") pod \"kserve-controller-manager-55b9566966-ch2xc\" (UID: \"ceccaadd-9d57-4932-947c-534e33017cad\") " pod="kserve/kserve-controller-manager-55b9566966-ch2xc"
Sep 09 20:22:32.510221 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:32.510143    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwxbr\" (UniqueName: \"kubernetes.io/projected/ceccaadd-9d57-4932-947c-534e33017cad-kube-api-access-mwxbr\") pod \"kserve-controller-manager-55b9566966-ch2xc\" (UID: \"ceccaadd-9d57-4932-947c-534e33017cad\") " pod="kserve/kserve-controller-manager-55b9566966-ch2xc"
Sep 09 20:22:32.607315 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:32.607287    2581 generic.go:356] "Generic (PLEG): container finished" podID="c6262414-82c7-4fa0-880c-bcd7cfb9ba50" containerID="40ae0f6eafc79da217a7d6f2089dec0039d8559b7e8c478c5e8f05f08832ad62" exitCode=0
Sep 09 20:22:32.607489 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:32.607366    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-55b9566966-jgss6" event={"ID":"c6262414-82c7-4fa0-880c-bcd7cfb9ba50","Type":"ContainerDied","Data":"40ae0f6eafc79da217a7d6f2089dec0039d8559b7e8c478c5e8f05f08832ad62"}
Sep 09 20:22:32.610787 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:32.610756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwxbr\" (UniqueName: \"kubernetes.io/projected/ceccaadd-9d57-4932-947c-534e33017cad-kube-api-access-mwxbr\") pod \"kserve-controller-manager-55b9566966-ch2xc\" (UID: \"ceccaadd-9d57-4932-947c-534e33017cad\") " pod="kserve/kserve-controller-manager-55b9566966-ch2xc"
Sep 09 20:22:32.610925 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:32.610879    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ceccaadd-9d57-4932-947c-534e33017cad-cert\") pod \"kserve-controller-manager-55b9566966-ch2xc\" (UID: \"ceccaadd-9d57-4932-947c-534e33017cad\") " pod="kserve/kserve-controller-manager-55b9566966-ch2xc"
Sep 09 20:22:32.613508 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:32.613480    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ceccaadd-9d57-4932-947c-534e33017cad-cert\") pod \"kserve-controller-manager-55b9566966-ch2xc\" (UID: \"ceccaadd-9d57-4932-947c-534e33017cad\") " pod="kserve/kserve-controller-manager-55b9566966-ch2xc"
Sep 09 20:22:32.620303 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:32.620278    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwxbr\" (UniqueName: \"kubernetes.io/projected/ceccaadd-9d57-4932-947c-534e33017cad-kube-api-access-mwxbr\") pod \"kserve-controller-manager-55b9566966-ch2xc\" (UID: \"ceccaadd-9d57-4932-947c-534e33017cad\") " pod="kserve/kserve-controller-manager-55b9566966-ch2xc"
Sep 09 20:22:32.671633 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:32.671612    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-55b9566966-jgss6"
Sep 09 20:22:32.711986 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:32.711914    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hqfk\" (UniqueName: \"kubernetes.io/projected/c6262414-82c7-4fa0-880c-bcd7cfb9ba50-kube-api-access-6hqfk\") pod \"c6262414-82c7-4fa0-880c-bcd7cfb9ba50\" (UID: \"c6262414-82c7-4fa0-880c-bcd7cfb9ba50\") "
Sep 09 20:22:32.712107 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:32.712024    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6262414-82c7-4fa0-880c-bcd7cfb9ba50-cert\") pod \"c6262414-82c7-4fa0-880c-bcd7cfb9ba50\" (UID: \"c6262414-82c7-4fa0-880c-bcd7cfb9ba50\") "
Sep 09 20:22:32.714030 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:32.714002    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6262414-82c7-4fa0-880c-bcd7cfb9ba50-cert" (OuterVolumeSpecName: "cert") pod "c6262414-82c7-4fa0-880c-bcd7cfb9ba50" (UID: "c6262414-82c7-4fa0-880c-bcd7cfb9ba50"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:22:32.714140 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:32.714054    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6262414-82c7-4fa0-880c-bcd7cfb9ba50-kube-api-access-6hqfk" (OuterVolumeSpecName: "kube-api-access-6hqfk") pod "c6262414-82c7-4fa0-880c-bcd7cfb9ba50" (UID: "c6262414-82c7-4fa0-880c-bcd7cfb9ba50"). InnerVolumeSpecName "kube-api-access-6hqfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:22:32.811135 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:32.811099    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-55b9566966-ch2xc"
Sep 09 20:22:32.812868 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:32.812711    2581 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6262414-82c7-4fa0-880c-bcd7cfb9ba50-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:22:32.812868 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:32.812736    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hqfk\" (UniqueName: \"kubernetes.io/projected/c6262414-82c7-4fa0-880c-bcd7cfb9ba50-kube-api-access-6hqfk\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:22:33.078641 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:33.078584    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-ch2xc"]
Sep 09 20:22:33.080852 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:22:33.080825    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceccaadd_9d57_4932_947c_534e33017cad.slice/crio-5cbd821506ab1aca7349813fae6a0f5bedb3f538d8040990906ba22a18be6ced WatchSource:0}: Error finding container 5cbd821506ab1aca7349813fae6a0f5bedb3f538d8040990906ba22a18be6ced: Status 404 returned error can't find the container with id 5cbd821506ab1aca7349813fae6a0f5bedb3f538d8040990906ba22a18be6ced
Sep 09 20:22:33.082167 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:33.082151    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:22:33.614109 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:33.614059    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-55b9566966-ch2xc" event={"ID":"ceccaadd-9d57-4932-947c-534e33017cad","Type":"ContainerStarted","Data":"0d055fb4fa56bc92ead8c4a86251e3ef5af5a65c9e0b8a477bac9617e0d814cb"}
Sep 09 20:22:33.614556 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:33.614138    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-55b9566966-ch2xc"
Sep 09 20:22:33.614556 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:33.614163    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-55b9566966-ch2xc" event={"ID":"ceccaadd-9d57-4932-947c-534e33017cad","Type":"ContainerStarted","Data":"5cbd821506ab1aca7349813fae6a0f5bedb3f538d8040990906ba22a18be6ced"}
Sep 09 20:22:33.615360 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:33.615343    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-55b9566966-jgss6"
Sep 09 20:22:33.615443 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:33.615341    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-55b9566966-jgss6" event={"ID":"c6262414-82c7-4fa0-880c-bcd7cfb9ba50","Type":"ContainerDied","Data":"0dbc2208be71add3d743f0e436393d47356742c22c03c30d47dfcb289e176e31"}
Sep 09 20:22:33.615650 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:33.615488    2581 scope.go:117] "RemoveContainer" containerID="40ae0f6eafc79da217a7d6f2089dec0039d8559b7e8c478c5e8f05f08832ad62"
Sep 09 20:22:33.631923 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:33.631869    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-55b9566966-ch2xc" podStartSLOduration=1.232170672 podStartE2EDuration="1.631850074s" totalImagesPullingTime="399.679402ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:22:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:22:33.629126726 +0000 UTC m=+695.634303720" watchObservedRunningTime="2026-09-09 20:22:33.631850074 +0000 UTC m=+695.637027071"
Sep 09 20:22:33.647925 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:33.647902    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-jgss6"]
Sep 09 20:22:33.651951 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:33.651928    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-55b9566966-jgss6"]
Sep 09 20:22:34.634727 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:22:34.634690    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6262414-82c7-4fa0-880c-bcd7cfb9ba50" path="/var/lib/kubelet/pods/c6262414-82c7-4fa0-880c-bcd7cfb9ba50/volumes"
Sep 09 20:23:04.626519 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:04.626485    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-55b9566966-ch2xc"
Sep 09 20:23:05.930003 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:05.929970    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-d6sfs"]
Sep 09 20:23:05.930449 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:05.930423    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6262414-82c7-4fa0-880c-bcd7cfb9ba50" containerName="manager"
Sep 09 20:23:05.930449 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:05.930440    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6262414-82c7-4fa0-880c-bcd7cfb9ba50" containerName="manager"
Sep 09 20:23:05.930596 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:05.930585    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6262414-82c7-4fa0-880c-bcd7cfb9ba50" containerName="manager"
Sep 09 20:23:05.934985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:05.934961    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-d6sfs"
Sep 09 20:23:05.937282 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:05.937256    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 09 20:23:05.937434 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:05.937350    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 09 20:23:05.937547 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:05.937529    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-5pfcd\""
Sep 09 20:23:05.941301 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:05.941279    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-d6sfs"]
Sep 09 20:23:06.068266 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:06.068235    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d84788dd-b7ed-43bc-affd-9b982be11af5-cert\") pod \"odh-model-controller-7b966d7984-d6sfs\" (UID: \"d84788dd-b7ed-43bc-affd-9b982be11af5\") " pod="kserve/odh-model-controller-7b966d7984-d6sfs"
Sep 09 20:23:06.068435 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:06.068279    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d84788dd-b7ed-43bc-affd-9b982be11af5-metrics-certs\") pod \"odh-model-controller-7b966d7984-d6sfs\" (UID: \"d84788dd-b7ed-43bc-affd-9b982be11af5\") " pod="kserve/odh-model-controller-7b966d7984-d6sfs"
Sep 09 20:23:06.068435 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:06.068302    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85glb\" (UniqueName: \"kubernetes.io/projected/d84788dd-b7ed-43bc-affd-9b982be11af5-kube-api-access-85glb\") pod \"odh-model-controller-7b966d7984-d6sfs\" (UID: \"d84788dd-b7ed-43bc-affd-9b982be11af5\") " pod="kserve/odh-model-controller-7b966d7984-d6sfs"
Sep 09 20:23:06.168932 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:06.168900    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85glb\" (UniqueName: \"kubernetes.io/projected/d84788dd-b7ed-43bc-affd-9b982be11af5-kube-api-access-85glb\") pod \"odh-model-controller-7b966d7984-d6sfs\" (UID: \"d84788dd-b7ed-43bc-affd-9b982be11af5\") " pod="kserve/odh-model-controller-7b966d7984-d6sfs"
Sep 09 20:23:06.169123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:06.168966    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d84788dd-b7ed-43bc-affd-9b982be11af5-cert\") pod \"odh-model-controller-7b966d7984-d6sfs\" (UID: \"d84788dd-b7ed-43bc-affd-9b982be11af5\") " pod="kserve/odh-model-controller-7b966d7984-d6sfs"
Sep 09 20:23:06.169123 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:06.169016    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d84788dd-b7ed-43bc-affd-9b982be11af5-metrics-certs\") pod \"odh-model-controller-7b966d7984-d6sfs\" (UID: \"d84788dd-b7ed-43bc-affd-9b982be11af5\") " pod="kserve/odh-model-controller-7b966d7984-d6sfs"
Sep 09 20:23:06.169123 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:23:06.169070    2581 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 09 20:23:06.169282 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:23:06.169146    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d84788dd-b7ed-43bc-affd-9b982be11af5-cert podName:d84788dd-b7ed-43bc-affd-9b982be11af5 nodeName:}" failed. No retries permitted until 2026-09-09 20:23:06.66912372 +0000 UTC m=+728.674300696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d84788dd-b7ed-43bc-affd-9b982be11af5-cert") pod "odh-model-controller-7b966d7984-d6sfs" (UID: "d84788dd-b7ed-43bc-affd-9b982be11af5") : secret "odh-model-controller-webhook-cert" not found
Sep 09 20:23:06.171418 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:06.171396    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d84788dd-b7ed-43bc-affd-9b982be11af5-metrics-certs\") pod \"odh-model-controller-7b966d7984-d6sfs\" (UID: \"d84788dd-b7ed-43bc-affd-9b982be11af5\") " pod="kserve/odh-model-controller-7b966d7984-d6sfs"
Sep 09 20:23:06.177589 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:06.177568    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85glb\" (UniqueName: \"kubernetes.io/projected/d84788dd-b7ed-43bc-affd-9b982be11af5-kube-api-access-85glb\") pod \"odh-model-controller-7b966d7984-d6sfs\" (UID: \"d84788dd-b7ed-43bc-affd-9b982be11af5\") " pod="kserve/odh-model-controller-7b966d7984-d6sfs"
Sep 09 20:23:06.673478 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:06.673415    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d84788dd-b7ed-43bc-affd-9b982be11af5-cert\") pod \"odh-model-controller-7b966d7984-d6sfs\" (UID: \"d84788dd-b7ed-43bc-affd-9b982be11af5\") " pod="kserve/odh-model-controller-7b966d7984-d6sfs"
Sep 09 20:23:06.675798 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:06.675768    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d84788dd-b7ed-43bc-affd-9b982be11af5-cert\") pod \"odh-model-controller-7b966d7984-d6sfs\" (UID: \"d84788dd-b7ed-43bc-affd-9b982be11af5\") " pod="kserve/odh-model-controller-7b966d7984-d6sfs"
Sep 09 20:23:06.847120 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:06.847089    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-d6sfs"
Sep 09 20:23:07.118646 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:07.118622    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-d6sfs"]
Sep 09 20:23:07.120595 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:23:07.120570    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd84788dd_b7ed_43bc_affd_9b982be11af5.slice/crio-b73d21843b8b38d46a4be999bb533118d3f97832ebe8b54d5b35960fa7eb7946 WatchSource:0}: Error finding container b73d21843b8b38d46a4be999bb533118d3f97832ebe8b54d5b35960fa7eb7946: Status 404 returned error can't find the container with id b73d21843b8b38d46a4be999bb533118d3f97832ebe8b54d5b35960fa7eb7946
Sep 09 20:23:07.782142 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:07.782100    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-d6sfs" event={"ID":"d84788dd-b7ed-43bc-affd-9b982be11af5","Type":"ContainerStarted","Data":"b73d21843b8b38d46a4be999bb533118d3f97832ebe8b54d5b35960fa7eb7946"}
Sep 09 20:23:09.794662 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:09.794624    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-d6sfs" event={"ID":"d84788dd-b7ed-43bc-affd-9b982be11af5","Type":"ContainerStarted","Data":"46825e15a2a9d7e36953131562eb470ef66872317ae186e2738199a7e4271c6e"}
Sep 09 20:23:09.795094 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:09.794684    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-d6sfs"
Sep 09 20:23:09.814353 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:09.814299    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-d6sfs" podStartSLOduration=2.33406009 podStartE2EDuration="4.81428331s" totalImagesPullingTime="2.48022322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:23:09.812052164 +0000 UTC m=+731.817229160" watchObservedRunningTime="2026-09-09 20:23:09.81428331 +0000 UTC m=+731.819460305"
Sep 09 20:23:20.802545 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:23:20.802515    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-d6sfs"
Sep 09 20:25:08.751610 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:08.751569    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 20:25:08.752003 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:08.751638    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4" podUID="4e089356-1131-499c-a8e5-afe9c2c9b0c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 20:25:10.398018 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:10.397986    2581 generic.go:356] "Generic (PLEG): container finished" podID="4e089356-1131-499c-a8e5-afe9c2c9b0c6" containerID="b4d08c412670293248497e6838e25fead94426d227a4082f8b0534290deb0c79" exitCode=0
Sep 09 20:25:10.398412 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:10.398062    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4" event={"ID":"4e089356-1131-499c-a8e5-afe9c2c9b0c6","Type":"ContainerDied","Data":"b4d08c412670293248497e6838e25fead94426d227a4082f8b0534290deb0c79"}
Sep 09 20:25:10.398455 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:10.398422    2581 scope.go:117] "RemoveContainer" containerID="b4d08c412670293248497e6838e25fead94426d227a4082f8b0534290deb0c79"
Sep 09 20:25:10.399333 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:10.399304    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:25:10.399406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:10.399389    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:25:10.399451 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:10.399437    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:25:11.405070 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:11.405041    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4" event={"ID":"4e089356-1131-499c-a8e5-afe9c2c9b0c6","Type":"ContainerStarted","Data":"9a70909648159f8230875c35d870834a004ae2626e8d503dce8c01aebb2cdc01"}
Sep 09 20:25:12.350220 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:12.350176    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:25:12.350825 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:12.350795    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 09 20:25:12.350953 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:12.350858    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4" podUID="4e089356-1131-499c-a8e5-afe9c2c9b0c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 09 20:25:13.350073 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:13.350028    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 09 20:25:13.350590 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:13.350098    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4" podUID="4e089356-1131-499c-a8e5-afe9c2c9b0c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 09 20:25:14.350333 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:14.350294    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 09 20:25:14.350967 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:14.350935    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4" podUID="4e089356-1131-499c-a8e5-afe9c2c9b0c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 09 20:25:15.350227 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:15.350185    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 09 20:25:15.350633 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:15.350602    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4" podUID="4e089356-1131-499c-a8e5-afe9c2c9b0c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 09 20:25:16.354863 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:16.354813    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:25:16.355315 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:16.355092    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:25:16.355791 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:16.355773    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4"
Sep 09 20:25:25.561202 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.561166    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"]
Sep 09 20:25:25.563573 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.563554    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:25:25.567240 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.567216    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 09 20:25:25.567342 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.567243    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 09 20:25:25.567396 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.567362    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-f6phs\""
Sep 09 20:25:25.567530 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.567513    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 09 20:25:25.574535 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.574508    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"]
Sep 09 20:25:25.726739 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.726705    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe0c858-0e7f-441a-915b-14a7b5f488d8-dshm\") pod \"fc-smoke-test-kserve-7f4687897-p5w4d\" (UID: \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:25:25.726893 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.726750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klfg4\" (UniqueName: \"kubernetes.io/projected/7fe0c858-0e7f-441a-915b-14a7b5f488d8-kube-api-access-klfg4\") pod \"fc-smoke-test-kserve-7f4687897-p5w4d\" (UID: \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:25:25.726893 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.726777    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe0c858-0e7f-441a-915b-14a7b5f488d8-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-p5w4d\" (UID: \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:25:25.726893 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.726830    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe0c858-0e7f-441a-915b-14a7b5f488d8-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-p5w4d\" (UID: \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:25:25.726893 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.726850    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe0c858-0e7f-441a-915b-14a7b5f488d8-home\") pod \"fc-smoke-test-kserve-7f4687897-p5w4d\" (UID: \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:25:25.726893 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.726868    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe0c858-0e7f-441a-915b-14a7b5f488d8-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-p5w4d\" (UID: \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:25:25.827891 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.827809    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe0c858-0e7f-441a-915b-14a7b5f488d8-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-p5w4d\" (UID: \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:25:25.827891 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.827851    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe0c858-0e7f-441a-915b-14a7b5f488d8-home\") pod \"fc-smoke-test-kserve-7f4687897-p5w4d\" (UID: \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:25:25.827891 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.827867    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe0c858-0e7f-441a-915b-14a7b5f488d8-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-p5w4d\" (UID: \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:25:25.828138 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.827912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe0c858-0e7f-441a-915b-14a7b5f488d8-dshm\") pod \"fc-smoke-test-kserve-7f4687897-p5w4d\" (UID: \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:25:25.828138 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.827940    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klfg4\" (UniqueName: \"kubernetes.io/projected/7fe0c858-0e7f-441a-915b-14a7b5f488d8-kube-api-access-klfg4\") pod \"fc-smoke-test-kserve-7f4687897-p5w4d\" (UID: \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:25:25.828138 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.827966    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe0c858-0e7f-441a-915b-14a7b5f488d8-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-p5w4d\" (UID: \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:25:25.828285 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.828250    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe0c858-0e7f-441a-915b-14a7b5f488d8-home\") pod \"fc-smoke-test-kserve-7f4687897-p5w4d\" (UID: \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:25:25.828332 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.828259    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe0c858-0e7f-441a-915b-14a7b5f488d8-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-p5w4d\" (UID: \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:25:25.828367 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.828328    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe0c858-0e7f-441a-915b-14a7b5f488d8-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-p5w4d\" (UID: \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:25:25.830235 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.830206    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe0c858-0e7f-441a-915b-14a7b5f488d8-dshm\") pod \"fc-smoke-test-kserve-7f4687897-p5w4d\" (UID: \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:25:25.830549 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.830529    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe0c858-0e7f-441a-915b-14a7b5f488d8-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-p5w4d\" (UID: \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:25:25.835670 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.835651    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klfg4\" (UniqueName: \"kubernetes.io/projected/7fe0c858-0e7f-441a-915b-14a7b5f488d8-kube-api-access-klfg4\") pod \"fc-smoke-test-kserve-7f4687897-p5w4d\" (UID: \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:25:25.874175 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:25.874146    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:25:26.149745 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:26.149603    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"]
Sep 09 20:25:26.151944 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:25:26.151905    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe0c858_0e7f_441a_915b_14a7b5f488d8.slice/crio-787cf6af49557c00d7f873fd03bc48d04510e82c933033287485d310b13ab573 WatchSource:0}: Error finding container 787cf6af49557c00d7f873fd03bc48d04510e82c933033287485d310b13ab573: Status 404 returned error can't find the container with id 787cf6af49557c00d7f873fd03bc48d04510e82c933033287485d310b13ab573
Sep 09 20:25:26.485818 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:26.485786    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d" event={"ID":"7fe0c858-0e7f-441a-915b-14a7b5f488d8","Type":"ContainerStarted","Data":"787cf6af49557c00d7f873fd03bc48d04510e82c933033287485d310b13ab573"}
Sep 09 20:25:28.499837 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:28.499804    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d" event={"ID":"7fe0c858-0e7f-441a-915b-14a7b5f488d8","Type":"ContainerStarted","Data":"9f5eb1882d1c54c70f995e43fb97bbc8f80deb37580e57edaa196f8be49f121a"}
Sep 09 20:25:28.518829 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:28.518774    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d" podStartSLOduration=2.05802746 podStartE2EDuration="3.518758599s" totalImagesPullingTime="1.460731139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:25:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:25:28.516858499 +0000 UTC m=+870.522035493" watchObservedRunningTime="2026-09-09 20:25:28.518758599 +0000 UTC m=+870.523935605"
Sep 09 20:25:28.874666 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:28.874584    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:25:35.874922 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:35.874884    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:25:35.887627 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:35.887601    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:25:36.549366 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:36.549336    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:25:58.453718 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:58.453687    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 20:25:58.455803 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:58.455781    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 20:25:58.456582 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:58.456551    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 20:25:58.458554 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:25:58.458534    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 20:26:10.390802 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.390765    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"]
Sep 09 20:26:10.399736 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:26:10.399698    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:26:10.399736 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:26:10.399736 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:26:10.399736 ip-10-0-134-84 kubenswrapper[2581]:  > containerID="9f5eb1882d1c54c70f995e43fb97bbc8f80deb37580e57edaa196f8be49f121a" cmd=["/bin/sleep","15"]
Sep 09 20:26:10.400159 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.399763    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d" message="" err=<
Sep 09 20:26:10.400159 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:26:10.400159 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:26:10.400159 ip-10-0-134-84 kubenswrapper[2581]:  >
Sep 09 20:26:10.400159 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:26:10.399780    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:26:10.400159 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:26:10.400159 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:26:10.400159 ip-10-0-134-84 kubenswrapper[2581]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d" podUID="7fe0c858-0e7f-441a-915b-14a7b5f488d8" containerName="main" containerID="cri-o://9f5eb1882d1c54c70f995e43fb97bbc8f80deb37580e57edaa196f8be49f121a"
Sep 09 20:26:10.400159 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.399835    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d" podUID="7fe0c858-0e7f-441a-915b-14a7b5f488d8" containerName="main" containerID="cri-o://9f5eb1882d1c54c70f995e43fb97bbc8f80deb37580e57edaa196f8be49f121a" gracePeriod=60
Sep 09 20:26:10.538905 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.538870    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d" podUID="7fe0c858-0e7f-441a-915b-14a7b5f488d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Sep 09 20:26:10.712022 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.711992    2581 generic.go:356] "Generic (PLEG): container finished" podID="7fe0c858-0e7f-441a-915b-14a7b5f488d8" containerID="9f5eb1882d1c54c70f995e43fb97bbc8f80deb37580e57edaa196f8be49f121a" exitCode=0
Sep 09 20:26:10.712202 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.712068    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d" event={"ID":"7fe0c858-0e7f-441a-915b-14a7b5f488d8","Type":"ContainerDied","Data":"9f5eb1882d1c54c70f995e43fb97bbc8f80deb37580e57edaa196f8be49f121a"}
Sep 09 20:26:10.748224 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.748201    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:26:10.781550 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.781524    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe0c858-0e7f-441a-915b-14a7b5f488d8-home\") pod \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\" (UID: \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\") "
Sep 09 20:26:10.781717 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.781593    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe0c858-0e7f-441a-915b-14a7b5f488d8-model-cache\") pod \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\" (UID: \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\") "
Sep 09 20:26:10.781717 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.781640    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe0c858-0e7f-441a-915b-14a7b5f488d8-dshm\") pod \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\" (UID: \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\") "
Sep 09 20:26:10.781717 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.781673    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe0c858-0e7f-441a-915b-14a7b5f488d8-tmp-dir\") pod \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\" (UID: \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\") "
Sep 09 20:26:10.781881 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.781717    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe0c858-0e7f-441a-915b-14a7b5f488d8-tls-certs\") pod \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\" (UID: \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\") "
Sep 09 20:26:10.781881 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.781764    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klfg4\" (UniqueName: \"kubernetes.io/projected/7fe0c858-0e7f-441a-915b-14a7b5f488d8-kube-api-access-klfg4\") pod \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\" (UID: \"7fe0c858-0e7f-441a-915b-14a7b5f488d8\") "
Sep 09 20:26:10.781881 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.781816    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe0c858-0e7f-441a-915b-14a7b5f488d8-home" (OuterVolumeSpecName: "home") pod "7fe0c858-0e7f-441a-915b-14a7b5f488d8" (UID: "7fe0c858-0e7f-441a-915b-14a7b5f488d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:26:10.781881 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.781826    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe0c858-0e7f-441a-915b-14a7b5f488d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "7fe0c858-0e7f-441a-915b-14a7b5f488d8" (UID: "7fe0c858-0e7f-441a-915b-14a7b5f488d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:26:10.782068 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.782000    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe0c858-0e7f-441a-915b-14a7b5f488d8-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:26:10.782068 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.782016    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe0c858-0e7f-441a-915b-14a7b5f488d8-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:26:10.782143 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.782057    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe0c858-0e7f-441a-915b-14a7b5f488d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7fe0c858-0e7f-441a-915b-14a7b5f488d8" (UID: "7fe0c858-0e7f-441a-915b-14a7b5f488d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:26:10.783857 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.783825    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe0c858-0e7f-441a-915b-14a7b5f488d8-kube-api-access-klfg4" (OuterVolumeSpecName: "kube-api-access-klfg4") pod "7fe0c858-0e7f-441a-915b-14a7b5f488d8" (UID: "7fe0c858-0e7f-441a-915b-14a7b5f488d8"). InnerVolumeSpecName "kube-api-access-klfg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:26:10.783972 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.783909    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe0c858-0e7f-441a-915b-14a7b5f488d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7fe0c858-0e7f-441a-915b-14a7b5f488d8" (UID: "7fe0c858-0e7f-441a-915b-14a7b5f488d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:26:10.784045 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.783995    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe0c858-0e7f-441a-915b-14a7b5f488d8-dshm" (OuterVolumeSpecName: "dshm") pod "7fe0c858-0e7f-441a-915b-14a7b5f488d8" (UID: "7fe0c858-0e7f-441a-915b-14a7b5f488d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:26:10.883188 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.883153    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klfg4\" (UniqueName: \"kubernetes.io/projected/7fe0c858-0e7f-441a-915b-14a7b5f488d8-kube-api-access-klfg4\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:26:10.883188 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.883181    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe0c858-0e7f-441a-915b-14a7b5f488d8-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:26:10.883365 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.883211    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe0c858-0e7f-441a-915b-14a7b5f488d8-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:26:10.883365 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:10.883219    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe0c858-0e7f-441a-915b-14a7b5f488d8-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:26:11.718025 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:11.717997    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"
Sep 09 20:26:11.718523 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:11.718024    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d" event={"ID":"7fe0c858-0e7f-441a-915b-14a7b5f488d8","Type":"ContainerDied","Data":"787cf6af49557c00d7f873fd03bc48d04510e82c933033287485d310b13ab573"}
Sep 09 20:26:11.718523 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:11.718090    2581 scope.go:117] "RemoveContainer" containerID="9f5eb1882d1c54c70f995e43fb97bbc8f80deb37580e57edaa196f8be49f121a"
Sep 09 20:26:11.739922 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:11.739899    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"]
Sep 09 20:26:11.744108 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:11.744083    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p5w4d"]
Sep 09 20:26:12.634931 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:26:12.634902    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fe0c858-0e7f-441a-915b-14a7b5f488d8" path="/var/lib/kubelet/pods/7fe0c858-0e7f-441a-915b-14a7b5f488d8/volumes"
Sep 09 20:27:08.745564 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.745524    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"]
Sep 09 20:27:08.746065 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.745995    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fe0c858-0e7f-441a-915b-14a7b5f488d8" containerName="main"
Sep 09 20:27:08.746065 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.746015    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fe0c858-0e7f-441a-915b-14a7b5f488d8" containerName="main"
Sep 09 20:27:08.746178 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.746102    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fe0c858-0e7f-441a-915b-14a7b5f488d8" containerName="main"
Sep 09 20:27:08.750067 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.750045    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:08.753301 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.753280    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 09 20:27:08.753301 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.753292    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 09 20:27:08.753469 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.753280    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 09 20:27:08.753469 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.753282    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-4pdh4\""
Sep 09 20:27:08.759872 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.759849    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"]
Sep 09 20:27:08.871850 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.871819    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0acea23c-8cb0-4814-95b9-9117ca767681-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-g4j2h\" (UID: \"0acea23c-8cb0-4814-95b9-9117ca767681\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:08.871850 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.871853    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0acea23c-8cb0-4814-95b9-9117ca767681-home\") pod \"fc-concurrency-test-kserve-b5d44467c-g4j2h\" (UID: \"0acea23c-8cb0-4814-95b9-9117ca767681\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:08.872057 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.871881    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0acea23c-8cb0-4814-95b9-9117ca767681-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-g4j2h\" (UID: \"0acea23c-8cb0-4814-95b9-9117ca767681\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:08.872057 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.871902    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpwq6\" (UniqueName: \"kubernetes.io/projected/0acea23c-8cb0-4814-95b9-9117ca767681-kube-api-access-kpwq6\") pod \"fc-concurrency-test-kserve-b5d44467c-g4j2h\" (UID: \"0acea23c-8cb0-4814-95b9-9117ca767681\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:08.872057 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.871918    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0acea23c-8cb0-4814-95b9-9117ca767681-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-g4j2h\" (UID: \"0acea23c-8cb0-4814-95b9-9117ca767681\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:08.872057 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.871960    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0acea23c-8cb0-4814-95b9-9117ca767681-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-g4j2h\" (UID: \"0acea23c-8cb0-4814-95b9-9117ca767681\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:08.972349 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.972319    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0acea23c-8cb0-4814-95b9-9117ca767681-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-g4j2h\" (UID: \"0acea23c-8cb0-4814-95b9-9117ca767681\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:08.972527 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.972356    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0acea23c-8cb0-4814-95b9-9117ca767681-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-g4j2h\" (UID: \"0acea23c-8cb0-4814-95b9-9117ca767681\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:08.972527 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.972382    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0acea23c-8cb0-4814-95b9-9117ca767681-home\") pod \"fc-concurrency-test-kserve-b5d44467c-g4j2h\" (UID: \"0acea23c-8cb0-4814-95b9-9117ca767681\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:08.972527 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.972421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0acea23c-8cb0-4814-95b9-9117ca767681-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-g4j2h\" (UID: \"0acea23c-8cb0-4814-95b9-9117ca767681\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:08.972527 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.972454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpwq6\" (UniqueName: \"kubernetes.io/projected/0acea23c-8cb0-4814-95b9-9117ca767681-kube-api-access-kpwq6\") pod \"fc-concurrency-test-kserve-b5d44467c-g4j2h\" (UID: \"0acea23c-8cb0-4814-95b9-9117ca767681\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:08.972527 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.972491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0acea23c-8cb0-4814-95b9-9117ca767681-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-g4j2h\" (UID: \"0acea23c-8cb0-4814-95b9-9117ca767681\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:08.972804 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.972766    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0acea23c-8cb0-4814-95b9-9117ca767681-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-g4j2h\" (UID: \"0acea23c-8cb0-4814-95b9-9117ca767681\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:08.972858 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.972803    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0acea23c-8cb0-4814-95b9-9117ca767681-home\") pod \"fc-concurrency-test-kserve-b5d44467c-g4j2h\" (UID: \"0acea23c-8cb0-4814-95b9-9117ca767681\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:08.972904 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.972859    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0acea23c-8cb0-4814-95b9-9117ca767681-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-g4j2h\" (UID: \"0acea23c-8cb0-4814-95b9-9117ca767681\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:08.974561 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.974534    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0acea23c-8cb0-4814-95b9-9117ca767681-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-g4j2h\" (UID: \"0acea23c-8cb0-4814-95b9-9117ca767681\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:08.974937 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.974918    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0acea23c-8cb0-4814-95b9-9117ca767681-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-g4j2h\" (UID: \"0acea23c-8cb0-4814-95b9-9117ca767681\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:08.981561 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:08.981542    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpwq6\" (UniqueName: \"kubernetes.io/projected/0acea23c-8cb0-4814-95b9-9117ca767681-kube-api-access-kpwq6\") pod \"fc-concurrency-test-kserve-b5d44467c-g4j2h\" (UID: \"0acea23c-8cb0-4814-95b9-9117ca767681\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:09.061111 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:09.061031    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:09.331415 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:09.331387    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"]
Sep 09 20:27:09.333228 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:27:09.333201    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0acea23c_8cb0_4814_95b9_9117ca767681.slice/crio-cc5a1491322b665ffe7761e668a66329a83b6595a3f0f1f02fc0b6722ac0e794 WatchSource:0}: Error finding container cc5a1491322b665ffe7761e668a66329a83b6595a3f0f1f02fc0b6722ac0e794: Status 404 returned error can't find the container with id cc5a1491322b665ffe7761e668a66329a83b6595a3f0f1f02fc0b6722ac0e794
Sep 09 20:27:10.008638 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:10.008602    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h" event={"ID":"0acea23c-8cb0-4814-95b9-9117ca767681","Type":"ContainerStarted","Data":"2a77c52057fb347c46860e03d7b89a363022d5b06357eb5ec8696ea846f90d33"}
Sep 09 20:27:10.009016 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:10.008657    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h" event={"ID":"0acea23c-8cb0-4814-95b9-9117ca767681","Type":"ContainerStarted","Data":"cc5a1491322b665ffe7761e668a66329a83b6595a3f0f1f02fc0b6722ac0e794"}
Sep 09 20:27:10.026108 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:10.026051    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h" podStartSLOduration=2.026030484 podStartE2EDuration="2.026030484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:27:10.024899771 +0000 UTC m=+972.030076766" watchObservedRunningTime="2026-09-09 20:27:10.026030484 +0000 UTC m=+972.031207479"
Sep 09 20:27:10.062138 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:10.062106    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:19.061783 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:19.061746    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:19.074415 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:19.074393    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:20.072861 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:20.072830    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:55.754162 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:55.754130    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"]
Sep 09 20:27:55.764939 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:27:55.764899    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:27:55.764939 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:27:55.764939 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:27:55.764939 ip-10-0-134-84 kubenswrapper[2581]:  > containerID="2a77c52057fb347c46860e03d7b89a363022d5b06357eb5ec8696ea846f90d33" cmd=["/bin/sleep","15"]
Sep 09 20:27:55.765160 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:55.764964    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h" message="" err=<
Sep 09 20:27:55.765160 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:27:55.765160 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:27:55.765160 ip-10-0-134-84 kubenswrapper[2581]:  >
Sep 09 20:27:55.765160 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:27:55.764980    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:27:55.765160 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:27:55.765160 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:27:55.765160 ip-10-0-134-84 kubenswrapper[2581]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h" podUID="0acea23c-8cb0-4814-95b9-9117ca767681" containerName="main" containerID="cri-o://2a77c52057fb347c46860e03d7b89a363022d5b06357eb5ec8696ea846f90d33"
Sep 09 20:27:55.765160 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:55.765052    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h" podUID="0acea23c-8cb0-4814-95b9-9117ca767681" containerName="main" containerID="cri-o://2a77c52057fb347c46860e03d7b89a363022d5b06357eb5ec8696ea846f90d33" gracePeriod=60
Sep 09 20:27:56.013798 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.013742    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:56.057710 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.057682    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0acea23c-8cb0-4814-95b9-9117ca767681-model-cache\") pod \"0acea23c-8cb0-4814-95b9-9117ca767681\" (UID: \"0acea23c-8cb0-4814-95b9-9117ca767681\") "
Sep 09 20:27:56.057858 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.057744    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpwq6\" (UniqueName: \"kubernetes.io/projected/0acea23c-8cb0-4814-95b9-9117ca767681-kube-api-access-kpwq6\") pod \"0acea23c-8cb0-4814-95b9-9117ca767681\" (UID: \"0acea23c-8cb0-4814-95b9-9117ca767681\") "
Sep 09 20:27:56.057858 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.057811    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0acea23c-8cb0-4814-95b9-9117ca767681-dshm\") pod \"0acea23c-8cb0-4814-95b9-9117ca767681\" (UID: \"0acea23c-8cb0-4814-95b9-9117ca767681\") "
Sep 09 20:27:56.057858 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.057852    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0acea23c-8cb0-4814-95b9-9117ca767681-tls-certs\") pod \"0acea23c-8cb0-4814-95b9-9117ca767681\" (UID: \"0acea23c-8cb0-4814-95b9-9117ca767681\") "
Sep 09 20:27:56.058032 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.057879    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0acea23c-8cb0-4814-95b9-9117ca767681-tmp-dir\") pod \"0acea23c-8cb0-4814-95b9-9117ca767681\" (UID: \"0acea23c-8cb0-4814-95b9-9117ca767681\") "
Sep 09 20:27:56.058032 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.057968    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0acea23c-8cb0-4814-95b9-9117ca767681-model-cache" (OuterVolumeSpecName: "model-cache") pod "0acea23c-8cb0-4814-95b9-9117ca767681" (UID: "0acea23c-8cb0-4814-95b9-9117ca767681"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:27:56.058032 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.057974    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0acea23c-8cb0-4814-95b9-9117ca767681-home\") pod \"0acea23c-8cb0-4814-95b9-9117ca767681\" (UID: \"0acea23c-8cb0-4814-95b9-9117ca767681\") "
Sep 09 20:27:56.058182 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.058124    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0acea23c-8cb0-4814-95b9-9117ca767681-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0acea23c-8cb0-4814-95b9-9117ca767681" (UID: "0acea23c-8cb0-4814-95b9-9117ca767681"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:27:56.058182 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.058152    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0acea23c-8cb0-4814-95b9-9117ca767681-home" (OuterVolumeSpecName: "home") pod "0acea23c-8cb0-4814-95b9-9117ca767681" (UID: "0acea23c-8cb0-4814-95b9-9117ca767681"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:27:56.058362 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.058337    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0acea23c-8cb0-4814-95b9-9117ca767681-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:27:56.058362 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.058363    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0acea23c-8cb0-4814-95b9-9117ca767681-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:27:56.058586 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.058376    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0acea23c-8cb0-4814-95b9-9117ca767681-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:27:56.060176 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.060142    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0acea23c-8cb0-4814-95b9-9117ca767681-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0acea23c-8cb0-4814-95b9-9117ca767681" (UID: "0acea23c-8cb0-4814-95b9-9117ca767681"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:27:56.060368 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.060351    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0acea23c-8cb0-4814-95b9-9117ca767681-dshm" (OuterVolumeSpecName: "dshm") pod "0acea23c-8cb0-4814-95b9-9117ca767681" (UID: "0acea23c-8cb0-4814-95b9-9117ca767681"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:27:56.060541 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.060524    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0acea23c-8cb0-4814-95b9-9117ca767681-kube-api-access-kpwq6" (OuterVolumeSpecName: "kube-api-access-kpwq6") pod "0acea23c-8cb0-4814-95b9-9117ca767681" (UID: "0acea23c-8cb0-4814-95b9-9117ca767681"). InnerVolumeSpecName "kube-api-access-kpwq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:27:56.159430 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.159398    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpwq6\" (UniqueName: \"kubernetes.io/projected/0acea23c-8cb0-4814-95b9-9117ca767681-kube-api-access-kpwq6\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:27:56.159430 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.159423    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0acea23c-8cb0-4814-95b9-9117ca767681-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:27:56.159430 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.159433    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0acea23c-8cb0-4814-95b9-9117ca767681-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:27:56.244781 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.244745    2581 generic.go:356] "Generic (PLEG): container finished" podID="0acea23c-8cb0-4814-95b9-9117ca767681" containerID="2a77c52057fb347c46860e03d7b89a363022d5b06357eb5ec8696ea846f90d33" exitCode=0
Sep 09 20:27:56.244964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.244802    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"
Sep 09 20:27:56.244964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.244838    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h" event={"ID":"0acea23c-8cb0-4814-95b9-9117ca767681","Type":"ContainerDied","Data":"2a77c52057fb347c46860e03d7b89a363022d5b06357eb5ec8696ea846f90d33"}
Sep 09 20:27:56.244964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.244900    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h" event={"ID":"0acea23c-8cb0-4814-95b9-9117ca767681","Type":"ContainerDied","Data":"cc5a1491322b665ffe7761e668a66329a83b6595a3f0f1f02fc0b6722ac0e794"}
Sep 09 20:27:56.244964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.244932    2581 scope.go:117] "RemoveContainer" containerID="2a77c52057fb347c46860e03d7b89a363022d5b06357eb5ec8696ea846f90d33"
Sep 09 20:27:56.254662 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.254566    2581 scope.go:117] "RemoveContainer" containerID="2a77c52057fb347c46860e03d7b89a363022d5b06357eb5ec8696ea846f90d33"
Sep 09 20:27:56.254904 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:27:56.254886    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a77c52057fb347c46860e03d7b89a363022d5b06357eb5ec8696ea846f90d33\": container with ID starting with 2a77c52057fb347c46860e03d7b89a363022d5b06357eb5ec8696ea846f90d33 not found: ID does not exist" containerID="2a77c52057fb347c46860e03d7b89a363022d5b06357eb5ec8696ea846f90d33"
Sep 09 20:27:56.254945 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.254913    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a77c52057fb347c46860e03d7b89a363022d5b06357eb5ec8696ea846f90d33"} err="failed to get container status \"2a77c52057fb347c46860e03d7b89a363022d5b06357eb5ec8696ea846f90d33\": rpc error: code = NotFound desc = could not find container \"2a77c52057fb347c46860e03d7b89a363022d5b06357eb5ec8696ea846f90d33\": container with ID starting with 2a77c52057fb347c46860e03d7b89a363022d5b06357eb5ec8696ea846f90d33 not found: ID does not exist"
Sep 09 20:27:56.269221 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.269152    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"]
Sep 09 20:27:56.272888 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.272863    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-g4j2h"]
Sep 09 20:27:56.635124 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:27:56.635046    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0acea23c-8cb0-4814-95b9-9117ca767681" path="/var/lib/kubelet/pods/0acea23c-8cb0-4814-95b9-9117ca767681/volumes"
Sep 09 20:28:35.652147 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.652101    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f"]
Sep 09 20:28:35.652746 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.652715    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0acea23c-8cb0-4814-95b9-9117ca767681" containerName="main"
Sep 09 20:28:35.652746 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.652737    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0acea23c-8cb0-4814-95b9-9117ca767681" containerName="main"
Sep 09 20:28:35.652855 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.652814    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0acea23c-8cb0-4814-95b9-9117ca767681" containerName="main"
Sep 09 20:28:35.657076 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.657052    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f"]
Sep 09 20:28:35.657200 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.657173    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f"
Sep 09 20:28:35.661090 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.661048    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 09 20:28:35.661090 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.661076    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 09 20:28:35.661276 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.661209    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-fdzks\""
Sep 09 20:28:35.661370 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.661342    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 09 20:28:35.661695 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.661678    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-ktjmc\""
Sep 09 20:28:35.668948 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.668926    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 09 20:28:35.672321 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.672304    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.674854 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.674832    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 09 20:28:35.674854 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.674849    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-qwdzz\""
Sep 09 20:28:35.675021 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.674840    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:28:35.684240 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.684217    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 09 20:28:35.767607 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.767572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.767771 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.767748    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.767833 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.767783    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.767833 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.767818    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.767919 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.767870    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.767919 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.767893    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3942e982-4fc6-4fbb-b0f0-08c405df520f-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f\" (UID: \"3942e982-4fc6-4fbb-b0f0-08c405df520f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f"
Sep 09 20:28:35.768005 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.767945    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbrmf\" (UniqueName: \"kubernetes.io/projected/3942e982-4fc6-4fbb-b0f0-08c405df520f-kube-api-access-dbrmf\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f\" (UID: \"3942e982-4fc6-4fbb-b0f0-08c405df520f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f"
Sep 09 20:28:35.768005 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.767984    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.768128 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.768107    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7mgs\" (UniqueName: \"kubernetes.io/projected/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-kube-api-access-b7mgs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.768190 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.768175    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.869166 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.869125    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.869366 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.869172    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.869366 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.869206    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.869366 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.869240    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.869366 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.869269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3942e982-4fc6-4fbb-b0f0-08c405df520f-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f\" (UID: \"3942e982-4fc6-4fbb-b0f0-08c405df520f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f"
Sep 09 20:28:35.869366 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.869320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbrmf\" (UniqueName: \"kubernetes.io/projected/3942e982-4fc6-4fbb-b0f0-08c405df520f-kube-api-access-dbrmf\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f\" (UID: \"3942e982-4fc6-4fbb-b0f0-08c405df520f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f"
Sep 09 20:28:35.869366 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.869366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.869715 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.869400    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7mgs\" (UniqueName: \"kubernetes.io/projected/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-kube-api-access-b7mgs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.869715 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.869427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.869715 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.869509    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.869715 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.869561    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.869715 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.869587    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.869996 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.869758    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.869996 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.869939    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.870112 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.870000    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.871921 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.871899    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.872069 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.872006    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.872185 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.872170    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3942e982-4fc6-4fbb-b0f0-08c405df520f-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f\" (UID: \"3942e982-4fc6-4fbb-b0f0-08c405df520f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f"
Sep 09 20:28:35.877478 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.877441    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbrmf\" (UniqueName: \"kubernetes.io/projected/3942e982-4fc6-4fbb-b0f0-08c405df520f-kube-api-access-dbrmf\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f\" (UID: \"3942e982-4fc6-4fbb-b0f0-08c405df520f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f"
Sep 09 20:28:35.877998 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.877979    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7mgs\" (UniqueName: \"kubernetes.io/projected/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-kube-api-access-b7mgs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:35.968519 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.968419    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f"
Sep 09 20:28:35.981702 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:35.981678    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:28:36.250579 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:36.250552    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f"]
Sep 09 20:28:36.252006 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:28:36.251977    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3942e982_4fc6_4fbb_b0f0_08c405df520f.slice/crio-51e6cfef865cbb7d649a8a684859fd86b2893f5404ad33fec646ecd6353d65bd WatchSource:0}: Error finding container 51e6cfef865cbb7d649a8a684859fd86b2893f5404ad33fec646ecd6353d65bd: Status 404 returned error can't find the container with id 51e6cfef865cbb7d649a8a684859fd86b2893f5404ad33fec646ecd6353d65bd
Sep 09 20:28:36.254341 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:36.254322    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:28:36.268815 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:36.268788    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 09 20:28:36.270244 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:28:36.270214    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d777fad_4e7b_495a_92db_d4e5c9b64e4d.slice/crio-6d6a5714237a42221dd62c3db0bee9add54d2e0c2d878e4b6a6b83ffd500b542 WatchSource:0}: Error finding container 6d6a5714237a42221dd62c3db0bee9add54d2e0c2d878e4b6a6b83ffd500b542: Status 404 returned error can't find the container with id 6d6a5714237a42221dd62c3db0bee9add54d2e0c2d878e4b6a6b83ffd500b542
Sep 09 20:28:36.447561 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:36.447527    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"5d777fad-4e7b-495a-92db-d4e5c9b64e4d","Type":"ContainerStarted","Data":"6d6a5714237a42221dd62c3db0bee9add54d2e0c2d878e4b6a6b83ffd500b542"}
Sep 09 20:28:36.448686 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:36.448663    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f" event={"ID":"3942e982-4fc6-4fbb-b0f0-08c405df520f","Type":"ContainerStarted","Data":"51e6cfef865cbb7d649a8a684859fd86b2893f5404ad33fec646ecd6353d65bd"}
Sep 09 20:28:39.469178 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:39.469137    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f" event={"ID":"3942e982-4fc6-4fbb-b0f0-08c405df520f","Type":"ContainerStarted","Data":"5a1b615ca598b5811e57f259c76faec34c6eb57a4a09e2c6a520d707b28c3a4c"}
Sep 09 20:28:39.469621 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:39.469242    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f"
Sep 09 20:28:39.488420 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:39.488368    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f" podStartSLOduration=1.93256309 podStartE2EDuration="4.488352435s" totalImagesPullingTime="2.555789345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:28:39.486598129 +0000 UTC m=+1061.491775125" watchObservedRunningTime="2026-09-09 20:28:39.488352435 +0000 UTC m=+1061.493529429"
Sep 09 20:28:40.476166 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:40.476123    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"5d777fad-4e7b-495a-92db-d4e5c9b64e4d","Type":"ContainerStarted","Data":"3550e660dac14ea3cd9c1582282deacc758cdbfb74e055ec621bd6a0af639924"}
Sep 09 20:28:43.491407 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:43.491375    2581 generic.go:356] "Generic (PLEG): container finished" podID="5d777fad-4e7b-495a-92db-d4e5c9b64e4d" containerID="3550e660dac14ea3cd9c1582282deacc758cdbfb74e055ec621bd6a0af639924" exitCode=0
Sep 09 20:28:43.491792 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:28:43.491446    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"5d777fad-4e7b-495a-92db-d4e5c9b64e4d","Type":"ContainerDied","Data":"3550e660dac14ea3cd9c1582282deacc758cdbfb74e055ec621bd6a0af639924"}
Sep 09 20:29:10.477886 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:29:10.477856    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f"
Sep 09 20:29:12.673018 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:29:12.672979    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"5d777fad-4e7b-495a-92db-d4e5c9b64e4d","Type":"ContainerStarted","Data":"815b70c60d3153fa8f781995e962687b54ada9d39b983df675653166b056e515"}
Sep 09 20:29:12.696068 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:29:12.696012    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.399853016 podStartE2EDuration="37.695982909s" totalImagesPullingTime="33.087157333s" totalInitContainerRuntime="2.20897256s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:28:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:29:12.69526893 +0000 UTC m=+1094.700445926" watchObservedRunningTime="2026-09-09 20:29:12.695982909 +0000 UTC m=+1094.701159907"
Sep 09 20:30:21.806704 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:21.806627    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f"]
Sep 09 20:30:30.478690 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:30.478649    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f" podUID="3942e982-4fc6-4fbb-b0f0-08c405df520f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:30:36.827859 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:36.827811    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f" podUID="3942e982-4fc6-4fbb-b0f0-08c405df520f" containerName="main" containerID="cri-o://5a1b615ca598b5811e57f259c76faec34c6eb57a4a09e2c6a520d707b28c3a4c" gracePeriod=45
Sep 09 20:30:37.931293 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:37.931251    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 09 20:30:40.477784 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:40.477741    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f" podUID="3942e982-4fc6-4fbb-b0f0-08c405df520f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:30:50.478252 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:50.478214    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f" podUID="3942e982-4fc6-4fbb-b0f0-08c405df520f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:30:50.478701 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:50.478332    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f"
Sep 09 20:30:52.956800 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:52.956749    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="5d777fad-4e7b-495a-92db-d4e5c9b64e4d" containerName="main" containerID="cri-o://815b70c60d3153fa8f781995e962687b54ada9d39b983df675653166b056e515" gracePeriod=45
Sep 09 20:30:53.706525 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.706501    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:30:53.793366 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.793338    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-tmp-dir\") pod \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") "
Sep 09 20:30:53.793559 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.793376    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-kserve-provision-location\") pod \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") "
Sep 09 20:30:53.793559 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.793395    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7mgs\" (UniqueName: \"kubernetes.io/projected/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-kube-api-access-b7mgs\") pod \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") "
Sep 09 20:30:53.793559 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.793469    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-tls-certs\") pod \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") "
Sep 09 20:30:53.793559 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.793484    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-model-cache\") pod \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") "
Sep 09 20:30:53.793559 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.793513    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-cabundle-cert\") pod \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") "
Sep 09 20:30:53.793559 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.793554    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-dshm\") pod \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") "
Sep 09 20:30:53.793887 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.793581    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-home\") pod \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\" (UID: \"5d777fad-4e7b-495a-92db-d4e5c9b64e4d\") "
Sep 09 20:30:53.793887 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.793779    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d777fad-4e7b-495a-92db-d4e5c9b64e4d" (UID: "5d777fad-4e7b-495a-92db-d4e5c9b64e4d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:53.794071 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.794044    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5d777fad-4e7b-495a-92db-d4e5c9b64e4d" (UID: "5d777fad-4e7b-495a-92db-d4e5c9b64e4d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:30:53.794391 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.794370    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-home" (OuterVolumeSpecName: "home") pod "5d777fad-4e7b-495a-92db-d4e5c9b64e4d" (UID: "5d777fad-4e7b-495a-92db-d4e5c9b64e4d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:53.795894 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.795869    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-dshm" (OuterVolumeSpecName: "dshm") pod "5d777fad-4e7b-495a-92db-d4e5c9b64e4d" (UID: "5d777fad-4e7b-495a-92db-d4e5c9b64e4d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:53.796033 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.796008    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d777fad-4e7b-495a-92db-d4e5c9b64e4d" (UID: "5d777fad-4e7b-495a-92db-d4e5c9b64e4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:30:53.796097 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.796074    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-kube-api-access-b7mgs" (OuterVolumeSpecName: "kube-api-access-b7mgs") pod "5d777fad-4e7b-495a-92db-d4e5c9b64e4d" (UID: "5d777fad-4e7b-495a-92db-d4e5c9b64e4d"). InnerVolumeSpecName "kube-api-access-b7mgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:30:53.805698 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.805674    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d777fad-4e7b-495a-92db-d4e5c9b64e4d" (UID: "5d777fad-4e7b-495a-92db-d4e5c9b64e4d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:53.813931 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.813909    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5d777fad-4e7b-495a-92db-d4e5c9b64e4d" (UID: "5d777fad-4e7b-495a-92db-d4e5c9b64e4d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:53.894585 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.894559    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:30:53.894585 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.894582    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:30:53.894585 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.894590    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:30:53.894756 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.894598    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-kserve-provision-location\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:30:53.894756 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.894607    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7mgs\" (UniqueName: \"kubernetes.io/projected/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-kube-api-access-b7mgs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:30:53.894756 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.894616    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:30:53.894756 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.894625    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:30:53.894756 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:53.894633    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d777fad-4e7b-495a-92db-d4e5c9b64e4d-cabundle-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:30:54.224247 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:54.224159    2581 generic.go:356] "Generic (PLEG): container finished" podID="5d777fad-4e7b-495a-92db-d4e5c9b64e4d" containerID="815b70c60d3153fa8f781995e962687b54ada9d39b983df675653166b056e515" exitCode=0
Sep 09 20:30:54.224247 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:54.224213    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:30:54.224247 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:54.224235    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"5d777fad-4e7b-495a-92db-d4e5c9b64e4d","Type":"ContainerDied","Data":"815b70c60d3153fa8f781995e962687b54ada9d39b983df675653166b056e515"}
Sep 09 20:30:54.224803 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:54.224285    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"5d777fad-4e7b-495a-92db-d4e5c9b64e4d","Type":"ContainerDied","Data":"6d6a5714237a42221dd62c3db0bee9add54d2e0c2d878e4b6a6b83ffd500b542"}
Sep 09 20:30:54.224803 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:54.224308    2581 scope.go:117] "RemoveContainer" containerID="815b70c60d3153fa8f781995e962687b54ada9d39b983df675653166b056e515"
Sep 09 20:30:54.233209 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:54.233191    2581 scope.go:117] "RemoveContainer" containerID="3550e660dac14ea3cd9c1582282deacc758cdbfb74e055ec621bd6a0af639924"
Sep 09 20:30:54.240749 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:54.240723    2581 scope.go:117] "RemoveContainer" containerID="815b70c60d3153fa8f781995e962687b54ada9d39b983df675653166b056e515"
Sep 09 20:30:54.241015 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:30:54.240996    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"815b70c60d3153fa8f781995e962687b54ada9d39b983df675653166b056e515\": container with ID starting with 815b70c60d3153fa8f781995e962687b54ada9d39b983df675653166b056e515 not found: ID does not exist" containerID="815b70c60d3153fa8f781995e962687b54ada9d39b983df675653166b056e515"
Sep 09 20:30:54.241078 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:54.241023    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815b70c60d3153fa8f781995e962687b54ada9d39b983df675653166b056e515"} err="failed to get container status \"815b70c60d3153fa8f781995e962687b54ada9d39b983df675653166b056e515\": rpc error: code = NotFound desc = could not find container \"815b70c60d3153fa8f781995e962687b54ada9d39b983df675653166b056e515\": container with ID starting with 815b70c60d3153fa8f781995e962687b54ada9d39b983df675653166b056e515 not found: ID does not exist"
Sep 09 20:30:54.241078 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:54.241041    2581 scope.go:117] "RemoveContainer" containerID="3550e660dac14ea3cd9c1582282deacc758cdbfb74e055ec621bd6a0af639924"
Sep 09 20:30:54.241302 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:30:54.241286    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3550e660dac14ea3cd9c1582282deacc758cdbfb74e055ec621bd6a0af639924\": container with ID starting with 3550e660dac14ea3cd9c1582282deacc758cdbfb74e055ec621bd6a0af639924 not found: ID does not exist" containerID="3550e660dac14ea3cd9c1582282deacc758cdbfb74e055ec621bd6a0af639924"
Sep 09 20:30:54.241347 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:54.241308    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3550e660dac14ea3cd9c1582282deacc758cdbfb74e055ec621bd6a0af639924"} err="failed to get container status \"3550e660dac14ea3cd9c1582282deacc758cdbfb74e055ec621bd6a0af639924\": rpc error: code = NotFound desc = could not find container \"3550e660dac14ea3cd9c1582282deacc758cdbfb74e055ec621bd6a0af639924\": container with ID starting with 3550e660dac14ea3cd9c1582282deacc758cdbfb74e055ec621bd6a0af639924 not found: ID does not exist"
Sep 09 20:30:54.247450 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:54.247425    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 09 20:30:54.252104 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:54.252084    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 09 20:30:54.635777 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:54.635739    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d777fad-4e7b-495a-92db-d4e5c9b64e4d" path="/var/lib/kubelet/pods/5d777fad-4e7b-495a-92db-d4e5c9b64e4d/volumes"
Sep 09 20:30:58.488679 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:58.488650    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 20:30:58.489233 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:58.489212    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 20:30:58.491563 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:58.491536    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 20:30:58.492018 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:30:58.492002    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 20:31:00.477904 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:00.477857    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f" podUID="3942e982-4fc6-4fbb-b0f0-08c405df520f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:31:07.077407 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:07.077383    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f"
Sep 09 20:31:07.208377 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:07.208297    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbrmf\" (UniqueName: \"kubernetes.io/projected/3942e982-4fc6-4fbb-b0f0-08c405df520f-kube-api-access-dbrmf\") pod \"3942e982-4fc6-4fbb-b0f0-08c405df520f\" (UID: \"3942e982-4fc6-4fbb-b0f0-08c405df520f\") "
Sep 09 20:31:07.208377 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:07.208334    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3942e982-4fc6-4fbb-b0f0-08c405df520f-tls-certs\") pod \"3942e982-4fc6-4fbb-b0f0-08c405df520f\" (UID: \"3942e982-4fc6-4fbb-b0f0-08c405df520f\") "
Sep 09 20:31:07.210351 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:07.210322    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3942e982-4fc6-4fbb-b0f0-08c405df520f-kube-api-access-dbrmf" (OuterVolumeSpecName: "kube-api-access-dbrmf") pod "3942e982-4fc6-4fbb-b0f0-08c405df520f" (UID: "3942e982-4fc6-4fbb-b0f0-08c405df520f"). InnerVolumeSpecName "kube-api-access-dbrmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:31:07.210453 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:07.210329    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3942e982-4fc6-4fbb-b0f0-08c405df520f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3942e982-4fc6-4fbb-b0f0-08c405df520f" (UID: "3942e982-4fc6-4fbb-b0f0-08c405df520f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:31:07.291790 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:07.291756    2581 generic.go:356] "Generic (PLEG): container finished" podID="3942e982-4fc6-4fbb-b0f0-08c405df520f" containerID="5a1b615ca598b5811e57f259c76faec34c6eb57a4a09e2c6a520d707b28c3a4c" exitCode=0
Sep 09 20:31:07.291952 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:07.291801    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f"
Sep 09 20:31:07.291952 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:07.291843    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f" event={"ID":"3942e982-4fc6-4fbb-b0f0-08c405df520f","Type":"ContainerDied","Data":"5a1b615ca598b5811e57f259c76faec34c6eb57a4a09e2c6a520d707b28c3a4c"}
Sep 09 20:31:07.291952 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:07.291898    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f" event={"ID":"3942e982-4fc6-4fbb-b0f0-08c405df520f","Type":"ContainerDied","Data":"51e6cfef865cbb7d649a8a684859fd86b2893f5404ad33fec646ecd6353d65bd"}
Sep 09 20:31:07.291952 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:07.291921    2581 scope.go:117] "RemoveContainer" containerID="5a1b615ca598b5811e57f259c76faec34c6eb57a4a09e2c6a520d707b28c3a4c"
Sep 09 20:31:07.300971 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:07.300952    2581 scope.go:117] "RemoveContainer" containerID="5a1b615ca598b5811e57f259c76faec34c6eb57a4a09e2c6a520d707b28c3a4c"
Sep 09 20:31:07.301244 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:31:07.301227    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a1b615ca598b5811e57f259c76faec34c6eb57a4a09e2c6a520d707b28c3a4c\": container with ID starting with 5a1b615ca598b5811e57f259c76faec34c6eb57a4a09e2c6a520d707b28c3a4c not found: ID does not exist" containerID="5a1b615ca598b5811e57f259c76faec34c6eb57a4a09e2c6a520d707b28c3a4c"
Sep 09 20:31:07.301297 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:07.301251    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a1b615ca598b5811e57f259c76faec34c6eb57a4a09e2c6a520d707b28c3a4c"} err="failed to get container status \"5a1b615ca598b5811e57f259c76faec34c6eb57a4a09e2c6a520d707b28c3a4c\": rpc error: code = NotFound desc = could not find container \"5a1b615ca598b5811e57f259c76faec34c6eb57a4a09e2c6a520d707b28c3a4c\": container with ID starting with 5a1b615ca598b5811e57f259c76faec34c6eb57a4a09e2c6a520d707b28c3a4c not found: ID does not exist"
Sep 09 20:31:07.309057 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:07.309034    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbrmf\" (UniqueName: \"kubernetes.io/projected/3942e982-4fc6-4fbb-b0f0-08c405df520f-kube-api-access-dbrmf\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:31:07.309143 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:07.309060    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3942e982-4fc6-4fbb-b0f0-08c405df520f-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:31:07.314226 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:07.314201    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f"]
Sep 09 20:31:07.320369 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:07.320345    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7qh9f"]
Sep 09 20:31:08.635558 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:08.635527    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3942e982-4fc6-4fbb-b0f0-08c405df520f" path="/var/lib/kubelet/pods/3942e982-4fc6-4fbb-b0f0-08c405df520f/volumes"
Sep 09 20:31:25.053045 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.053007    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"]
Sep 09 20:31:25.053531 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.053339    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d777fad-4e7b-495a-92db-d4e5c9b64e4d" containerName="storage-initializer"
Sep 09 20:31:25.053531 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.053350    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d777fad-4e7b-495a-92db-d4e5c9b64e4d" containerName="storage-initializer"
Sep 09 20:31:25.053531 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.053367    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d777fad-4e7b-495a-92db-d4e5c9b64e4d" containerName="main"
Sep 09 20:31:25.053531 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.053372    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d777fad-4e7b-495a-92db-d4e5c9b64e4d" containerName="main"
Sep 09 20:31:25.053531 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.053379    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3942e982-4fc6-4fbb-b0f0-08c405df520f" containerName="main"
Sep 09 20:31:25.053531 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.053385    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3942e982-4fc6-4fbb-b0f0-08c405df520f" containerName="main"
Sep 09 20:31:25.053531 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.053429    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3942e982-4fc6-4fbb-b0f0-08c405df520f" containerName="main"
Sep 09 20:31:25.053531 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.053439    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d777fad-4e7b-495a-92db-d4e5c9b64e4d" containerName="main"
Sep 09 20:31:25.057686 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.057666    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:25.060377 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.060342    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 09 20:31:25.060526 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.060473    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 09 20:31:25.060603 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.060588    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-6zklr\""
Sep 09 20:31:25.060683 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.060669    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 09 20:31:25.060781 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.060757    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b8b37df-7265-4162-bcb7-7819b63429b7-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-s9vn8\" (UID: \"3b8b37df-7265-4162-bcb7-7819b63429b7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:25.060846 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.060826    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b8b37df-7265-4162-bcb7-7819b63429b7-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-s9vn8\" (UID: \"3b8b37df-7265-4162-bcb7-7819b63429b7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:25.060910 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.060896    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktgp8\" (UniqueName: \"kubernetes.io/projected/3b8b37df-7265-4162-bcb7-7819b63429b7-kube-api-access-ktgp8\") pod \"scheduler-inline-config-test-kserve-8655bc757-s9vn8\" (UID: \"3b8b37df-7265-4162-bcb7-7819b63429b7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:25.060947 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.060931    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b8b37df-7265-4162-bcb7-7819b63429b7-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-s9vn8\" (UID: \"3b8b37df-7265-4162-bcb7-7819b63429b7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:25.060987 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.060974    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8b37df-7265-4162-bcb7-7819b63429b7-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-s9vn8\" (UID: \"3b8b37df-7265-4162-bcb7-7819b63429b7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:25.061065 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.061045    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8b37df-7265-4162-bcb7-7819b63429b7-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-s9vn8\" (UID: \"3b8b37df-7265-4162-bcb7-7819b63429b7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:25.065974 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.065954    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"]
Sep 09 20:31:25.162417 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.162387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b8b37df-7265-4162-bcb7-7819b63429b7-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-s9vn8\" (UID: \"3b8b37df-7265-4162-bcb7-7819b63429b7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:25.162417 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.162425    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b8b37df-7265-4162-bcb7-7819b63429b7-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-s9vn8\" (UID: \"3b8b37df-7265-4162-bcb7-7819b63429b7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:25.162691 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.162516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktgp8\" (UniqueName: \"kubernetes.io/projected/3b8b37df-7265-4162-bcb7-7819b63429b7-kube-api-access-ktgp8\") pod \"scheduler-inline-config-test-kserve-8655bc757-s9vn8\" (UID: \"3b8b37df-7265-4162-bcb7-7819b63429b7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:25.162691 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.162552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b8b37df-7265-4162-bcb7-7819b63429b7-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-s9vn8\" (UID: \"3b8b37df-7265-4162-bcb7-7819b63429b7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:25.162691 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.162591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8b37df-7265-4162-bcb7-7819b63429b7-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-s9vn8\" (UID: \"3b8b37df-7265-4162-bcb7-7819b63429b7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:25.162691 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.162624    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8b37df-7265-4162-bcb7-7819b63429b7-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-s9vn8\" (UID: \"3b8b37df-7265-4162-bcb7-7819b63429b7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:25.162911 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.162805    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b8b37df-7265-4162-bcb7-7819b63429b7-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-s9vn8\" (UID: \"3b8b37df-7265-4162-bcb7-7819b63429b7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:25.162972 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.162931    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b8b37df-7265-4162-bcb7-7819b63429b7-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-s9vn8\" (UID: \"3b8b37df-7265-4162-bcb7-7819b63429b7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:25.163041 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.162982    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8b37df-7265-4162-bcb7-7819b63429b7-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-s9vn8\" (UID: \"3b8b37df-7265-4162-bcb7-7819b63429b7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:25.164766 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.164745    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b8b37df-7265-4162-bcb7-7819b63429b7-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-s9vn8\" (UID: \"3b8b37df-7265-4162-bcb7-7819b63429b7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:25.164954 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.164939    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8b37df-7265-4162-bcb7-7819b63429b7-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-s9vn8\" (UID: \"3b8b37df-7265-4162-bcb7-7819b63429b7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:25.173336 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.173314    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktgp8\" (UniqueName: \"kubernetes.io/projected/3b8b37df-7265-4162-bcb7-7819b63429b7-kube-api-access-ktgp8\") pod \"scheduler-inline-config-test-kserve-8655bc757-s9vn8\" (UID: \"3b8b37df-7265-4162-bcb7-7819b63429b7\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:25.311288 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.311197    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq"]
Sep 09 20:31:25.314980 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.314958    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq"
Sep 09 20:31:25.317633 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.317610    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-mgft9\""
Sep 09 20:31:25.322874 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.322853    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq"]
Sep 09 20:31:25.364507 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.364449    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a828beed-f3fb-4da0-9345-edc2599f9673-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbffljq\" (UID: \"a828beed-f3fb-4da0-9345-edc2599f9673\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq"
Sep 09 20:31:25.364678 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.364544    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp2n8\" (UniqueName: \"kubernetes.io/projected/a828beed-f3fb-4da0-9345-edc2599f9673-kube-api-access-kp2n8\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbffljq\" (UID: \"a828beed-f3fb-4da0-9345-edc2599f9673\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq"
Sep 09 20:31:25.368394 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.368374    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:25.465090 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.465061    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp2n8\" (UniqueName: \"kubernetes.io/projected/a828beed-f3fb-4da0-9345-edc2599f9673-kube-api-access-kp2n8\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbffljq\" (UID: \"a828beed-f3fb-4da0-9345-edc2599f9673\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq"
Sep 09 20:31:25.465242 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.465120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a828beed-f3fb-4da0-9345-edc2599f9673-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbffljq\" (UID: \"a828beed-f3fb-4da0-9345-edc2599f9673\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq"
Sep 09 20:31:25.467555 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.467534    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a828beed-f3fb-4da0-9345-edc2599f9673-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbffljq\" (UID: \"a828beed-f3fb-4da0-9345-edc2599f9673\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq"
Sep 09 20:31:25.474434 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.474409    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp2n8\" (UniqueName: \"kubernetes.io/projected/a828beed-f3fb-4da0-9345-edc2599f9673-kube-api-access-kp2n8\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbffljq\" (UID: \"a828beed-f3fb-4da0-9345-edc2599f9673\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq"
Sep 09 20:31:25.628134 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.628063    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq"
Sep 09 20:31:25.638716 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.638691    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"]
Sep 09 20:31:25.640441 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:31:25.640311    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8b37df_7265_4162_bcb7_7819b63429b7.slice/crio-368828eb7df815c356f30923a164e3a1be7bb9e4d8cf5204136fe5eb31c18700 WatchSource:0}: Error finding container 368828eb7df815c356f30923a164e3a1be7bb9e4d8cf5204136fe5eb31c18700: Status 404 returned error can't find the container with id 368828eb7df815c356f30923a164e3a1be7bb9e4d8cf5204136fe5eb31c18700
Sep 09 20:31:25.902442 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:25.902236    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq"]
Sep 09 20:31:25.905142 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:31:25.905116    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda828beed_f3fb_4da0_9345_edc2599f9673.slice/crio-ffcf50ba4b17c0d3f7fbb03ff8771274b1794e6702c6ed0cdd82bc49fbd4a463 WatchSource:0}: Error finding container ffcf50ba4b17c0d3f7fbb03ff8771274b1794e6702c6ed0cdd82bc49fbd4a463: Status 404 returned error can't find the container with id ffcf50ba4b17c0d3f7fbb03ff8771274b1794e6702c6ed0cdd82bc49fbd4a463
Sep 09 20:31:26.390131 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:26.390094    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq" event={"ID":"a828beed-f3fb-4da0-9345-edc2599f9673","Type":"ContainerStarted","Data":"aad94ce223d227896c06a19adb22eebba04379ce67746b9bb155cf8437c54dc0"}
Sep 09 20:31:26.390608 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:26.390169    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq"
Sep 09 20:31:26.390608 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:26.390196    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq" event={"ID":"a828beed-f3fb-4da0-9345-edc2599f9673","Type":"ContainerStarted","Data":"ffcf50ba4b17c0d3f7fbb03ff8771274b1794e6702c6ed0cdd82bc49fbd4a463"}
Sep 09 20:31:26.391603 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:26.391581    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8" event={"ID":"3b8b37df-7265-4162-bcb7-7819b63429b7","Type":"ContainerStarted","Data":"b34b216c4e39c5bcf9c5f3d7c199b8fbf7350576c529e82247af870af3e8bef4"}
Sep 09 20:31:26.391704 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:26.391620    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8" event={"ID":"3b8b37df-7265-4162-bcb7-7819b63429b7","Type":"ContainerStarted","Data":"368828eb7df815c356f30923a164e3a1be7bb9e4d8cf5204136fe5eb31c18700"}
Sep 09 20:31:26.408737 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:26.408693    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq" podStartSLOduration=1.408680118 podStartE2EDuration="1.408680118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:31:26.407605224 +0000 UTC m=+1228.412782220" watchObservedRunningTime="2026-09-09 20:31:26.408680118 +0000 UTC m=+1228.413857113"
Sep 09 20:31:26.427243 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:26.427199    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8" podStartSLOduration=1.427185198 podStartE2EDuration="1.427185198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:31:26.426712976 +0000 UTC m=+1228.431889972" watchObservedRunningTime="2026-09-09 20:31:26.427185198 +0000 UTC m=+1228.432362192"
Sep 09 20:31:27.369001 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:27.368963    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:35.369337 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:35.369295    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:35.381863 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:35.381841    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:35.448227 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:35.448197    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:57.398622 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:57.398590    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq"
Sep 09 20:31:59.081198 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.081161    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"]
Sep 09 20:31:59.084692 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.084664    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq"]
Sep 09 20:31:59.091078 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:31:59.091042    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:31:59.091078 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:31:59.091078 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:31:59.091078 ip-10-0-134-84 kubenswrapper[2581]:  > containerID="b34b216c4e39c5bcf9c5f3d7c199b8fbf7350576c529e82247af870af3e8bef4" cmd=["/bin/sleep","15"]
Sep 09 20:31:59.091405 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.091091    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8" message="" err=<
Sep 09 20:31:59.091405 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:31:59.091405 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:31:59.091405 ip-10-0-134-84 kubenswrapper[2581]:  >
Sep 09 20:31:59.091405 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:31:59.091106    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:31:59.091405 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:31:59.091405 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:31:59.091405 ip-10-0-134-84 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8" podUID="3b8b37df-7265-4162-bcb7-7819b63429b7" containerName="main" containerID="cri-o://b34b216c4e39c5bcf9c5f3d7c199b8fbf7350576c529e82247af870af3e8bef4"
Sep 09 20:31:59.091405 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.091146    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8" podUID="3b8b37df-7265-4162-bcb7-7819b63429b7" containerName="main" containerID="cri-o://b34b216c4e39c5bcf9c5f3d7c199b8fbf7350576c529e82247af870af3e8bef4" gracePeriod=60
Sep 09 20:31:59.442342 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.442313    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:59.547692 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.547663    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b8b37df-7265-4162-bcb7-7819b63429b7-dshm\") pod \"3b8b37df-7265-4162-bcb7-7819b63429b7\" (UID: \"3b8b37df-7265-4162-bcb7-7819b63429b7\") "
Sep 09 20:31:59.547867 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.547712    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktgp8\" (UniqueName: \"kubernetes.io/projected/3b8b37df-7265-4162-bcb7-7819b63429b7-kube-api-access-ktgp8\") pod \"3b8b37df-7265-4162-bcb7-7819b63429b7\" (UID: \"3b8b37df-7265-4162-bcb7-7819b63429b7\") "
Sep 09 20:31:59.547867 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.547749    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b8b37df-7265-4162-bcb7-7819b63429b7-model-cache\") pod \"3b8b37df-7265-4162-bcb7-7819b63429b7\" (UID: \"3b8b37df-7265-4162-bcb7-7819b63429b7\") "
Sep 09 20:31:59.547867 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.547771    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8b37df-7265-4162-bcb7-7819b63429b7-tls-certs\") pod \"3b8b37df-7265-4162-bcb7-7819b63429b7\" (UID: \"3b8b37df-7265-4162-bcb7-7819b63429b7\") "
Sep 09 20:31:59.547867 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.547786    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b8b37df-7265-4162-bcb7-7819b63429b7-home\") pod \"3b8b37df-7265-4162-bcb7-7819b63429b7\" (UID: \"3b8b37df-7265-4162-bcb7-7819b63429b7\") "
Sep 09 20:31:59.547867 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.547808    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8b37df-7265-4162-bcb7-7819b63429b7-tmp-dir\") pod \"3b8b37df-7265-4162-bcb7-7819b63429b7\" (UID: \"3b8b37df-7265-4162-bcb7-7819b63429b7\") "
Sep 09 20:31:59.548127 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.548011    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8b37df-7265-4162-bcb7-7819b63429b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "3b8b37df-7265-4162-bcb7-7819b63429b7" (UID: "3b8b37df-7265-4162-bcb7-7819b63429b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:31:59.548222 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.548184    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8b37df-7265-4162-bcb7-7819b63429b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3b8b37df-7265-4162-bcb7-7819b63429b7" (UID: "3b8b37df-7265-4162-bcb7-7819b63429b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:31:59.548308 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.548285    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8b37df-7265-4162-bcb7-7819b63429b7-home" (OuterVolumeSpecName: "home") pod "3b8b37df-7265-4162-bcb7-7819b63429b7" (UID: "3b8b37df-7265-4162-bcb7-7819b63429b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:31:59.549940 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.549920    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8b37df-7265-4162-bcb7-7819b63429b7-dshm" (OuterVolumeSpecName: "dshm") pod "3b8b37df-7265-4162-bcb7-7819b63429b7" (UID: "3b8b37df-7265-4162-bcb7-7819b63429b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:31:59.550039 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.549981    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b8b37df-7265-4162-bcb7-7819b63429b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b8b37df-7265-4162-bcb7-7819b63429b7" (UID: "3b8b37df-7265-4162-bcb7-7819b63429b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:31:59.550039 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.550005    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b8b37df-7265-4162-bcb7-7819b63429b7-kube-api-access-ktgp8" (OuterVolumeSpecName: "kube-api-access-ktgp8") pod "3b8b37df-7265-4162-bcb7-7819b63429b7" (UID: "3b8b37df-7265-4162-bcb7-7819b63429b7"). InnerVolumeSpecName "kube-api-access-ktgp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:31:59.563553 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.563524    2581 generic.go:356] "Generic (PLEG): container finished" podID="3b8b37df-7265-4162-bcb7-7819b63429b7" containerID="b34b216c4e39c5bcf9c5f3d7c199b8fbf7350576c529e82247af870af3e8bef4" exitCode=0
Sep 09 20:31:59.563640 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.563579    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"
Sep 09 20:31:59.563640 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.563605    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8" event={"ID":"3b8b37df-7265-4162-bcb7-7819b63429b7","Type":"ContainerDied","Data":"b34b216c4e39c5bcf9c5f3d7c199b8fbf7350576c529e82247af870af3e8bef4"}
Sep 09 20:31:59.563729 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.563658    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8" event={"ID":"3b8b37df-7265-4162-bcb7-7819b63429b7","Type":"ContainerDied","Data":"368828eb7df815c356f30923a164e3a1be7bb9e4d8cf5204136fe5eb31c18700"}
Sep 09 20:31:59.563729 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.563680    2581 scope.go:117] "RemoveContainer" containerID="b34b216c4e39c5bcf9c5f3d7c199b8fbf7350576c529e82247af870af3e8bef4"
Sep 09 20:31:59.572646 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.572631    2581 scope.go:117] "RemoveContainer" containerID="b34b216c4e39c5bcf9c5f3d7c199b8fbf7350576c529e82247af870af3e8bef4"
Sep 09 20:31:59.572913 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:31:59.572894    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34b216c4e39c5bcf9c5f3d7c199b8fbf7350576c529e82247af870af3e8bef4\": container with ID starting with b34b216c4e39c5bcf9c5f3d7c199b8fbf7350576c529e82247af870af3e8bef4 not found: ID does not exist" containerID="b34b216c4e39c5bcf9c5f3d7c199b8fbf7350576c529e82247af870af3e8bef4"
Sep 09 20:31:59.572967 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.572922    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34b216c4e39c5bcf9c5f3d7c199b8fbf7350576c529e82247af870af3e8bef4"} err="failed to get container status \"b34b216c4e39c5bcf9c5f3d7c199b8fbf7350576c529e82247af870af3e8bef4\": rpc error: code = NotFound desc = could not find container \"b34b216c4e39c5bcf9c5f3d7c199b8fbf7350576c529e82247af870af3e8bef4\": container with ID starting with b34b216c4e39c5bcf9c5f3d7c199b8fbf7350576c529e82247af870af3e8bef4 not found: ID does not exist"
Sep 09 20:31:59.588784 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.588195    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"]
Sep 09 20:31:59.593654 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.593631    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8"]
Sep 09 20:31:59.649361 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.649312    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktgp8\" (UniqueName: \"kubernetes.io/projected/3b8b37df-7265-4162-bcb7-7819b63429b7-kube-api-access-ktgp8\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:31:59.649361 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.649355    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b8b37df-7265-4162-bcb7-7819b63429b7-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:31:59.649361 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.649366    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8b37df-7265-4162-bcb7-7819b63429b7-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:31:59.649361 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.649374    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b8b37df-7265-4162-bcb7-7819b63429b7-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:31:59.649673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.649381    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8b37df-7265-4162-bcb7-7819b63429b7-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:31:59.649673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:31:59.649388    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b8b37df-7265-4162-bcb7-7819b63429b7-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:32:00.438297 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:00.438248    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s9vn8" podUID="3b8b37df-7265-4162-bcb7-7819b63429b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 09 20:32:00.634680 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:00.634646    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b8b37df-7265-4162-bcb7-7819b63429b7" path="/var/lib/kubelet/pods/3b8b37df-7265-4162-bcb7-7819b63429b7/volumes"
Sep 09 20:32:07.398837 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:07.398796    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq" podUID="a828beed-f3fb-4da0-9345-edc2599f9673" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:32:14.105952 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:14.105908    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq" podUID="a828beed-f3fb-4da0-9345-edc2599f9673" containerName="main" containerID="cri-o://aad94ce223d227896c06a19adb22eebba04379ce67746b9bb155cf8437c54dc0" gracePeriod=45
Sep 09 20:32:17.399069 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:17.399032    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq" podUID="a828beed-f3fb-4da0-9345-edc2599f9673" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:32:27.398249 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:27.398208    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq" podUID="a828beed-f3fb-4da0-9345-edc2599f9673" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:32:27.398703 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:27.398327    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq"
Sep 09 20:32:37.398947 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:37.398909    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq" podUID="a828beed-f3fb-4da0-9345-edc2599f9673" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:32:44.358295 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:44.358271    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq"
Sep 09 20:32:44.401316 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:44.401234    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a828beed-f3fb-4da0-9345-edc2599f9673-tls-certs\") pod \"a828beed-f3fb-4da0-9345-edc2599f9673\" (UID: \"a828beed-f3fb-4da0-9345-edc2599f9673\") "
Sep 09 20:32:44.401316 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:44.401274    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kp2n8\" (UniqueName: \"kubernetes.io/projected/a828beed-f3fb-4da0-9345-edc2599f9673-kube-api-access-kp2n8\") pod \"a828beed-f3fb-4da0-9345-edc2599f9673\" (UID: \"a828beed-f3fb-4da0-9345-edc2599f9673\") "
Sep 09 20:32:44.403334 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:44.403303    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a828beed-f3fb-4da0-9345-edc2599f9673-kube-api-access-kp2n8" (OuterVolumeSpecName: "kube-api-access-kp2n8") pod "a828beed-f3fb-4da0-9345-edc2599f9673" (UID: "a828beed-f3fb-4da0-9345-edc2599f9673"). InnerVolumeSpecName "kube-api-access-kp2n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:32:44.403483 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:44.403357    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a828beed-f3fb-4da0-9345-edc2599f9673-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a828beed-f3fb-4da0-9345-edc2599f9673" (UID: "a828beed-f3fb-4da0-9345-edc2599f9673"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:32:44.502251 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:44.502218    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a828beed-f3fb-4da0-9345-edc2599f9673-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:32:44.502251 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:44.502247    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp2n8\" (UniqueName: \"kubernetes.io/projected/a828beed-f3fb-4da0-9345-edc2599f9673-kube-api-access-kp2n8\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:32:44.792883 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:44.792846    2581 generic.go:356] "Generic (PLEG): container finished" podID="a828beed-f3fb-4da0-9345-edc2599f9673" containerID="aad94ce223d227896c06a19adb22eebba04379ce67746b9bb155cf8437c54dc0" exitCode=0
Sep 09 20:32:44.793083 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:44.792894    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq"
Sep 09 20:32:44.793083 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:44.792936    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq" event={"ID":"a828beed-f3fb-4da0-9345-edc2599f9673","Type":"ContainerDied","Data":"aad94ce223d227896c06a19adb22eebba04379ce67746b9bb155cf8437c54dc0"}
Sep 09 20:32:44.793083 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:44.792999    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq" event={"ID":"a828beed-f3fb-4da0-9345-edc2599f9673","Type":"ContainerDied","Data":"ffcf50ba4b17c0d3f7fbb03ff8771274b1794e6702c6ed0cdd82bc49fbd4a463"}
Sep 09 20:32:44.793083 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:44.793029    2581 scope.go:117] "RemoveContainer" containerID="aad94ce223d227896c06a19adb22eebba04379ce67746b9bb155cf8437c54dc0"
Sep 09 20:32:44.801636 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:44.801613    2581 scope.go:117] "RemoveContainer" containerID="aad94ce223d227896c06a19adb22eebba04379ce67746b9bb155cf8437c54dc0"
Sep 09 20:32:44.801888 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:32:44.801866    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aad94ce223d227896c06a19adb22eebba04379ce67746b9bb155cf8437c54dc0\": container with ID starting with aad94ce223d227896c06a19adb22eebba04379ce67746b9bb155cf8437c54dc0 not found: ID does not exist" containerID="aad94ce223d227896c06a19adb22eebba04379ce67746b9bb155cf8437c54dc0"
Sep 09 20:32:44.801943 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:44.801898    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aad94ce223d227896c06a19adb22eebba04379ce67746b9bb155cf8437c54dc0"} err="failed to get container status \"aad94ce223d227896c06a19adb22eebba04379ce67746b9bb155cf8437c54dc0\": rpc error: code = NotFound desc = could not find container \"aad94ce223d227896c06a19adb22eebba04379ce67746b9bb155cf8437c54dc0\": container with ID starting with aad94ce223d227896c06a19adb22eebba04379ce67746b9bb155cf8437c54dc0 not found: ID does not exist"
Sep 09 20:32:44.812196 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:44.812167    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq"]
Sep 09 20:32:44.816361 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:44.816339    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbffljq"]
Sep 09 20:32:46.634696 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:46.634663    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a828beed-f3fb-4da0-9345-edc2599f9673" path="/var/lib/kubelet/pods/a828beed-f3fb-4da0-9345-edc2599f9673/volumes"
Sep 09 20:32:55.207526 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:55.207485    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld"]
Sep 09 20:32:55.207988 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:55.207924    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a828beed-f3fb-4da0-9345-edc2599f9673" containerName="main"
Sep 09 20:32:55.207988 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:55.207946    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a828beed-f3fb-4da0-9345-edc2599f9673" containerName="main"
Sep 09 20:32:55.207988 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:55.207973    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b8b37df-7265-4162-bcb7-7819b63429b7" containerName="main"
Sep 09 20:32:55.207988 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:55.207982    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b8b37df-7265-4162-bcb7-7819b63429b7" containerName="main"
Sep 09 20:32:55.208143 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:55.208075    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a828beed-f3fb-4da0-9345-edc2599f9673" containerName="main"
Sep 09 20:32:55.208143 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:55.208092    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b8b37df-7265-4162-bcb7-7819b63429b7" containerName="main"
Sep 09 20:32:55.220342 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:55.220316    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld"
Sep 09 20:32:55.221369 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:55.221341    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld"]
Sep 09 20:32:55.222943 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:55.222915    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 09 20:32:55.224095 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:55.223947    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-6fw4v\""
Sep 09 20:32:55.224095 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:55.223959    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 09 20:32:55.224095 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:55.223980    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 09 20:32:55.224095 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:55.223993    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-pwx87\""
Sep 09 20:32:55.290748 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:55.290703    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b4qr\" (UniqueName: \"kubernetes.io/projected/909f8be9-92b8-4614-9b1b-61e2f5c2445a-kube-api-access-5b4qr\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld\" (UID: \"909f8be9-92b8-4614-9b1b-61e2f5c2445a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld"
Sep 09 20:32:55.290748 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:55.290754    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909f8be9-92b8-4614-9b1b-61e2f5c2445a-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld\" (UID: \"909f8be9-92b8-4614-9b1b-61e2f5c2445a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld"
Sep 09 20:32:55.392086 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:55.392032    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b4qr\" (UniqueName: \"kubernetes.io/projected/909f8be9-92b8-4614-9b1b-61e2f5c2445a-kube-api-access-5b4qr\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld\" (UID: \"909f8be9-92b8-4614-9b1b-61e2f5c2445a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld"
Sep 09 20:32:55.392086 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:55.392094    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909f8be9-92b8-4614-9b1b-61e2f5c2445a-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld\" (UID: \"909f8be9-92b8-4614-9b1b-61e2f5c2445a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld"
Sep 09 20:32:55.394520 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:55.394499    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909f8be9-92b8-4614-9b1b-61e2f5c2445a-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld\" (UID: \"909f8be9-92b8-4614-9b1b-61e2f5c2445a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld"
Sep 09 20:32:55.400358 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:55.400337    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b4qr\" (UniqueName: \"kubernetes.io/projected/909f8be9-92b8-4614-9b1b-61e2f5c2445a-kube-api-access-5b4qr\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld\" (UID: \"909f8be9-92b8-4614-9b1b-61e2f5c2445a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld"
Sep 09 20:32:55.534574 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:55.534537    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld"
Sep 09 20:32:55.806506 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:55.806479    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld"]
Sep 09 20:32:55.808585 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:32:55.808555    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909f8be9_92b8_4614_9b1b_61e2f5c2445a.slice/crio-f0dcfc0614949c23846a1eab5e15c802c9c82bc752445f3af43a83d63234b7fd WatchSource:0}: Error finding container f0dcfc0614949c23846a1eab5e15c802c9c82bc752445f3af43a83d63234b7fd: Status 404 returned error can't find the container with id f0dcfc0614949c23846a1eab5e15c802c9c82bc752445f3af43a83d63234b7fd
Sep 09 20:32:55.851413 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:55.851383    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld" event={"ID":"909f8be9-92b8-4614-9b1b-61e2f5c2445a","Type":"ContainerStarted","Data":"f0dcfc0614949c23846a1eab5e15c802c9c82bc752445f3af43a83d63234b7fd"}
Sep 09 20:32:56.858536 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:56.858495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld" event={"ID":"909f8be9-92b8-4614-9b1b-61e2f5c2445a","Type":"ContainerStarted","Data":"a09273a9294be9c5f5a6f641100245994acab5d099b8edcae9f3409ccbe49b84"}
Sep 09 20:32:56.858925 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:56.858574    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld"
Sep 09 20:32:56.876829 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:32:56.876767    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld" podStartSLOduration=1.876750699 podStartE2EDuration="1.876750699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:32:56.875018669 +0000 UTC m=+1318.880195658" watchObservedRunningTime="2026-09-09 20:32:56.876750699 +0000 UTC m=+1318.881927694"
Sep 09 20:33:07.162662 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:07.162616    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528"]
Sep 09 20:33:07.167496 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:07.167473    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528"
Sep 09 20:33:07.169960 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:07.169937    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 09 20:33:07.171053 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:07.171030    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 09 20:33:07.171165 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:07.171090    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 09 20:33:07.171165 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:07.171160    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-kdkj6\""
Sep 09 20:33:07.171289 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:07.171210    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-zx88m\""
Sep 09 20:33:07.176671 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:07.176647    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528"]
Sep 09 20:33:07.296360 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:07.296318    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3530c720-1a21-48c8-8be3-08642ca518d5-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528\" (UID: \"3530c720-1a21-48c8-8be3-08642ca518d5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528"
Sep 09 20:33:07.296580 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:07.296385    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbpbf\" (UniqueName: \"kubernetes.io/projected/3530c720-1a21-48c8-8be3-08642ca518d5-kube-api-access-gbpbf\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528\" (UID: \"3530c720-1a21-48c8-8be3-08642ca518d5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528"
Sep 09 20:33:07.397129 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:07.397088    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3530c720-1a21-48c8-8be3-08642ca518d5-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528\" (UID: \"3530c720-1a21-48c8-8be3-08642ca518d5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528"
Sep 09 20:33:07.397293 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:07.397246    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbpbf\" (UniqueName: \"kubernetes.io/projected/3530c720-1a21-48c8-8be3-08642ca518d5-kube-api-access-gbpbf\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528\" (UID: \"3530c720-1a21-48c8-8be3-08642ca518d5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528"
Sep 09 20:33:07.399780 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:07.399757    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3530c720-1a21-48c8-8be3-08642ca518d5-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528\" (UID: \"3530c720-1a21-48c8-8be3-08642ca518d5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528"
Sep 09 20:33:07.406189 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:07.406165    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbpbf\" (UniqueName: \"kubernetes.io/projected/3530c720-1a21-48c8-8be3-08642ca518d5-kube-api-access-gbpbf\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528\" (UID: \"3530c720-1a21-48c8-8be3-08642ca518d5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528"
Sep 09 20:33:07.480236 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:07.480208    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528"
Sep 09 20:33:07.756356 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:07.756319    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528"]
Sep 09 20:33:07.757625 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:33:07.757595    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3530c720_1a21_48c8_8be3_08642ca518d5.slice/crio-90a0f99dc9d92d8b5e60be7e557f6e7c699a90d48a7df4955b30c6ee54c0d146 WatchSource:0}: Error finding container 90a0f99dc9d92d8b5e60be7e557f6e7c699a90d48a7df4955b30c6ee54c0d146: Status 404 returned error can't find the container with id 90a0f99dc9d92d8b5e60be7e557f6e7c699a90d48a7df4955b30c6ee54c0d146
Sep 09 20:33:07.915236 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:07.915196    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528" event={"ID":"3530c720-1a21-48c8-8be3-08642ca518d5","Type":"ContainerStarted","Data":"754b27db4ef5237d3620aa86ae4474362b4c1c1c464680a3db4109672c3fadc2"}
Sep 09 20:33:07.915392 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:07.915259    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528" event={"ID":"3530c720-1a21-48c8-8be3-08642ca518d5","Type":"ContainerStarted","Data":"90a0f99dc9d92d8b5e60be7e557f6e7c699a90d48a7df4955b30c6ee54c0d146"}
Sep 09 20:33:07.915392 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:07.915285    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528"
Sep 09 20:33:07.933000 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:07.932946    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528" podStartSLOduration=0.932931119 podStartE2EDuration="932.931119ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:07.930617855 +0000 UTC m=+1329.935794869" watchObservedRunningTime="2026-09-09 20:33:07.932931119 +0000 UTC m=+1329.938108111"
Sep 09 20:33:27.866085 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:27.866003    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld"
Sep 09 20:33:38.921718 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:38.921686    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528"
Sep 09 20:33:40.479881 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:40.479849    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528"]
Sep 09 20:33:48.922276 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:48.922238    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528" podUID="3530c720-1a21-48c8-8be3-08642ca518d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:33:55.506070 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:55.506027    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528" podUID="3530c720-1a21-48c8-8be3-08642ca518d5" containerName="main" containerID="cri-o://754b27db4ef5237d3620aa86ae4474362b4c1c1c464680a3db4109672c3fadc2" gracePeriod=45
Sep 09 20:33:58.922571 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:33:58.922527    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528" podUID="3530c720-1a21-48c8-8be3-08642ca518d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:34:08.922163 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:08.922128    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528" podUID="3530c720-1a21-48c8-8be3-08642ca518d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:34:08.922575 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:08.922240    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528"
Sep 09 20:34:18.921798 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:18.921755    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528" podUID="3530c720-1a21-48c8-8be3-08642ca518d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:34:25.748789 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:25.748766    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528"
Sep 09 20:34:25.852769 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:25.852688    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3530c720-1a21-48c8-8be3-08642ca518d5-tls-certs\") pod \"3530c720-1a21-48c8-8be3-08642ca518d5\" (UID: \"3530c720-1a21-48c8-8be3-08642ca518d5\") "
Sep 09 20:34:25.852769 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:25.852755    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbpbf\" (UniqueName: \"kubernetes.io/projected/3530c720-1a21-48c8-8be3-08642ca518d5-kube-api-access-gbpbf\") pod \"3530c720-1a21-48c8-8be3-08642ca518d5\" (UID: \"3530c720-1a21-48c8-8be3-08642ca518d5\") "
Sep 09 20:34:25.854949 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:25.854925    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3530c720-1a21-48c8-8be3-08642ca518d5-kube-api-access-gbpbf" (OuterVolumeSpecName: "kube-api-access-gbpbf") pod "3530c720-1a21-48c8-8be3-08642ca518d5" (UID: "3530c720-1a21-48c8-8be3-08642ca518d5"). InnerVolumeSpecName "kube-api-access-gbpbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:34:25.855049 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:25.854925    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3530c720-1a21-48c8-8be3-08642ca518d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3530c720-1a21-48c8-8be3-08642ca518d5" (UID: "3530c720-1a21-48c8-8be3-08642ca518d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:34:25.953794 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:25.953759    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbpbf\" (UniqueName: \"kubernetes.io/projected/3530c720-1a21-48c8-8be3-08642ca518d5-kube-api-access-gbpbf\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:34:25.953794 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:25.953794    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3530c720-1a21-48c8-8be3-08642ca518d5-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:34:26.302503 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:26.302439    2581 generic.go:356] "Generic (PLEG): container finished" podID="3530c720-1a21-48c8-8be3-08642ca518d5" containerID="754b27db4ef5237d3620aa86ae4474362b4c1c1c464680a3db4109672c3fadc2" exitCode=0
Sep 09 20:34:26.302699 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:26.302518    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528"
Sep 09 20:34:26.302699 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:26.302532    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528" event={"ID":"3530c720-1a21-48c8-8be3-08642ca518d5","Type":"ContainerDied","Data":"754b27db4ef5237d3620aa86ae4474362b4c1c1c464680a3db4109672c3fadc2"}
Sep 09 20:34:26.302699 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:26.302622    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528" event={"ID":"3530c720-1a21-48c8-8be3-08642ca518d5","Type":"ContainerDied","Data":"90a0f99dc9d92d8b5e60be7e557f6e7c699a90d48a7df4955b30c6ee54c0d146"}
Sep 09 20:34:26.302699 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:26.302646    2581 scope.go:117] "RemoveContainer" containerID="754b27db4ef5237d3620aa86ae4474362b4c1c1c464680a3db4109672c3fadc2"
Sep 09 20:34:26.311663 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:26.311648    2581 scope.go:117] "RemoveContainer" containerID="754b27db4ef5237d3620aa86ae4474362b4c1c1c464680a3db4109672c3fadc2"
Sep 09 20:34:26.311942 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:34:26.311922    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"754b27db4ef5237d3620aa86ae4474362b4c1c1c464680a3db4109672c3fadc2\": container with ID starting with 754b27db4ef5237d3620aa86ae4474362b4c1c1c464680a3db4109672c3fadc2 not found: ID does not exist" containerID="754b27db4ef5237d3620aa86ae4474362b4c1c1c464680a3db4109672c3fadc2"
Sep 09 20:34:26.311988 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:26.311950    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"754b27db4ef5237d3620aa86ae4474362b4c1c1c464680a3db4109672c3fadc2"} err="failed to get container status \"754b27db4ef5237d3620aa86ae4474362b4c1c1c464680a3db4109672c3fadc2\": rpc error: code = NotFound desc = could not find container \"754b27db4ef5237d3620aa86ae4474362b4c1c1c464680a3db4109672c3fadc2\": container with ID starting with 754b27db4ef5237d3620aa86ae4474362b4c1c1c464680a3db4109672c3fadc2 not found: ID does not exist"
Sep 09 20:34:26.324675 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:26.324652    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528"]
Sep 09 20:34:26.328752 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:26.328730    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche87528"]
Sep 09 20:34:26.635639 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:26.635563    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3530c720-1a21-48c8-8be3-08642ca518d5" path="/var/lib/kubelet/pods/3530c720-1a21-48c8-8be3-08642ca518d5/volumes"
Sep 09 20:34:38.061926 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:38.061892    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw"]
Sep 09 20:34:38.062537 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:38.062249    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3530c720-1a21-48c8-8be3-08642ca518d5" containerName="main"
Sep 09 20:34:38.062537 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:38.062260    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3530c720-1a21-48c8-8be3-08642ca518d5" containerName="main"
Sep 09 20:34:38.062537 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:38.062311    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3530c720-1a21-48c8-8be3-08642ca518d5" containerName="main"
Sep 09 20:34:38.065377 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:38.065353    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw"
Sep 09 20:34:38.069148 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:38.069002    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-rfl8n\""
Sep 09 20:34:38.069148 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:38.069002    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 09 20:34:38.069148 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:38.069007    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-gqknt\""
Sep 09 20:34:38.069148 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:38.069058    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 09 20:34:38.069148 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:38.069101    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 09 20:34:38.076007 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:38.075985    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw"]
Sep 09 20:34:38.153436 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:38.153400    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2w57\" (UniqueName: \"kubernetes.io/projected/e0456889-f1db-4400-b3c7-09b4e8b7c79d-kube-api-access-v2w57\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw\" (UID: \"e0456889-f1db-4400-b3c7-09b4e8b7c79d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw"
Sep 09 20:34:38.153621 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:38.153443    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0456889-f1db-4400-b3c7-09b4e8b7c79d-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw\" (UID: \"e0456889-f1db-4400-b3c7-09b4e8b7c79d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw"
Sep 09 20:34:38.253873 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:38.253841    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2w57\" (UniqueName: \"kubernetes.io/projected/e0456889-f1db-4400-b3c7-09b4e8b7c79d-kube-api-access-v2w57\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw\" (UID: \"e0456889-f1db-4400-b3c7-09b4e8b7c79d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw"
Sep 09 20:34:38.254053 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:38.253879    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0456889-f1db-4400-b3c7-09b4e8b7c79d-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw\" (UID: \"e0456889-f1db-4400-b3c7-09b4e8b7c79d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw"
Sep 09 20:34:38.256429 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:38.256399    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0456889-f1db-4400-b3c7-09b4e8b7c79d-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw\" (UID: \"e0456889-f1db-4400-b3c7-09b4e8b7c79d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw"
Sep 09 20:34:38.263751 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:38.263729    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2w57\" (UniqueName: \"kubernetes.io/projected/e0456889-f1db-4400-b3c7-09b4e8b7c79d-kube-api-access-v2w57\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw\" (UID: \"e0456889-f1db-4400-b3c7-09b4e8b7c79d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw"
Sep 09 20:34:38.378095 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:38.378015    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw"
Sep 09 20:34:38.652829 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:38.652801    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw"]
Sep 09 20:34:38.654288 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:34:38.654261    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0456889_f1db_4400_b3c7_09b4e8b7c79d.slice/crio-b0f957515026f6b5877e778f2a399cd06692448f50916b566f868972a16d4a03 WatchSource:0}: Error finding container b0f957515026f6b5877e778f2a399cd06692448f50916b566f868972a16d4a03: Status 404 returned error can't find the container with id b0f957515026f6b5877e778f2a399cd06692448f50916b566f868972a16d4a03
Sep 09 20:34:39.368264 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:39.368231    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw" event={"ID":"e0456889-f1db-4400-b3c7-09b4e8b7c79d","Type":"ContainerStarted","Data":"bce222a9847d8b8d0d6ac14e581589b306d93042264c5aa3822a3aa6c4219a44"}
Sep 09 20:34:39.368708 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:39.368287    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw" event={"ID":"e0456889-f1db-4400-b3c7-09b4e8b7c79d","Type":"ContainerStarted","Data":"b0f957515026f6b5877e778f2a399cd06692448f50916b566f868972a16d4a03"}
Sep 09 20:34:39.368708 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:39.368310    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw"
Sep 09 20:34:39.386925 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:39.386797    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw" podStartSLOduration=1.386782856 podStartE2EDuration="1.386782856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:34:39.384877208 +0000 UTC m=+1421.390054206" watchObservedRunningTime="2026-09-09 20:34:39.386782856 +0000 UTC m=+1421.391959851"
Sep 09 20:34:56.978708 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:56.978672    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld"]
Sep 09 20:34:57.866639 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:34:57.866595    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld" podUID="909f8be9-92b8-4614-9b1b-61e2f5c2445a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:35:07.866085 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:07.866037    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld" podUID="909f8be9-92b8-4614-9b1b-61e2f5c2445a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:35:10.374372 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:10.374341    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw"
Sep 09 20:35:11.999939 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:11.999893    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld" podUID="909f8be9-92b8-4614-9b1b-61e2f5c2445a" containerName="main" containerID="cri-o://a09273a9294be9c5f5a6f641100245994acab5d099b8edcae9f3409ccbe49b84" gracePeriod=45
Sep 09 20:35:12.087782 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:12.087748    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw"]
Sep 09 20:35:17.866180 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:17.866142    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld" podUID="909f8be9-92b8-4614-9b1b-61e2f5c2445a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:35:17.866628 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:17.866250    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld"
Sep 09 20:35:20.374546 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:20.374500    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw" podUID="e0456889-f1db-4400-b3c7-09b4e8b7c79d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:35:27.108864 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:27.108816    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw" podUID="e0456889-f1db-4400-b3c7-09b4e8b7c79d" containerName="main" containerID="cri-o://bce222a9847d8b8d0d6ac14e581589b306d93042264c5aa3822a3aa6c4219a44" gracePeriod=45
Sep 09 20:35:27.865888 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:27.865839    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld" podUID="909f8be9-92b8-4614-9b1b-61e2f5c2445a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:35:30.374499 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:30.374434    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw" podUID="e0456889-f1db-4400-b3c7-09b4e8b7c79d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:35:37.866311 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:37.866276    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld" podUID="909f8be9-92b8-4614-9b1b-61e2f5c2445a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:35:40.374410 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:40.374373    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw" podUID="e0456889-f1db-4400-b3c7-09b4e8b7c79d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:35:40.374829 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:40.374536    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw"
Sep 09 20:35:42.253635 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:42.253612    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld"
Sep 09 20:35:42.296411 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:42.296335    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5b4qr\" (UniqueName: \"kubernetes.io/projected/909f8be9-92b8-4614-9b1b-61e2f5c2445a-kube-api-access-5b4qr\") pod \"909f8be9-92b8-4614-9b1b-61e2f5c2445a\" (UID: \"909f8be9-92b8-4614-9b1b-61e2f5c2445a\") "
Sep 09 20:35:42.296411 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:42.296397    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909f8be9-92b8-4614-9b1b-61e2f5c2445a-tls-certs\") pod \"909f8be9-92b8-4614-9b1b-61e2f5c2445a\" (UID: \"909f8be9-92b8-4614-9b1b-61e2f5c2445a\") "
Sep 09 20:35:42.298666 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:42.298630    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/909f8be9-92b8-4614-9b1b-61e2f5c2445a-kube-api-access-5b4qr" (OuterVolumeSpecName: "kube-api-access-5b4qr") pod "909f8be9-92b8-4614-9b1b-61e2f5c2445a" (UID: "909f8be9-92b8-4614-9b1b-61e2f5c2445a"). InnerVolumeSpecName "kube-api-access-5b4qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:42.298666 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:42.298654    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/909f8be9-92b8-4614-9b1b-61e2f5c2445a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "909f8be9-92b8-4614-9b1b-61e2f5c2445a" (UID: "909f8be9-92b8-4614-9b1b-61e2f5c2445a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:42.397103 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:42.397071    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909f8be9-92b8-4614-9b1b-61e2f5c2445a-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:35:42.397103 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:42.397098    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b4qr\" (UniqueName: \"kubernetes.io/projected/909f8be9-92b8-4614-9b1b-61e2f5c2445a-kube-api-access-5b4qr\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:35:42.691908 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:42.691877    2581 generic.go:356] "Generic (PLEG): container finished" podID="909f8be9-92b8-4614-9b1b-61e2f5c2445a" containerID="a09273a9294be9c5f5a6f641100245994acab5d099b8edcae9f3409ccbe49b84" exitCode=0
Sep 09 20:35:42.692060 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:42.691917    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld"
Sep 09 20:35:42.692060 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:42.691967    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld" event={"ID":"909f8be9-92b8-4614-9b1b-61e2f5c2445a","Type":"ContainerDied","Data":"a09273a9294be9c5f5a6f641100245994acab5d099b8edcae9f3409ccbe49b84"}
Sep 09 20:35:42.692060 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:42.692015    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld" event={"ID":"909f8be9-92b8-4614-9b1b-61e2f5c2445a","Type":"ContainerDied","Data":"f0dcfc0614949c23846a1eab5e15c802c9c82bc752445f3af43a83d63234b7fd"}
Sep 09 20:35:42.692060 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:42.692036    2581 scope.go:117] "RemoveContainer" containerID="a09273a9294be9c5f5a6f641100245994acab5d099b8edcae9f3409ccbe49b84"
Sep 09 20:35:42.700643 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:42.700624    2581 scope.go:117] "RemoveContainer" containerID="a09273a9294be9c5f5a6f641100245994acab5d099b8edcae9f3409ccbe49b84"
Sep 09 20:35:42.700906 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:35:42.700887    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a09273a9294be9c5f5a6f641100245994acab5d099b8edcae9f3409ccbe49b84\": container with ID starting with a09273a9294be9c5f5a6f641100245994acab5d099b8edcae9f3409ccbe49b84 not found: ID does not exist" containerID="a09273a9294be9c5f5a6f641100245994acab5d099b8edcae9f3409ccbe49b84"
Sep 09 20:35:42.700972 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:42.700917    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a09273a9294be9c5f5a6f641100245994acab5d099b8edcae9f3409ccbe49b84"} err="failed to get container status \"a09273a9294be9c5f5a6f641100245994acab5d099b8edcae9f3409ccbe49b84\": rpc error: code = NotFound desc = could not find container \"a09273a9294be9c5f5a6f641100245994acab5d099b8edcae9f3409ccbe49b84\": container with ID starting with a09273a9294be9c5f5a6f641100245994acab5d099b8edcae9f3409ccbe49b84 not found: ID does not exist"
Sep 09 20:35:42.711864 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:42.711841    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld"]
Sep 09 20:35:42.717163 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:42.717141    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ch6dld"]
Sep 09 20:35:44.637984 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:44.637947    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="909f8be9-92b8-4614-9b1b-61e2f5c2445a" path="/var/lib/kubelet/pods/909f8be9-92b8-4614-9b1b-61e2f5c2445a/volumes"
Sep 09 20:35:50.374784 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:50.374749    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw" podUID="e0456889-f1db-4400-b3c7-09b4e8b7c79d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:35:57.358920 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:57.358891    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw"
Sep 09 20:35:57.423261 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:57.423175    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0456889-f1db-4400-b3c7-09b4e8b7c79d-tls-certs\") pod \"e0456889-f1db-4400-b3c7-09b4e8b7c79d\" (UID: \"e0456889-f1db-4400-b3c7-09b4e8b7c79d\") "
Sep 09 20:35:57.423261 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:57.423253    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2w57\" (UniqueName: \"kubernetes.io/projected/e0456889-f1db-4400-b3c7-09b4e8b7c79d-kube-api-access-v2w57\") pod \"e0456889-f1db-4400-b3c7-09b4e8b7c79d\" (UID: \"e0456889-f1db-4400-b3c7-09b4e8b7c79d\") "
Sep 09 20:35:57.425418 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:57.425384    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0456889-f1db-4400-b3c7-09b4e8b7c79d-kube-api-access-v2w57" (OuterVolumeSpecName: "kube-api-access-v2w57") pod "e0456889-f1db-4400-b3c7-09b4e8b7c79d" (UID: "e0456889-f1db-4400-b3c7-09b4e8b7c79d"). InnerVolumeSpecName "kube-api-access-v2w57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:57.425418 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:57.425397    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0456889-f1db-4400-b3c7-09b4e8b7c79d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0456889-f1db-4400-b3c7-09b4e8b7c79d" (UID: "e0456889-f1db-4400-b3c7-09b4e8b7c79d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:57.524307 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:57.524268    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2w57\" (UniqueName: \"kubernetes.io/projected/e0456889-f1db-4400-b3c7-09b4e8b7c79d-kube-api-access-v2w57\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:35:57.524307 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:57.524298    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0456889-f1db-4400-b3c7-09b4e8b7c79d-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:35:57.766335 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:57.766302    2581 generic.go:356] "Generic (PLEG): container finished" podID="e0456889-f1db-4400-b3c7-09b4e8b7c79d" containerID="bce222a9847d8b8d0d6ac14e581589b306d93042264c5aa3822a3aa6c4219a44" exitCode=0
Sep 09 20:35:57.766513 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:57.766350    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw"
Sep 09 20:35:57.766513 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:57.766390    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw" event={"ID":"e0456889-f1db-4400-b3c7-09b4e8b7c79d","Type":"ContainerDied","Data":"bce222a9847d8b8d0d6ac14e581589b306d93042264c5aa3822a3aa6c4219a44"}
Sep 09 20:35:57.766513 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:57.766445    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw" event={"ID":"e0456889-f1db-4400-b3c7-09b4e8b7c79d","Type":"ContainerDied","Data":"b0f957515026f6b5877e778f2a399cd06692448f50916b566f868972a16d4a03"}
Sep 09 20:35:57.766513 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:57.766492    2581 scope.go:117] "RemoveContainer" containerID="bce222a9847d8b8d0d6ac14e581589b306d93042264c5aa3822a3aa6c4219a44"
Sep 09 20:35:57.775192 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:57.775176    2581 scope.go:117] "RemoveContainer" containerID="bce222a9847d8b8d0d6ac14e581589b306d93042264c5aa3822a3aa6c4219a44"
Sep 09 20:35:57.775435 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:35:57.775417    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce222a9847d8b8d0d6ac14e581589b306d93042264c5aa3822a3aa6c4219a44\": container with ID starting with bce222a9847d8b8d0d6ac14e581589b306d93042264c5aa3822a3aa6c4219a44 not found: ID does not exist" containerID="bce222a9847d8b8d0d6ac14e581589b306d93042264c5aa3822a3aa6c4219a44"
Sep 09 20:35:57.775517 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:57.775442    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce222a9847d8b8d0d6ac14e581589b306d93042264c5aa3822a3aa6c4219a44"} err="failed to get container status \"bce222a9847d8b8d0d6ac14e581589b306d93042264c5aa3822a3aa6c4219a44\": rpc error: code = NotFound desc = could not find container \"bce222a9847d8b8d0d6ac14e581589b306d93042264c5aa3822a3aa6c4219a44\": container with ID starting with bce222a9847d8b8d0d6ac14e581589b306d93042264c5aa3822a3aa6c4219a44 not found: ID does not exist"
Sep 09 20:35:57.787717 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:57.787687    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw"]
Sep 09 20:35:57.791698 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:57.791673    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wz5fw"]
Sep 09 20:35:58.525884 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:58.525848    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 20:35:58.526829 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:58.526811    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 20:35:58.528591 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:58.528571    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 20:35:58.529616 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:58.529597    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 20:35:58.634862 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:35:58.634823    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0456889-f1db-4400-b3c7-09b4e8b7c79d" path="/var/lib/kubelet/pods/e0456889-f1db-4400-b3c7-09b4e8b7c79d/volumes"
Sep 09 20:36:22.342663 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.342581    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"]
Sep 09 20:36:22.343096 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.342962    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="909f8be9-92b8-4614-9b1b-61e2f5c2445a" containerName="main"
Sep 09 20:36:22.343096 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.342977    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="909f8be9-92b8-4614-9b1b-61e2f5c2445a" containerName="main"
Sep 09 20:36:22.343096 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.342985    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0456889-f1db-4400-b3c7-09b4e8b7c79d" containerName="main"
Sep 09 20:36:22.343096 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.342991    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0456889-f1db-4400-b3c7-09b4e8b7c79d" containerName="main"
Sep 09 20:36:22.343096 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.343054    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="909f8be9-92b8-4614-9b1b-61e2f5c2445a" containerName="main"
Sep 09 20:36:22.343096 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.343062    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0456889-f1db-4400-b3c7-09b4e8b7c79d" containerName="main"
Sep 09 20:36:22.345745 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.345730    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:22.349630 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.349572    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 09 20:36:22.349630 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.349603    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 09 20:36:22.349832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.349634    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-zzqnc\""
Sep 09 20:36:22.349832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.349649    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 09 20:36:22.353918 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.353897    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"]
Sep 09 20:36:22.434815 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.434774    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6cc7f01-228d-4141-8dfe-4b868650ad5c-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf\" (UID: \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:22.434994 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.434825    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6cc7f01-228d-4141-8dfe-4b868650ad5c-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf\" (UID: \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:22.434994 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.434905    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgprq\" (UniqueName: \"kubernetes.io/projected/f6cc7f01-228d-4141-8dfe-4b868650ad5c-kube-api-access-kgprq\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf\" (UID: \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:22.434994 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.434948    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6cc7f01-228d-4141-8dfe-4b868650ad5c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf\" (UID: \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:22.435129 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.435006    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6cc7f01-228d-4141-8dfe-4b868650ad5c-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf\" (UID: \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:22.435129 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.435032    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6cc7f01-228d-4141-8dfe-4b868650ad5c-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf\" (UID: \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:22.536149 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.536110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6cc7f01-228d-4141-8dfe-4b868650ad5c-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf\" (UID: \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:22.536149 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.536150    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6cc7f01-228d-4141-8dfe-4b868650ad5c-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf\" (UID: \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:22.536381 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.536171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6cc7f01-228d-4141-8dfe-4b868650ad5c-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf\" (UID: \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:22.536381 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.536200    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6cc7f01-228d-4141-8dfe-4b868650ad5c-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf\" (UID: \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:22.536381 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.536232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgprq\" (UniqueName: \"kubernetes.io/projected/f6cc7f01-228d-4141-8dfe-4b868650ad5c-kube-api-access-kgprq\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf\" (UID: \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:22.536381 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.536259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6cc7f01-228d-4141-8dfe-4b868650ad5c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf\" (UID: \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:22.536709 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.536682    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6cc7f01-228d-4141-8dfe-4b868650ad5c-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf\" (UID: \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:22.536938 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.536701    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6cc7f01-228d-4141-8dfe-4b868650ad5c-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf\" (UID: \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:22.536938 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.536701    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6cc7f01-228d-4141-8dfe-4b868650ad5c-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf\" (UID: \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:22.538720 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.538691    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6cc7f01-228d-4141-8dfe-4b868650ad5c-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf\" (UID: \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:22.539199 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.539172    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6cc7f01-228d-4141-8dfe-4b868650ad5c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf\" (UID: \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:22.544950 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.544906    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb"]
Sep 09 20:36:22.548671 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.548650    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb"
Sep 09 20:36:22.548797 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.548762    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgprq\" (UniqueName: \"kubernetes.io/projected/f6cc7f01-228d-4141-8dfe-4b868650ad5c-kube-api-access-kgprq\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf\" (UID: \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:22.551119 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.551095    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-kl5vr\""
Sep 09 20:36:22.556994 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.556957    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb"]
Sep 09 20:36:22.637669 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.637596    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb\" (UID: \"59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb"
Sep 09 20:36:22.637669 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.637654    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6rck\" (UniqueName: \"kubernetes.io/projected/59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b-kube-api-access-r6rck\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb\" (UID: \"59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb"
Sep 09 20:36:22.657599 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.657573    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:22.738267 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.738228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb\" (UID: \"59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb"
Sep 09 20:36:22.738446 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.738368    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6rck\" (UniqueName: \"kubernetes.io/projected/59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b-kube-api-access-r6rck\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb\" (UID: \"59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb"
Sep 09 20:36:22.740893 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.740869    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb\" (UID: \"59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb"
Sep 09 20:36:22.747255 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.747237    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6rck\" (UniqueName: \"kubernetes.io/projected/59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b-kube-api-access-r6rck\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb\" (UID: \"59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb"
Sep 09 20:36:22.874142 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.874109    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb"
Sep 09 20:36:22.929774 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:22.929675    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"]
Sep 09 20:36:23.150748 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:23.150724    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb"]
Sep 09 20:36:23.152594 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:36:23.152566    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e8ea5d_b959_4882_8f30_7a9ee1b2ee5b.slice/crio-9e0a707c698c6bf536593203bb1c449e61d0e46e84543b549203738e419763eb WatchSource:0}: Error finding container 9e0a707c698c6bf536593203bb1c449e61d0e46e84543b549203738e419763eb: Status 404 returned error can't find the container with id 9e0a707c698c6bf536593203bb1c449e61d0e46e84543b549203738e419763eb
Sep 09 20:36:23.898259 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:23.898217    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf" event={"ID":"f6cc7f01-228d-4141-8dfe-4b868650ad5c","Type":"ContainerStarted","Data":"0d9b453b414b6e874c8cc305c3b260d47728cfcc14d4a7656c2842d8204ff631"}
Sep 09 20:36:23.898720 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:23.898287    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf" event={"ID":"f6cc7f01-228d-4141-8dfe-4b868650ad5c","Type":"ContainerStarted","Data":"3009e2ca9e9d3e733ad9c3b4dee98d4403c23adf2e77c9a43fe92e03637dd8d6"}
Sep 09 20:36:23.899819 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:23.899793    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb" event={"ID":"59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b","Type":"ContainerStarted","Data":"6519db8d1728a50d5193f6b3673477b7da6351134d7ab75f474e0a264f1b60f0"}
Sep 09 20:36:23.899943 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:23.899842    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb" event={"ID":"59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b","Type":"ContainerStarted","Data":"9e0a707c698c6bf536593203bb1c449e61d0e46e84543b549203738e419763eb"}
Sep 09 20:36:23.899943 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:23.899883    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb"
Sep 09 20:36:23.916255 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:23.916202    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf" podStartSLOduration=1.916188104 podStartE2EDuration="1.916188104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:23.914318573 +0000 UTC m=+1525.919495583" watchObservedRunningTime="2026-09-09 20:36:23.916188104 +0000 UTC m=+1525.921365099"
Sep 09 20:36:23.930338 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:23.930290    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb" podStartSLOduration=1.9302751200000001 podStartE2EDuration="1.93027512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:23.929094612 +0000 UTC m=+1525.934271631" watchObservedRunningTime="2026-09-09 20:36:23.93027512 +0000 UTC m=+1525.935452114"
Sep 09 20:36:24.657965 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:24.657930    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:32.658540 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:32.658503    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:32.671070 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:32.671044    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:32.960475 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:32.960365    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:54.906761 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:54.906728    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb"
Sep 09 20:36:56.520777 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:56.520736    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"]
Sep 09 20:36:56.530807 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:36:56.530770    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:36:56.530807 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:36:56.530807 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:36:56.530807 ip-10-0-134-84 kubenswrapper[2581]:  > containerID="0d9b453b414b6e874c8cc305c3b260d47728cfcc14d4a7656c2842d8204ff631" cmd=["/bin/sleep","15"]
Sep 09 20:36:56.531077 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:56.530822    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf" message="" err=<
Sep 09 20:36:56.531077 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:36:56.531077 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:36:56.531077 ip-10-0-134-84 kubenswrapper[2581]:  >
Sep 09 20:36:56.531077 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:36:56.530834    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:36:56.531077 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:36:56.531077 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:36:56.531077 ip-10-0-134-84 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf" podUID="f6cc7f01-228d-4141-8dfe-4b868650ad5c" containerName="main" containerID="cri-o://0d9b453b414b6e874c8cc305c3b260d47728cfcc14d4a7656c2842d8204ff631"
Sep 09 20:36:56.531077 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:56.530876    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf" podUID="f6cc7f01-228d-4141-8dfe-4b868650ad5c" containerName="main" containerID="cri-o://0d9b453b414b6e874c8cc305c3b260d47728cfcc14d4a7656c2842d8204ff631" gracePeriod=60
Sep 09 20:36:56.535619 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:56.535591    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb"]
Sep 09 20:36:56.875537 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:56.875516    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:57.031279 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.031245    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6cc7f01-228d-4141-8dfe-4b868650ad5c-dshm\") pod \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\" (UID: \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\") "
Sep 09 20:36:57.031505 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.031299    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6cc7f01-228d-4141-8dfe-4b868650ad5c-tmp-dir\") pod \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\" (UID: \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\") "
Sep 09 20:36:57.031505 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.031334    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgprq\" (UniqueName: \"kubernetes.io/projected/f6cc7f01-228d-4141-8dfe-4b868650ad5c-kube-api-access-kgprq\") pod \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\" (UID: \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\") "
Sep 09 20:36:57.031505 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.031369    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6cc7f01-228d-4141-8dfe-4b868650ad5c-model-cache\") pod \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\" (UID: \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\") "
Sep 09 20:36:57.031505 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.031415    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6cc7f01-228d-4141-8dfe-4b868650ad5c-tls-certs\") pod \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\" (UID: \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\") "
Sep 09 20:36:57.031751 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.031519    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6cc7f01-228d-4141-8dfe-4b868650ad5c-home\") pod \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\" (UID: \"f6cc7f01-228d-4141-8dfe-4b868650ad5c\") "
Sep 09 20:36:57.031751 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.031586    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6cc7f01-228d-4141-8dfe-4b868650ad5c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6cc7f01-228d-4141-8dfe-4b868650ad5c" (UID: "f6cc7f01-228d-4141-8dfe-4b868650ad5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:36:57.031751 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.031708    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6cc7f01-228d-4141-8dfe-4b868650ad5c-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6cc7f01-228d-4141-8dfe-4b868650ad5c" (UID: "f6cc7f01-228d-4141-8dfe-4b868650ad5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:36:57.031909 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.031748    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6cc7f01-228d-4141-8dfe-4b868650ad5c-home" (OuterVolumeSpecName: "home") pod "f6cc7f01-228d-4141-8dfe-4b868650ad5c" (UID: "f6cc7f01-228d-4141-8dfe-4b868650ad5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:36:57.031909 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.031817    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6cc7f01-228d-4141-8dfe-4b868650ad5c-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:36:57.033774 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.033749    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6cc7f01-228d-4141-8dfe-4b868650ad5c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6cc7f01-228d-4141-8dfe-4b868650ad5c" (UID: "f6cc7f01-228d-4141-8dfe-4b868650ad5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:36:57.033872 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.033775    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6cc7f01-228d-4141-8dfe-4b868650ad5c-dshm" (OuterVolumeSpecName: "dshm") pod "f6cc7f01-228d-4141-8dfe-4b868650ad5c" (UID: "f6cc7f01-228d-4141-8dfe-4b868650ad5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:36:57.033872 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.033820    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6cc7f01-228d-4141-8dfe-4b868650ad5c-kube-api-access-kgprq" (OuterVolumeSpecName: "kube-api-access-kgprq") pod "f6cc7f01-228d-4141-8dfe-4b868650ad5c" (UID: "f6cc7f01-228d-4141-8dfe-4b868650ad5c"). InnerVolumeSpecName "kube-api-access-kgprq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:36:57.074858 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.074773    2581 generic.go:356] "Generic (PLEG): container finished" podID="f6cc7f01-228d-4141-8dfe-4b868650ad5c" containerID="0d9b453b414b6e874c8cc305c3b260d47728cfcc14d4a7656c2842d8204ff631" exitCode=0
Sep 09 20:36:57.074858 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.074820    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"
Sep 09 20:36:57.074858 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.074830    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf" event={"ID":"f6cc7f01-228d-4141-8dfe-4b868650ad5c","Type":"ContainerDied","Data":"0d9b453b414b6e874c8cc305c3b260d47728cfcc14d4a7656c2842d8204ff631"}
Sep 09 20:36:57.075134 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.074870    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf" event={"ID":"f6cc7f01-228d-4141-8dfe-4b868650ad5c","Type":"ContainerDied","Data":"3009e2ca9e9d3e733ad9c3b4dee98d4403c23adf2e77c9a43fe92e03637dd8d6"}
Sep 09 20:36:57.075134 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.074891    2581 scope.go:117] "RemoveContainer" containerID="0d9b453b414b6e874c8cc305c3b260d47728cfcc14d4a7656c2842d8204ff631"
Sep 09 20:36:57.085423 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.085404    2581 scope.go:117] "RemoveContainer" containerID="0d9b453b414b6e874c8cc305c3b260d47728cfcc14d4a7656c2842d8204ff631"
Sep 09 20:36:57.085756 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:36:57.085730    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9b453b414b6e874c8cc305c3b260d47728cfcc14d4a7656c2842d8204ff631\": container with ID starting with 0d9b453b414b6e874c8cc305c3b260d47728cfcc14d4a7656c2842d8204ff631 not found: ID does not exist" containerID="0d9b453b414b6e874c8cc305c3b260d47728cfcc14d4a7656c2842d8204ff631"
Sep 09 20:36:57.085906 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.085757    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9b453b414b6e874c8cc305c3b260d47728cfcc14d4a7656c2842d8204ff631"} err="failed to get container status \"0d9b453b414b6e874c8cc305c3b260d47728cfcc14d4a7656c2842d8204ff631\": rpc error: code = NotFound desc = could not find container \"0d9b453b414b6e874c8cc305c3b260d47728cfcc14d4a7656c2842d8204ff631\": container with ID starting with 0d9b453b414b6e874c8cc305c3b260d47728cfcc14d4a7656c2842d8204ff631 not found: ID does not exist"
Sep 09 20:36:57.097954 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.097929    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"]
Sep 09 20:36:57.102944 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.102923    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-sw8zf"]
Sep 09 20:36:57.133199 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.133170    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgprq\" (UniqueName: \"kubernetes.io/projected/f6cc7f01-228d-4141-8dfe-4b868650ad5c-kube-api-access-kgprq\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:36:57.133199 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.133196    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6cc7f01-228d-4141-8dfe-4b868650ad5c-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:36:57.133337 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.133210    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6cc7f01-228d-4141-8dfe-4b868650ad5c-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:36:57.133337 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.133222    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6cc7f01-228d-4141-8dfe-4b868650ad5c-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:36:57.133337 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:57.133234    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6cc7f01-228d-4141-8dfe-4b868650ad5c-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:36:58.635747 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:36:58.635718    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6cc7f01-228d-4141-8dfe-4b868650ad5c" path="/var/lib/kubelet/pods/f6cc7f01-228d-4141-8dfe-4b868650ad5c/volumes"
Sep 09 20:37:04.906406 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:37:04.906362    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb" podUID="59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:37:07.440483 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:37:07.440433    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 09 20:37:07.440783 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:37:07.440486    2581 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb" podUID="59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b" containerName="main" containerID="cri-o://6519db8d1728a50d5193f6b3673477b7da6351134d7ab75f474e0a264f1b60f0"
Sep 09 20:37:07.440783 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:37:07.440537    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb" podUID="59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b" containerName="main" containerID="cri-o://6519db8d1728a50d5193f6b3673477b7da6351134d7ab75f474e0a264f1b60f0" gracePeriod=50
Sep 09 20:37:07.593524 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:37:07.593502    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb_59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b/main/0.log"
Sep 09 20:37:07.595726 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:37:07.595712    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb"
Sep 09 20:37:07.723399 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:37:07.723316    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b-tls-certs\") pod \"59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b\" (UID: \"59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b\") "
Sep 09 20:37:07.723571 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:37:07.723434    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6rck\" (UniqueName: \"kubernetes.io/projected/59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b-kube-api-access-r6rck\") pod \"59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b\" (UID: \"59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b\") "
Sep 09 20:37:07.725445 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:37:07.725420    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b-kube-api-access-r6rck" (OuterVolumeSpecName: "kube-api-access-r6rck") pod "59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b" (UID: "59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b"). InnerVolumeSpecName "kube-api-access-r6rck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:37:07.725546 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:37:07.725442    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b" (UID: "59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:37:07.824141 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:37:07.824115    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:37:07.824141 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:37:07.824139    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6rck\" (UniqueName: \"kubernetes.io/projected/59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b-kube-api-access-r6rck\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:37:08.134452 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:37:08.134424    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb_59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b/main/0.log"
Sep 09 20:37:08.134630 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:37:08.134478    2581 generic.go:356] "Generic (PLEG): container finished" podID="59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b" containerID="6519db8d1728a50d5193f6b3673477b7da6351134d7ab75f474e0a264f1b60f0" exitCode=1
Sep 09 20:37:08.134630 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:37:08.134577    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb"
Sep 09 20:37:08.134704 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:37:08.134581    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb" event={"ID":"59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b","Type":"ContainerDied","Data":"6519db8d1728a50d5193f6b3673477b7da6351134d7ab75f474e0a264f1b60f0"}
Sep 09 20:37:08.134704 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:37:08.134698    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb" event={"ID":"59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b","Type":"ContainerDied","Data":"9e0a707c698c6bf536593203bb1c449e61d0e46e84543b549203738e419763eb"}
Sep 09 20:37:08.134774 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:37:08.134753    2581 scope.go:117] "RemoveContainer" containerID="6519db8d1728a50d5193f6b3673477b7da6351134d7ab75f474e0a264f1b60f0"
Sep 09 20:37:08.143727 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:37:08.143709    2581 scope.go:117] "RemoveContainer" containerID="6519db8d1728a50d5193f6b3673477b7da6351134d7ab75f474e0a264f1b60f0"
Sep 09 20:37:08.144008 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:37:08.143990    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6519db8d1728a50d5193f6b3673477b7da6351134d7ab75f474e0a264f1b60f0\": container with ID starting with 6519db8d1728a50d5193f6b3673477b7da6351134d7ab75f474e0a264f1b60f0 not found: ID does not exist" containerID="6519db8d1728a50d5193f6b3673477b7da6351134d7ab75f474e0a264f1b60f0"
Sep 09 20:37:08.144062 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:37:08.144015    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6519db8d1728a50d5193f6b3673477b7da6351134d7ab75f474e0a264f1b60f0"} err="failed to get container status \"6519db8d1728a50d5193f6b3673477b7da6351134d7ab75f474e0a264f1b60f0\": rpc error: code = NotFound desc = could not find container \"6519db8d1728a50d5193f6b3673477b7da6351134d7ab75f474e0a264f1b60f0\": container with ID starting with 6519db8d1728a50d5193f6b3673477b7da6351134d7ab75f474e0a264f1b60f0 not found: ID does not exist"
Sep 09 20:37:08.156270 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:37:08.156247    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb"]
Sep 09 20:37:08.160272 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:37:08.160252    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5sr4mb"]
Sep 09 20:37:08.638872 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:37:08.638834    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b" path="/var/lib/kubelet/pods/59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b/volumes"
Sep 09 20:38:01.951975 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:01.951936    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"]
Sep 09 20:38:01.952360 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:01.952281    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6cc7f01-228d-4141-8dfe-4b868650ad5c" containerName="main"
Sep 09 20:38:01.952360 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:01.952292    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6cc7f01-228d-4141-8dfe-4b868650ad5c" containerName="main"
Sep 09 20:38:01.952360 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:01.952301    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b" containerName="main"
Sep 09 20:38:01.952360 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:01.952307    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b" containerName="main"
Sep 09 20:38:01.952520 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:01.952369    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="59e8ea5d-b959-4882-8f30-7a9ee1b2ee5b" containerName="main"
Sep 09 20:38:01.952520 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:01.952377    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6cc7f01-228d-4141-8dfe-4b868650ad5c" containerName="main"
Sep 09 20:38:01.955200 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:01.955174    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:01.958772 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:01.958743    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 09 20:38:01.958772 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:01.958761    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-tmvgb\""
Sep 09 20:38:01.958772 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:01.958743    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 09 20:38:01.959076 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:01.958752    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 09 20:38:01.964217 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:01.964195    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"]
Sep 09 20:38:02.069863 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.069828    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nd588\" (UID: \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:02.069863 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.069865    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nd588\" (UID: \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:02.070096 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.069980    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nd588\" (UID: \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:02.070096 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.070016    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nd588\" (UID: \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:02.070096 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.070049    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nd588\" (UID: \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:02.070096 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.070078    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dt87\" (UniqueName: \"kubernetes.io/projected/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-kube-api-access-9dt87\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nd588\" (UID: \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:02.171357 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.171319    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nd588\" (UID: \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:02.171357 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.171357    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nd588\" (UID: \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:02.171658 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.171392    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nd588\" (UID: \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:02.171658 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.171413    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nd588\" (UID: \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:02.171658 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.171431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nd588\" (UID: \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:02.171658 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.171489    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dt87\" (UniqueName: \"kubernetes.io/projected/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-kube-api-access-9dt87\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nd588\" (UID: \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:02.171873 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.171827    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nd588\" (UID: \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:02.171873 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.171836    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nd588\" (UID: \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:02.171968 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.171895    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nd588\" (UID: \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:02.173837 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.173819    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nd588\" (UID: \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:02.173928 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.173910    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nd588\" (UID: \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:02.181977 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.181954    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dt87\" (UniqueName: \"kubernetes.io/projected/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-kube-api-access-9dt87\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-nd588\" (UID: \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:02.228049 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.228019    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"]
Sep 09 20:38:02.231639 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.231614    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"
Sep 09 20:38:02.234610 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.234585    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-s5vlm\""
Sep 09 20:38:02.242639 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.242611    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"]
Sep 09 20:38:02.267049 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.266998    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:02.373160 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.373122    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm\" (UID: \"a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"
Sep 09 20:38:02.373329 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.373165    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj67r\" (UniqueName: \"kubernetes.io/projected/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16-kube-api-access-lj67r\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm\" (UID: \"a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"
Sep 09 20:38:02.373329 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.373266    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm\" (UID: \"a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"
Sep 09 20:38:02.373329 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.373302    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm\" (UID: \"a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"
Sep 09 20:38:02.474211 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.474169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm\" (UID: \"a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"
Sep 09 20:38:02.474211 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.474211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm\" (UID: \"a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"
Sep 09 20:38:02.474512 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.474267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm\" (UID: \"a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"
Sep 09 20:38:02.474512 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.474285    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj67r\" (UniqueName: \"kubernetes.io/projected/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16-kube-api-access-lj67r\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm\" (UID: \"a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"
Sep 09 20:38:02.474684 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.474656    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm\" (UID: \"a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"
Sep 09 20:38:02.474746 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.474665    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm\" (UID: \"a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"
Sep 09 20:38:02.476914 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.476872    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm\" (UID: \"a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"
Sep 09 20:38:02.483185 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.483126    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj67r\" (UniqueName: \"kubernetes.io/projected/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16-kube-api-access-lj67r\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm\" (UID: \"a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"
Sep 09 20:38:02.541060 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.541033    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"]
Sep 09 20:38:02.542774 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:38:02.542744    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e5daa9_c45d_42c2_bf2c_88c4cad2fc8a.slice/crio-83b934bddc883a8ed8ac5c20354f3c738970a3c8d9a1a9bcdb6c3c2fd1dcac15 WatchSource:0}: Error finding container 83b934bddc883a8ed8ac5c20354f3c738970a3c8d9a1a9bcdb6c3c2fd1dcac15: Status 404 returned error can't find the container with id 83b934bddc883a8ed8ac5c20354f3c738970a3c8d9a1a9bcdb6c3c2fd1dcac15
Sep 09 20:38:02.544286 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.544261    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"
Sep 09 20:38:02.826416 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:02.826377    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"]
Sep 09 20:38:02.828039 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:38:02.828014    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda68fdc2a_4eb4_4286_ac24_1c43f3f8ba16.slice/crio-b2f1ebdd5840b68c8bada3ebf6065cb040e72ceda8d856dc24baa43e0a8a9434 WatchSource:0}: Error finding container b2f1ebdd5840b68c8bada3ebf6065cb040e72ceda8d856dc24baa43e0a8a9434: Status 404 returned error can't find the container with id b2f1ebdd5840b68c8bada3ebf6065cb040e72ceda8d856dc24baa43e0a8a9434
Sep 09 20:38:03.405050 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:03.405011    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588" event={"ID":"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a","Type":"ContainerStarted","Data":"55b4a1197b8d95fafa1092d4d685b13685541b97a879e960e4dc02f8aa7976db"}
Sep 09 20:38:03.405514 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:03.405077    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588" event={"ID":"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a","Type":"ContainerStarted","Data":"83b934bddc883a8ed8ac5c20354f3c738970a3c8d9a1a9bcdb6c3c2fd1dcac15"}
Sep 09 20:38:03.406584 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:03.406561    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm" event={"ID":"a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16","Type":"ContainerStarted","Data":"03643be74beb4f88655b58ba95652494291fb091bf41f76104cd6355e329fdd9"}
Sep 09 20:38:03.406703 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:03.406598    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm" event={"ID":"a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16","Type":"ContainerStarted","Data":"b2f1ebdd5840b68c8bada3ebf6065cb040e72ceda8d856dc24baa43e0a8a9434"}
Sep 09 20:38:03.406703 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:03.406690    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"
Sep 09 20:38:03.420899 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:03.420853    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588" podStartSLOduration=2.420839199 podStartE2EDuration="2.420839199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:38:03.420060794 +0000 UTC m=+1625.425237789" watchObservedRunningTime="2026-09-09 20:38:03.420839199 +0000 UTC m=+1625.426016237"
Sep 09 20:38:03.436679 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:03.436631    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm" podStartSLOduration=1.436616818 podStartE2EDuration="1.436616818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:38:03.435703512 +0000 UTC m=+1625.440880507" watchObservedRunningTime="2026-09-09 20:38:03.436616818 +0000 UTC m=+1625.441793813"
Sep 09 20:38:04.268045 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:04.268012    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:12.267469 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:12.267432    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:12.280100 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:12.280071    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:12.463484 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:12.463427    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:34.413256 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:34.413223    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"
Sep 09 20:38:36.020332 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.020284    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"]
Sep 09 20:38:36.022592 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.022565    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"]
Sep 09 20:38:36.030871 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:38:36.030838    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:38:36.030871 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:38:36.030871 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:38:36.030871 ip-10-0-134-84 kubenswrapper[2581]:  > containerID="55b4a1197b8d95fafa1092d4d685b13685541b97a879e960e4dc02f8aa7976db" cmd=["/bin/sleep","15"]
Sep 09 20:38:36.031167 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.030886    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588" message="" err=<
Sep 09 20:38:36.031167 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:38:36.031167 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:38:36.031167 ip-10-0-134-84 kubenswrapper[2581]:  >
Sep 09 20:38:36.031167 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:38:36.030900    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:38:36.031167 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:38:36.031167 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:38:36.031167 ip-10-0-134-84 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588" podUID="94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a" containerName="main" containerID="cri-o://55b4a1197b8d95fafa1092d4d685b13685541b97a879e960e4dc02f8aa7976db"
Sep 09 20:38:36.031167 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.030936    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588" podUID="94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a" containerName="main" containerID="cri-o://55b4a1197b8d95fafa1092d4d685b13685541b97a879e960e4dc02f8aa7976db" gracePeriod=60
Sep 09 20:38:36.375045 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.375023    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:36.468001 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.467969    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-model-cache\") pod \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\" (UID: \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\") "
Sep 09 20:38:36.468164 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.468024    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-tls-certs\") pod \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\" (UID: \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\") "
Sep 09 20:38:36.468164 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.468056    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-home\") pod \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\" (UID: \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\") "
Sep 09 20:38:36.468164 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.468074    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-dshm\") pod \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\" (UID: \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\") "
Sep 09 20:38:36.468164 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.468093    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dt87\" (UniqueName: \"kubernetes.io/projected/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-kube-api-access-9dt87\") pod \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\" (UID: \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\") "
Sep 09 20:38:36.468164 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.468151    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-tmp-dir\") pod \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\" (UID: \"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a\") "
Sep 09 20:38:36.468531 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.468237    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-model-cache" (OuterVolumeSpecName: "model-cache") pod "94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a" (UID: "94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:36.468531 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.468352    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-home" (OuterVolumeSpecName: "home") pod "94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a" (UID: "94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:36.468531 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.468492    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:38:36.468531 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.468493    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a" (UID: "94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:36.468531 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.468505    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:38:36.470394 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.470353    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-dshm" (OuterVolumeSpecName: "dshm") pod "94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a" (UID: "94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:36.470513 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.470433    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-kube-api-access-9dt87" (OuterVolumeSpecName: "kube-api-access-9dt87") pod "94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a" (UID: "94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a"). InnerVolumeSpecName "kube-api-access-9dt87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:38:36.470513 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.470448    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a" (UID: "94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:38:36.569016 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.568922    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:38:36.569016 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.568946    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:38:36.569016 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.568954    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:38:36.569016 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.568963    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dt87\" (UniqueName: \"kubernetes.io/projected/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a-kube-api-access-9dt87\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:38:36.574072 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.574046    2581 generic.go:356] "Generic (PLEG): container finished" podID="94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a" containerID="55b4a1197b8d95fafa1092d4d685b13685541b97a879e960e4dc02f8aa7976db" exitCode=0
Sep 09 20:38:36.574199 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.574103    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"
Sep 09 20:38:36.574199 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.574137    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588" event={"ID":"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a","Type":"ContainerDied","Data":"55b4a1197b8d95fafa1092d4d685b13685541b97a879e960e4dc02f8aa7976db"}
Sep 09 20:38:36.574288 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.574206    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588" event={"ID":"94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a","Type":"ContainerDied","Data":"83b934bddc883a8ed8ac5c20354f3c738970a3c8d9a1a9bcdb6c3c2fd1dcac15"}
Sep 09 20:38:36.574288 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.574238    2581 scope.go:117] "RemoveContainer" containerID="55b4a1197b8d95fafa1092d4d685b13685541b97a879e960e4dc02f8aa7976db"
Sep 09 20:38:36.583388 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.583372    2581 scope.go:117] "RemoveContainer" containerID="55b4a1197b8d95fafa1092d4d685b13685541b97a879e960e4dc02f8aa7976db"
Sep 09 20:38:36.583679 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:38:36.583663    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b4a1197b8d95fafa1092d4d685b13685541b97a879e960e4dc02f8aa7976db\": container with ID starting with 55b4a1197b8d95fafa1092d4d685b13685541b97a879e960e4dc02f8aa7976db not found: ID does not exist" containerID="55b4a1197b8d95fafa1092d4d685b13685541b97a879e960e4dc02f8aa7976db"
Sep 09 20:38:36.583752 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.583686    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b4a1197b8d95fafa1092d4d685b13685541b97a879e960e4dc02f8aa7976db"} err="failed to get container status \"55b4a1197b8d95fafa1092d4d685b13685541b97a879e960e4dc02f8aa7976db\": rpc error: code = NotFound desc = could not find container \"55b4a1197b8d95fafa1092d4d685b13685541b97a879e960e4dc02f8aa7976db\": container with ID starting with 55b4a1197b8d95fafa1092d4d685b13685541b97a879e960e4dc02f8aa7976db not found: ID does not exist"
Sep 09 20:38:36.598265 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.598243    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"]
Sep 09 20:38:36.602157 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.602139    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-nd588"]
Sep 09 20:38:36.635180 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:36.635154    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a" path="/var/lib/kubelet/pods/94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a/volumes"
Sep 09 20:38:44.413396 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:44.413357    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm" podUID="a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:38:51.046698 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:51.046652    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm" podUID="a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16" containerName="main" containerID="cri-o://03643be74beb4f88655b58ba95652494291fb091bf41f76104cd6355e329fdd9" gracePeriod=45
Sep 09 20:38:54.413598 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:38:54.413559    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm" podUID="a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:39:04.413546 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:04.413506    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm" podUID="a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:39:04.414022 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:04.413601    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"
Sep 09 20:39:14.413659 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:14.413621    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm" podUID="a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:39:21.300645 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.300618    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"
Sep 09 20:39:21.346576 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.346482    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lj67r\" (UniqueName: \"kubernetes.io/projected/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16-kube-api-access-lj67r\") pod \"a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16\" (UID: \"a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16\") "
Sep 09 20:39:21.346576 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.346549    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16-tokenizer-cache\") pod \"a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16\" (UID: \"a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16\") "
Sep 09 20:39:21.346807 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.346601    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16-cachi2-cache\") pod \"a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16\" (UID: \"a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16\") "
Sep 09 20:39:21.346807 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.346640    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16-tls-certs\") pod \"a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16\" (UID: \"a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16\") "
Sep 09 20:39:21.346912 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.346873    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16" (UID: "a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:39:21.346962 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.346921    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16" (UID: "a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:39:21.348804 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.348775    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16" (UID: "a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:39:21.349059 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.349045    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16-kube-api-access-lj67r" (OuterVolumeSpecName: "kube-api-access-lj67r") pod "a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16" (UID: "a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16"). InnerVolumeSpecName "kube-api-access-lj67r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:39:21.448156 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.448105    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj67r\" (UniqueName: \"kubernetes.io/projected/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16-kube-api-access-lj67r\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:39:21.448156 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.448151    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16-tokenizer-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:39:21.448156 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.448160    2581 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16-cachi2-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:39:21.448156 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.448170    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:39:21.797121 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.797083    2581 generic.go:356] "Generic (PLEG): container finished" podID="a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16" containerID="03643be74beb4f88655b58ba95652494291fb091bf41f76104cd6355e329fdd9" exitCode=0
Sep 09 20:39:21.797121 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.797121    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"
Sep 09 20:39:21.797327 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.797169    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm" event={"ID":"a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16","Type":"ContainerDied","Data":"03643be74beb4f88655b58ba95652494291fb091bf41f76104cd6355e329fdd9"}
Sep 09 20:39:21.797327 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.797226    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm" event={"ID":"a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16","Type":"ContainerDied","Data":"b2f1ebdd5840b68c8bada3ebf6065cb040e72ceda8d856dc24baa43e0a8a9434"}
Sep 09 20:39:21.797327 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.797250    2581 scope.go:117] "RemoveContainer" containerID="03643be74beb4f88655b58ba95652494291fb091bf41f76104cd6355e329fdd9"
Sep 09 20:39:21.806112 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.806093    2581 scope.go:117] "RemoveContainer" containerID="03643be74beb4f88655b58ba95652494291fb091bf41f76104cd6355e329fdd9"
Sep 09 20:39:21.806372 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:39:21.806350    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03643be74beb4f88655b58ba95652494291fb091bf41f76104cd6355e329fdd9\": container with ID starting with 03643be74beb4f88655b58ba95652494291fb091bf41f76104cd6355e329fdd9 not found: ID does not exist" containerID="03643be74beb4f88655b58ba95652494291fb091bf41f76104cd6355e329fdd9"
Sep 09 20:39:21.806437 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.806381    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03643be74beb4f88655b58ba95652494291fb091bf41f76104cd6355e329fdd9"} err="failed to get container status \"03643be74beb4f88655b58ba95652494291fb091bf41f76104cd6355e329fdd9\": rpc error: code = NotFound desc = could not find container \"03643be74beb4f88655b58ba95652494291fb091bf41f76104cd6355e329fdd9\": container with ID starting with 03643be74beb4f88655b58ba95652494291fb091bf41f76104cd6355e329fdd9 not found: ID does not exist"
Sep 09 20:39:21.819051 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.819032    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"]
Sep 09 20:39:21.822649 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.822624    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs8dgm"]
Sep 09 20:39:21.938950 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.938912    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"]
Sep 09 20:39:21.939295 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.939282    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16" containerName="main"
Sep 09 20:39:21.939340 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.939297    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16" containerName="main"
Sep 09 20:39:21.939340 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.939314    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a" containerName="main"
Sep 09 20:39:21.939340 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.939322    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a" containerName="main"
Sep 09 20:39:21.939440 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.939392    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="94e5daa9-c45d-42c2-bf2c-88c4cad2fc8a" containerName="main"
Sep 09 20:39:21.939440 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.939403    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16" containerName="main"
Sep 09 20:39:21.943599 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.943582    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:39:21.946056 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.946030    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Sep 09 20:39:21.946185 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.946075    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v2-kserve-self-signed-certs\""
Sep 09 20:39:21.946185 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.946153    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Sep 09 20:39:21.946509 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.946492    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-7fp44\""
Sep 09 20:39:21.956425 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:21.956399    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"]
Sep 09 20:39:22.058182 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:22.058096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aeac195-1f9c-4605-a84a-72495304eec8-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-lwl7v\" (UID: \"0aeac195-1f9c-4605-a84a-72495304eec8\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:39:22.058182 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:22.058131    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aeac195-1f9c-4605-a84a-72495304eec8-dshm\") pod \"canary-v2-kserve-7f65dbbccc-lwl7v\" (UID: \"0aeac195-1f9c-4605-a84a-72495304eec8\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:39:22.058182 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:22.058161    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aeac195-1f9c-4605-a84a-72495304eec8-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-lwl7v\" (UID: \"0aeac195-1f9c-4605-a84a-72495304eec8\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:39:22.058482 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:22.058220    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aeac195-1f9c-4605-a84a-72495304eec8-home\") pod \"canary-v2-kserve-7f65dbbccc-lwl7v\" (UID: \"0aeac195-1f9c-4605-a84a-72495304eec8\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:39:22.058482 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:22.058263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnvxj\" (UniqueName: \"kubernetes.io/projected/0aeac195-1f9c-4605-a84a-72495304eec8-kube-api-access-rnvxj\") pod \"canary-v2-kserve-7f65dbbccc-lwl7v\" (UID: \"0aeac195-1f9c-4605-a84a-72495304eec8\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:39:22.058482 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:22.058332    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aeac195-1f9c-4605-a84a-72495304eec8-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-lwl7v\" (UID: \"0aeac195-1f9c-4605-a84a-72495304eec8\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:39:22.159792 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:22.159756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aeac195-1f9c-4605-a84a-72495304eec8-home\") pod \"canary-v2-kserve-7f65dbbccc-lwl7v\" (UID: \"0aeac195-1f9c-4605-a84a-72495304eec8\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:39:22.159792 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:22.159796    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnvxj\" (UniqueName: \"kubernetes.io/projected/0aeac195-1f9c-4605-a84a-72495304eec8-kube-api-access-rnvxj\") pod \"canary-v2-kserve-7f65dbbccc-lwl7v\" (UID: \"0aeac195-1f9c-4605-a84a-72495304eec8\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:39:22.159986 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:22.159818    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aeac195-1f9c-4605-a84a-72495304eec8-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-lwl7v\" (UID: \"0aeac195-1f9c-4605-a84a-72495304eec8\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:39:22.159986 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:22.159852    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aeac195-1f9c-4605-a84a-72495304eec8-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-lwl7v\" (UID: \"0aeac195-1f9c-4605-a84a-72495304eec8\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:39:22.159986 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:22.159868    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aeac195-1f9c-4605-a84a-72495304eec8-dshm\") pod \"canary-v2-kserve-7f65dbbccc-lwl7v\" (UID: \"0aeac195-1f9c-4605-a84a-72495304eec8\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:39:22.159986 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:22.159896    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aeac195-1f9c-4605-a84a-72495304eec8-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-lwl7v\" (UID: \"0aeac195-1f9c-4605-a84a-72495304eec8\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:39:22.160235 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:22.160213    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aeac195-1f9c-4605-a84a-72495304eec8-home\") pod \"canary-v2-kserve-7f65dbbccc-lwl7v\" (UID: \"0aeac195-1f9c-4605-a84a-72495304eec8\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:39:22.160285 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:22.160269    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aeac195-1f9c-4605-a84a-72495304eec8-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-lwl7v\" (UID: \"0aeac195-1f9c-4605-a84a-72495304eec8\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:39:22.160323 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:22.160293    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aeac195-1f9c-4605-a84a-72495304eec8-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-lwl7v\" (UID: \"0aeac195-1f9c-4605-a84a-72495304eec8\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:39:22.162324 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:22.162299    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aeac195-1f9c-4605-a84a-72495304eec8-dshm\") pod \"canary-v2-kserve-7f65dbbccc-lwl7v\" (UID: \"0aeac195-1f9c-4605-a84a-72495304eec8\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:39:22.162782 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:22.162764    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aeac195-1f9c-4605-a84a-72495304eec8-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-lwl7v\" (UID: \"0aeac195-1f9c-4605-a84a-72495304eec8\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:39:22.168111 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:22.168087    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnvxj\" (UniqueName: \"kubernetes.io/projected/0aeac195-1f9c-4605-a84a-72495304eec8-kube-api-access-rnvxj\") pod \"canary-v2-kserve-7f65dbbccc-lwl7v\" (UID: \"0aeac195-1f9c-4605-a84a-72495304eec8\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:39:22.264171 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:22.264136    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:39:22.543699 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:22.543670    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"]
Sep 09 20:39:22.544926 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:39:22.544897    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aeac195_1f9c_4605_a84a_72495304eec8.slice/crio-48ac4f57f2bb6f590acd1b582da0e06a724cea9e3a9d1e87e10cf7df18004f44 WatchSource:0}: Error finding container 48ac4f57f2bb6f590acd1b582da0e06a724cea9e3a9d1e87e10cf7df18004f44: Status 404 returned error can't find the container with id 48ac4f57f2bb6f590acd1b582da0e06a724cea9e3a9d1e87e10cf7df18004f44
Sep 09 20:39:22.547313 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:22.547293    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:39:22.637915 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:22.637820    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16" path="/var/lib/kubelet/pods/a68fdc2a-4eb4-4286-ac24-1c43f3f8ba16/volumes"
Sep 09 20:39:22.807077 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:22.807040    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v" event={"ID":"0aeac195-1f9c-4605-a84a-72495304eec8","Type":"ContainerStarted","Data":"48ac4f57f2bb6f590acd1b582da0e06a724cea9e3a9d1e87e10cf7df18004f44"}
Sep 09 20:39:23.817218 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:23.817117    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v" event={"ID":"0aeac195-1f9c-4605-a84a-72495304eec8","Type":"ContainerStarted","Data":"16cb0bc4b240ad287506c0fe49d7c3bfa401f7a6f20da536f25bb6050ac222ae"}
Sep 09 20:39:23.837191 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:23.837132    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v" podStartSLOduration=1.8862614949999998 podStartE2EDuration="2.837110834s" totalImagesPullingTime="950.849339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:39:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:39:23.833786106 +0000 UTC m=+1705.838963129" watchObservedRunningTime="2026-09-09 20:39:23.837110834 +0000 UTC m=+1705.842287828"
Sep 09 20:39:24.264937 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:24.264906    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:39:32.264440 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:32.264404    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:39:32.276713 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:32.276686    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:39:32.874424 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:32.874389    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:39:51.472884 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.472844    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"]
Sep 09 20:39:51.476619 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.476594    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:39:51.480644 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.480618    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 09 20:39:51.480886 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.480665    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 09 20:39:51.480886 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.480687    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-jqxss\""
Sep 09 20:39:51.481113 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.480721    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-dsrt4\""
Sep 09 20:39:51.481113 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.480750    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 09 20:39:51.489131 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.489109    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"]
Sep 09 20:39:51.509200 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.509174    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a76a561b-e201-46fe-81cf-2e865906cac0-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j\" (UID: \"a76a561b-e201-46fe-81cf-2e865906cac0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:39:51.509332 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.509207    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a76a561b-e201-46fe-81cf-2e865906cac0-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j\" (UID: \"a76a561b-e201-46fe-81cf-2e865906cac0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:39:51.509332 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.509231    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvnr9\" (UniqueName: \"kubernetes.io/projected/a76a561b-e201-46fe-81cf-2e865906cac0-kube-api-access-dvnr9\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j\" (UID: \"a76a561b-e201-46fe-81cf-2e865906cac0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:39:51.509332 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.509295    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76a561b-e201-46fe-81cf-2e865906cac0-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j\" (UID: \"a76a561b-e201-46fe-81cf-2e865906cac0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:39:51.509332 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.509323    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a76a561b-e201-46fe-81cf-2e865906cac0-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j\" (UID: \"a76a561b-e201-46fe-81cf-2e865906cac0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:39:51.509491 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.509356    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a76a561b-e201-46fe-81cf-2e865906cac0-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j\" (UID: \"a76a561b-e201-46fe-81cf-2e865906cac0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:39:51.610549 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.610512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a76a561b-e201-46fe-81cf-2e865906cac0-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j\" (UID: \"a76a561b-e201-46fe-81cf-2e865906cac0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:39:51.610549 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.610552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a76a561b-e201-46fe-81cf-2e865906cac0-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j\" (UID: \"a76a561b-e201-46fe-81cf-2e865906cac0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:39:51.610831 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.610587    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvnr9\" (UniqueName: \"kubernetes.io/projected/a76a561b-e201-46fe-81cf-2e865906cac0-kube-api-access-dvnr9\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j\" (UID: \"a76a561b-e201-46fe-81cf-2e865906cac0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:39:51.610831 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.610617    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76a561b-e201-46fe-81cf-2e865906cac0-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j\" (UID: \"a76a561b-e201-46fe-81cf-2e865906cac0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:39:51.610831 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.610634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a76a561b-e201-46fe-81cf-2e865906cac0-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j\" (UID: \"a76a561b-e201-46fe-81cf-2e865906cac0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:39:51.610831 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.610674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a76a561b-e201-46fe-81cf-2e865906cac0-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j\" (UID: \"a76a561b-e201-46fe-81cf-2e865906cac0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:39:51.611088 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.611002    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a76a561b-e201-46fe-81cf-2e865906cac0-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j\" (UID: \"a76a561b-e201-46fe-81cf-2e865906cac0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:39:51.611153 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.611109    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76a561b-e201-46fe-81cf-2e865906cac0-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j\" (UID: \"a76a561b-e201-46fe-81cf-2e865906cac0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:39:51.611221 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.611198    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a76a561b-e201-46fe-81cf-2e865906cac0-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j\" (UID: \"a76a561b-e201-46fe-81cf-2e865906cac0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:39:51.613012 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.612990    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a76a561b-e201-46fe-81cf-2e865906cac0-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j\" (UID: \"a76a561b-e201-46fe-81cf-2e865906cac0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:39:51.613291 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.613275    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a76a561b-e201-46fe-81cf-2e865906cac0-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j\" (UID: \"a76a561b-e201-46fe-81cf-2e865906cac0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:39:51.622020 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.621991    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvnr9\" (UniqueName: \"kubernetes.io/projected/a76a561b-e201-46fe-81cf-2e865906cac0-kube-api-access-dvnr9\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j\" (UID: \"a76a561b-e201-46fe-81cf-2e865906cac0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:39:51.787288 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:51.787254    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:39:52.064075 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:52.064048    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"]
Sep 09 20:39:52.065939 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:39:52.065912    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76a561b_e201_46fe_81cf_2e865906cac0.slice/crio-ff0d31b72945758e18d259ee8b2bd15a1832b99f744e0452d03b38ca992b23f0 WatchSource:0}: Error finding container ff0d31b72945758e18d259ee8b2bd15a1832b99f744e0452d03b38ca992b23f0: Status 404 returned error can't find the container with id ff0d31b72945758e18d259ee8b2bd15a1832b99f744e0452d03b38ca992b23f0
Sep 09 20:39:52.964011 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:52.963971    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j" event={"ID":"a76a561b-e201-46fe-81cf-2e865906cac0","Type":"ContainerStarted","Data":"ff0d31b72945758e18d259ee8b2bd15a1832b99f744e0452d03b38ca992b23f0"}
Sep 09 20:39:53.970790 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:53.970709    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j" event={"ID":"a76a561b-e201-46fe-81cf-2e865906cac0","Type":"ContainerStarted","Data":"936cfc87152491e5356d37788b322304395db5d95b43ea299703e37fd312d567"}
Sep 09 20:39:53.971142 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:53.970846    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:39:54.977856 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:54.977820    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j" event={"ID":"a76a561b-e201-46fe-81cf-2e865906cac0","Type":"ContainerStarted","Data":"950f429d860f8a32e5c7aa5e66f7950f98cea8a41d0ed9fda212b50754e48b7b"}
Sep 09 20:39:54.998139 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:54.998089    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j" podStartSLOduration=2.357293357 podStartE2EDuration="3.998073164s" totalImagesPullingTime="1.640779807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:39:54.995508869 +0000 UTC m=+1737.000685868" watchObservedRunningTime="2026-09-09 20:39:54.998073164 +0000 UTC m=+1737.003250159"
Sep 09 20:39:55.787558 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:39:55.787522    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:40:01.788529 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:01.788490    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:40:01.801204 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:01.801173    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:40:02.025871 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:02.025841    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:40:03.036138 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:03.036107    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:40:31.601198 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:31.601162    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"]
Sep 09 20:40:31.611712 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:40:31.611676    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:40:31.611712 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:40:31.611712 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:40:31.611712 ip-10-0-134-84 kubenswrapper[2581]:  > containerID="950f429d860f8a32e5c7aa5e66f7950f98cea8a41d0ed9fda212b50754e48b7b" cmd=["/bin/sleep","15"]
Sep 09 20:40:31.611936 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:31.611740    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j" message="" err=<
Sep 09 20:40:31.611936 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:40:31.611936 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:40:31.611936 ip-10-0-134-84 kubenswrapper[2581]:  >
Sep 09 20:40:31.611936 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:40:31.611756    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:40:31.611936 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:40:31.611936 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:40:31.611936 ip-10-0-134-84 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j" podUID="a76a561b-e201-46fe-81cf-2e865906cac0" containerName="main" containerID="cri-o://950f429d860f8a32e5c7aa5e66f7950f98cea8a41d0ed9fda212b50754e48b7b"
Sep 09 20:40:31.611936 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:31.611810    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j" podUID="a76a561b-e201-46fe-81cf-2e865906cac0" containerName="main" containerID="cri-o://950f429d860f8a32e5c7aa5e66f7950f98cea8a41d0ed9fda212b50754e48b7b" gracePeriod=60
Sep 09 20:40:31.828087 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:31.828053    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j" podUID="a76a561b-e201-46fe-81cf-2e865906cac0" containerName="llm-d-routing-sidecar" containerID="cri-o://936cfc87152491e5356d37788b322304395db5d95b43ea299703e37fd312d567" gracePeriod=60
Sep 09 20:40:32.075639 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.075614    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:40:32.168617 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.168580    2581 generic.go:356] "Generic (PLEG): container finished" podID="a76a561b-e201-46fe-81cf-2e865906cac0" containerID="950f429d860f8a32e5c7aa5e66f7950f98cea8a41d0ed9fda212b50754e48b7b" exitCode=0
Sep 09 20:40:32.168617 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.168610    2581 generic.go:356] "Generic (PLEG): container finished" podID="a76a561b-e201-46fe-81cf-2e865906cac0" containerID="936cfc87152491e5356d37788b322304395db5d95b43ea299703e37fd312d567" exitCode=0
Sep 09 20:40:32.168862 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.168623    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j" event={"ID":"a76a561b-e201-46fe-81cf-2e865906cac0","Type":"ContainerDied","Data":"950f429d860f8a32e5c7aa5e66f7950f98cea8a41d0ed9fda212b50754e48b7b"}
Sep 09 20:40:32.168862 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.168685    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j" event={"ID":"a76a561b-e201-46fe-81cf-2e865906cac0","Type":"ContainerDied","Data":"936cfc87152491e5356d37788b322304395db5d95b43ea299703e37fd312d567"}
Sep 09 20:40:32.168862 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.168634    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"
Sep 09 20:40:32.168862 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.168712    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j" event={"ID":"a76a561b-e201-46fe-81cf-2e865906cac0","Type":"ContainerDied","Data":"ff0d31b72945758e18d259ee8b2bd15a1832b99f744e0452d03b38ca992b23f0"}
Sep 09 20:40:32.168862 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.168739    2581 scope.go:117] "RemoveContainer" containerID="950f429d860f8a32e5c7aa5e66f7950f98cea8a41d0ed9fda212b50754e48b7b"
Sep 09 20:40:32.177912 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.177892    2581 scope.go:117] "RemoveContainer" containerID="936cfc87152491e5356d37788b322304395db5d95b43ea299703e37fd312d567"
Sep 09 20:40:32.185742 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.185708    2581 scope.go:117] "RemoveContainer" containerID="950f429d860f8a32e5c7aa5e66f7950f98cea8a41d0ed9fda212b50754e48b7b"
Sep 09 20:40:32.185999 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:40:32.185978    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"950f429d860f8a32e5c7aa5e66f7950f98cea8a41d0ed9fda212b50754e48b7b\": container with ID starting with 950f429d860f8a32e5c7aa5e66f7950f98cea8a41d0ed9fda212b50754e48b7b not found: ID does not exist" containerID="950f429d860f8a32e5c7aa5e66f7950f98cea8a41d0ed9fda212b50754e48b7b"
Sep 09 20:40:32.186107 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.186005    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"950f429d860f8a32e5c7aa5e66f7950f98cea8a41d0ed9fda212b50754e48b7b"} err="failed to get container status \"950f429d860f8a32e5c7aa5e66f7950f98cea8a41d0ed9fda212b50754e48b7b\": rpc error: code = NotFound desc = could not find container \"950f429d860f8a32e5c7aa5e66f7950f98cea8a41d0ed9fda212b50754e48b7b\": container with ID starting with 950f429d860f8a32e5c7aa5e66f7950f98cea8a41d0ed9fda212b50754e48b7b not found: ID does not exist"
Sep 09 20:40:32.186107 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.186021    2581 scope.go:117] "RemoveContainer" containerID="936cfc87152491e5356d37788b322304395db5d95b43ea299703e37fd312d567"
Sep 09 20:40:32.186285 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:40:32.186268    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"936cfc87152491e5356d37788b322304395db5d95b43ea299703e37fd312d567\": container with ID starting with 936cfc87152491e5356d37788b322304395db5d95b43ea299703e37fd312d567 not found: ID does not exist" containerID="936cfc87152491e5356d37788b322304395db5d95b43ea299703e37fd312d567"
Sep 09 20:40:32.186332 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.186288    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"936cfc87152491e5356d37788b322304395db5d95b43ea299703e37fd312d567"} err="failed to get container status \"936cfc87152491e5356d37788b322304395db5d95b43ea299703e37fd312d567\": rpc error: code = NotFound desc = could not find container \"936cfc87152491e5356d37788b322304395db5d95b43ea299703e37fd312d567\": container with ID starting with 936cfc87152491e5356d37788b322304395db5d95b43ea299703e37fd312d567 not found: ID does not exist"
Sep 09 20:40:32.186332 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.186300    2581 scope.go:117] "RemoveContainer" containerID="950f429d860f8a32e5c7aa5e66f7950f98cea8a41d0ed9fda212b50754e48b7b"
Sep 09 20:40:32.186561 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.186542    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"950f429d860f8a32e5c7aa5e66f7950f98cea8a41d0ed9fda212b50754e48b7b"} err="failed to get container status \"950f429d860f8a32e5c7aa5e66f7950f98cea8a41d0ed9fda212b50754e48b7b\": rpc error: code = NotFound desc = could not find container \"950f429d860f8a32e5c7aa5e66f7950f98cea8a41d0ed9fda212b50754e48b7b\": container with ID starting with 950f429d860f8a32e5c7aa5e66f7950f98cea8a41d0ed9fda212b50754e48b7b not found: ID does not exist"
Sep 09 20:40:32.186641 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.186561    2581 scope.go:117] "RemoveContainer" containerID="936cfc87152491e5356d37788b322304395db5d95b43ea299703e37fd312d567"
Sep 09 20:40:32.186821 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.186801    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"936cfc87152491e5356d37788b322304395db5d95b43ea299703e37fd312d567"} err="failed to get container status \"936cfc87152491e5356d37788b322304395db5d95b43ea299703e37fd312d567\": rpc error: code = NotFound desc = could not find container \"936cfc87152491e5356d37788b322304395db5d95b43ea299703e37fd312d567\": container with ID starting with 936cfc87152491e5356d37788b322304395db5d95b43ea299703e37fd312d567 not found: ID does not exist"
Sep 09 20:40:32.263166 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.263132    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a76a561b-e201-46fe-81cf-2e865906cac0-home\") pod \"a76a561b-e201-46fe-81cf-2e865906cac0\" (UID: \"a76a561b-e201-46fe-81cf-2e865906cac0\") "
Sep 09 20:40:32.263339 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.263184    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76a561b-e201-46fe-81cf-2e865906cac0-tmp-dir\") pod \"a76a561b-e201-46fe-81cf-2e865906cac0\" (UID: \"a76a561b-e201-46fe-81cf-2e865906cac0\") "
Sep 09 20:40:32.263339 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.263234    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvnr9\" (UniqueName: \"kubernetes.io/projected/a76a561b-e201-46fe-81cf-2e865906cac0-kube-api-access-dvnr9\") pod \"a76a561b-e201-46fe-81cf-2e865906cac0\" (UID: \"a76a561b-e201-46fe-81cf-2e865906cac0\") "
Sep 09 20:40:32.263339 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.263274    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a76a561b-e201-46fe-81cf-2e865906cac0-tls-certs\") pod \"a76a561b-e201-46fe-81cf-2e865906cac0\" (UID: \"a76a561b-e201-46fe-81cf-2e865906cac0\") "
Sep 09 20:40:32.263339 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.263298    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a76a561b-e201-46fe-81cf-2e865906cac0-dshm\") pod \"a76a561b-e201-46fe-81cf-2e865906cac0\" (UID: \"a76a561b-e201-46fe-81cf-2e865906cac0\") "
Sep 09 20:40:32.263588 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.263349    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a76a561b-e201-46fe-81cf-2e865906cac0-model-cache\") pod \"a76a561b-e201-46fe-81cf-2e865906cac0\" (UID: \"a76a561b-e201-46fe-81cf-2e865906cac0\") "
Sep 09 20:40:32.263588 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.263497    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76a561b-e201-46fe-81cf-2e865906cac0-home" (OuterVolumeSpecName: "home") pod "a76a561b-e201-46fe-81cf-2e865906cac0" (UID: "a76a561b-e201-46fe-81cf-2e865906cac0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:32.263588 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.263551    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76a561b-e201-46fe-81cf-2e865906cac0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a76a561b-e201-46fe-81cf-2e865906cac0" (UID: "a76a561b-e201-46fe-81cf-2e865906cac0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:32.263751 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.263661    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a76a561b-e201-46fe-81cf-2e865906cac0-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:40:32.263751 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.263679    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76a561b-e201-46fe-81cf-2e865906cac0-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:40:32.263751 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.263697    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76a561b-e201-46fe-81cf-2e865906cac0-model-cache" (OuterVolumeSpecName: "model-cache") pod "a76a561b-e201-46fe-81cf-2e865906cac0" (UID: "a76a561b-e201-46fe-81cf-2e865906cac0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:32.266261 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.266105    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76a561b-e201-46fe-81cf-2e865906cac0-kube-api-access-dvnr9" (OuterVolumeSpecName: "kube-api-access-dvnr9") pod "a76a561b-e201-46fe-81cf-2e865906cac0" (UID: "a76a561b-e201-46fe-81cf-2e865906cac0"). InnerVolumeSpecName "kube-api-access-dvnr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:40:32.266261 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.266184    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76a561b-e201-46fe-81cf-2e865906cac0-dshm" (OuterVolumeSpecName: "dshm") pod "a76a561b-e201-46fe-81cf-2e865906cac0" (UID: "a76a561b-e201-46fe-81cf-2e865906cac0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:32.266507 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.266274    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a76a561b-e201-46fe-81cf-2e865906cac0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a76a561b-e201-46fe-81cf-2e865906cac0" (UID: "a76a561b-e201-46fe-81cf-2e865906cac0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:40:32.364883 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.364850    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a76a561b-e201-46fe-81cf-2e865906cac0-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:40:32.364883 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.364876    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a76a561b-e201-46fe-81cf-2e865906cac0-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:40:32.364883 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.364886    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a76a561b-e201-46fe-81cf-2e865906cac0-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:40:32.365100 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.364895    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvnr9\" (UniqueName: \"kubernetes.io/projected/a76a561b-e201-46fe-81cf-2e865906cac0-kube-api-access-dvnr9\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:40:32.491546 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.491513    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"]
Sep 09 20:40:32.495398 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.495372    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j"]
Sep 09 20:40:32.635818 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:32.635742    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a76a561b-e201-46fe-81cf-2e865906cac0" path="/var/lib/kubelet/pods/a76a561b-e201-46fe-81cf-2e865906cac0/volumes"
Sep 09 20:40:33.015963 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:33.015921    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-m524j" podUID="a76a561b-e201-46fe-81cf-2e865906cac0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8001/health\": context deadline exceeded"
Sep 09 20:40:58.565522 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:58.565494    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 20:40:58.568064 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:58.568040    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 20:40:58.568493 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:58.568453    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 20:40:58.570988 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:40:58.570972    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 20:41:06.179492 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:06.179443    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"]
Sep 09 20:41:16.465187 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.465151    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"]
Sep 09 20:41:16.465697 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.465677    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a76a561b-e201-46fe-81cf-2e865906cac0" containerName="llm-d-routing-sidecar"
Sep 09 20:41:16.465781 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.465700    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a76a561b-e201-46fe-81cf-2e865906cac0" containerName="llm-d-routing-sidecar"
Sep 09 20:41:16.465781 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.465712    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a76a561b-e201-46fe-81cf-2e865906cac0" containerName="main"
Sep 09 20:41:16.465781 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.465720    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a76a561b-e201-46fe-81cf-2e865906cac0" containerName="main"
Sep 09 20:41:16.465930 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.465802    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a76a561b-e201-46fe-81cf-2e865906cac0" containerName="llm-d-routing-sidecar"
Sep 09 20:41:16.465930 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.465816    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a76a561b-e201-46fe-81cf-2e865906cac0" containerName="main"
Sep 09 20:41:16.468979 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.468955    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:41:16.471427 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.471403    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Sep 09 20:41:16.472613 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.472592    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 09 20:41:16.472785 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.472623    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Sep 09 20:41:16.472891 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.472868    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-zcnfj\""
Sep 09 20:41:16.477809 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.477789    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"]
Sep 09 20:41:16.629353 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.629317    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-tls-certs\") pod \"test-div-v1-kserve-fb566d667-898wx\" (UID: \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:41:16.629550 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.629381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-home\") pod \"test-div-v1-kserve-fb566d667-898wx\" (UID: \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:41:16.629550 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.629448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-model-cache\") pod \"test-div-v1-kserve-fb566d667-898wx\" (UID: \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:41:16.629643 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.629549    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgv6l\" (UniqueName: \"kubernetes.io/projected/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-kube-api-access-pgv6l\") pod \"test-div-v1-kserve-fb566d667-898wx\" (UID: \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:41:16.629643 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.629577    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-dshm\") pod \"test-div-v1-kserve-fb566d667-898wx\" (UID: \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:41:16.629643 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.629599    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-898wx\" (UID: \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:41:16.730915 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.730887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-tls-certs\") pod \"test-div-v1-kserve-fb566d667-898wx\" (UID: \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:41:16.731098 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.730929    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-home\") pod \"test-div-v1-kserve-fb566d667-898wx\" (UID: \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:41:16.731098 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.730956    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-model-cache\") pod \"test-div-v1-kserve-fb566d667-898wx\" (UID: \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:41:16.731098 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.731015    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgv6l\" (UniqueName: \"kubernetes.io/projected/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-kube-api-access-pgv6l\") pod \"test-div-v1-kserve-fb566d667-898wx\" (UID: \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:41:16.731098 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.731042    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-dshm\") pod \"test-div-v1-kserve-fb566d667-898wx\" (UID: \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:41:16.731098 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.731072    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-898wx\" (UID: \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:41:16.731375 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.731346    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-home\") pod \"test-div-v1-kserve-fb566d667-898wx\" (UID: \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:41:16.731442 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.731424    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-model-cache\") pod \"test-div-v1-kserve-fb566d667-898wx\" (UID: \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:41:16.731512 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.731491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-898wx\" (UID: \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:41:16.733176 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.733155    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-dshm\") pod \"test-div-v1-kserve-fb566d667-898wx\" (UID: \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:41:16.733549 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.733528    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-tls-certs\") pod \"test-div-v1-kserve-fb566d667-898wx\" (UID: \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:41:16.740064 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.740032    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgv6l\" (UniqueName: \"kubernetes.io/projected/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-kube-api-access-pgv6l\") pod \"test-div-v1-kserve-fb566d667-898wx\" (UID: \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:41:16.781229 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:16.781196    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:41:17.063734 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:17.063701    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"]
Sep 09 20:41:17.065354 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:41:17.065329    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf0fbde_b44f_4e7d_9dc1_e75548f65eb8.slice/crio-f3f08da599d07e8b41d44be65afb307cbf2766b4ff81f95a2d861b7f6c994ba2 WatchSource:0}: Error finding container f3f08da599d07e8b41d44be65afb307cbf2766b4ff81f95a2d861b7f6c994ba2: Status 404 returned error can't find the container with id f3f08da599d07e8b41d44be65afb307cbf2766b4ff81f95a2d861b7f6c994ba2
Sep 09 20:41:17.395190 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:17.395107    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx" event={"ID":"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8","Type":"ContainerStarted","Data":"013a4dd34bd7b54166ead0e721f00de95036d517d3d4c6db260dc9017ef8154d"}
Sep 09 20:41:17.395190 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:17.395164    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx" event={"ID":"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8","Type":"ContainerStarted","Data":"f3f08da599d07e8b41d44be65afb307cbf2766b4ff81f95a2d861b7f6c994ba2"}
Sep 09 20:41:17.415162 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:17.415110    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx" podStartSLOduration=1.415090222 podStartE2EDuration="1.415090222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:41:17.414768104 +0000 UTC m=+1819.419945099" watchObservedRunningTime="2026-09-09 20:41:17.415090222 +0000 UTC m=+1819.420267217"
Sep 09 20:41:17.781789 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:17.781752    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:41:21.200382 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:21.200323    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v" podUID="0aeac195-1f9c-4605-a84a-72495304eec8" containerName="main" containerID="cri-o://16cb0bc4b240ad287506c0fe49d7c3bfa401f7a6f20da536f25bb6050ac222ae" gracePeriod=45
Sep 09 20:41:21.419001 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:21.418972    2581 generic.go:356] "Generic (PLEG): container finished" podID="0aeac195-1f9c-4605-a84a-72495304eec8" containerID="16cb0bc4b240ad287506c0fe49d7c3bfa401f7a6f20da536f25bb6050ac222ae" exitCode=0
Sep 09 20:41:21.419165 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:21.419051    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v" event={"ID":"0aeac195-1f9c-4605-a84a-72495304eec8","Type":"ContainerDied","Data":"16cb0bc4b240ad287506c0fe49d7c3bfa401f7a6f20da536f25bb6050ac222ae"}
Sep 09 20:41:21.448853 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:21.448829    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:41:21.572521 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:21.572493    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aeac195-1f9c-4605-a84a-72495304eec8-model-cache\") pod \"0aeac195-1f9c-4605-a84a-72495304eec8\" (UID: \"0aeac195-1f9c-4605-a84a-72495304eec8\") "
Sep 09 20:41:21.572686 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:21.572535    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnvxj\" (UniqueName: \"kubernetes.io/projected/0aeac195-1f9c-4605-a84a-72495304eec8-kube-api-access-rnvxj\") pod \"0aeac195-1f9c-4605-a84a-72495304eec8\" (UID: \"0aeac195-1f9c-4605-a84a-72495304eec8\") "
Sep 09 20:41:21.572686 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:21.572567    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aeac195-1f9c-4605-a84a-72495304eec8-tls-certs\") pod \"0aeac195-1f9c-4605-a84a-72495304eec8\" (UID: \"0aeac195-1f9c-4605-a84a-72495304eec8\") "
Sep 09 20:41:21.572686 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:21.572595    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aeac195-1f9c-4605-a84a-72495304eec8-dshm\") pod \"0aeac195-1f9c-4605-a84a-72495304eec8\" (UID: \"0aeac195-1f9c-4605-a84a-72495304eec8\") "
Sep 09 20:41:21.572686 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:21.572641    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aeac195-1f9c-4605-a84a-72495304eec8-tmp-dir\") pod \"0aeac195-1f9c-4605-a84a-72495304eec8\" (UID: \"0aeac195-1f9c-4605-a84a-72495304eec8\") "
Sep 09 20:41:21.572686 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:21.572675    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aeac195-1f9c-4605-a84a-72495304eec8-home\") pod \"0aeac195-1f9c-4605-a84a-72495304eec8\" (UID: \"0aeac195-1f9c-4605-a84a-72495304eec8\") "
Sep 09 20:41:21.572924 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:21.572798    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aeac195-1f9c-4605-a84a-72495304eec8-model-cache" (OuterVolumeSpecName: "model-cache") pod "0aeac195-1f9c-4605-a84a-72495304eec8" (UID: "0aeac195-1f9c-4605-a84a-72495304eec8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:41:21.572924 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:21.572877    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aeac195-1f9c-4605-a84a-72495304eec8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0aeac195-1f9c-4605-a84a-72495304eec8" (UID: "0aeac195-1f9c-4605-a84a-72495304eec8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:41:21.573028 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:21.572936    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0aeac195-1f9c-4605-a84a-72495304eec8-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:41:21.573028 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:21.572999    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aeac195-1f9c-4605-a84a-72495304eec8-home" (OuterVolumeSpecName: "home") pod "0aeac195-1f9c-4605-a84a-72495304eec8" (UID: "0aeac195-1f9c-4605-a84a-72495304eec8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:41:21.574808 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:21.574776    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aeac195-1f9c-4605-a84a-72495304eec8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0aeac195-1f9c-4605-a84a-72495304eec8" (UID: "0aeac195-1f9c-4605-a84a-72495304eec8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:41:21.574921 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:21.574845    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aeac195-1f9c-4605-a84a-72495304eec8-kube-api-access-rnvxj" (OuterVolumeSpecName: "kube-api-access-rnvxj") pod "0aeac195-1f9c-4605-a84a-72495304eec8" (UID: "0aeac195-1f9c-4605-a84a-72495304eec8"). InnerVolumeSpecName "kube-api-access-rnvxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:41:21.574921 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:21.574860    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aeac195-1f9c-4605-a84a-72495304eec8-dshm" (OuterVolumeSpecName: "dshm") pod "0aeac195-1f9c-4605-a84a-72495304eec8" (UID: "0aeac195-1f9c-4605-a84a-72495304eec8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:41:21.673883 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:21.673839    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0aeac195-1f9c-4605-a84a-72495304eec8-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:41:21.673883 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:21.673880    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0aeac195-1f9c-4605-a84a-72495304eec8-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:41:21.673883 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:21.673889    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aeac195-1f9c-4605-a84a-72495304eec8-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:41:21.673883 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:21.673897    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0aeac195-1f9c-4605-a84a-72495304eec8-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:41:21.674135 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:21.673905    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnvxj\" (UniqueName: \"kubernetes.io/projected/0aeac195-1f9c-4605-a84a-72495304eec8-kube-api-access-rnvxj\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:41:22.425661 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:22.425621    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v" event={"ID":"0aeac195-1f9c-4605-a84a-72495304eec8","Type":"ContainerDied","Data":"48ac4f57f2bb6f590acd1b582da0e06a724cea9e3a9d1e87e10cf7df18004f44"}
Sep 09 20:41:22.426095 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:22.425682    2581 scope.go:117] "RemoveContainer" containerID="16cb0bc4b240ad287506c0fe49d7c3bfa401f7a6f20da536f25bb6050ac222ae"
Sep 09 20:41:22.426095 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:22.425642    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"
Sep 09 20:41:22.449006 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:22.448980    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"]
Sep 09 20:41:22.454351 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:22.454322    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-lwl7v"]
Sep 09 20:41:22.635568 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:22.635526    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0aeac195-1f9c-4605-a84a-72495304eec8" path="/var/lib/kubelet/pods/0aeac195-1f9c-4605-a84a-72495304eec8/volumes"
Sep 09 20:41:26.781401 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:26.781362    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:41:26.794142 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:26.794112    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:41:27.463148 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:27.463114    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:41:36.643513 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.643477    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"]
Sep 09 20:41:36.643919 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.643809    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0aeac195-1f9c-4605-a84a-72495304eec8" containerName="main"
Sep 09 20:41:36.643919 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.643820    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0aeac195-1f9c-4605-a84a-72495304eec8" containerName="main"
Sep 09 20:41:36.643919 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.643910    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0aeac195-1f9c-4605-a84a-72495304eec8" containerName="main"
Sep 09 20:41:36.646832 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.646800    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:41:36.649190 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.649167    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 09 20:41:36.659687 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.659662    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"]
Sep 09 20:41:36.703481 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.703432    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34166acf-fd9c-474e-88cc-1fd02e73696c-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-8jssg\" (UID: \"34166acf-fd9c-474e-88cc-1fd02e73696c\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:41:36.703627 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.703510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34166acf-fd9c-474e-88cc-1fd02e73696c-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-8jssg\" (UID: \"34166acf-fd9c-474e-88cc-1fd02e73696c\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:41:36.703627 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.703532    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkc2z\" (UniqueName: \"kubernetes.io/projected/34166acf-fd9c-474e-88cc-1fd02e73696c-kube-api-access-pkc2z\") pod \"test-div-v2-kserve-5f6fdb6b77-8jssg\" (UID: \"34166acf-fd9c-474e-88cc-1fd02e73696c\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:41:36.703627 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.703590    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34166acf-fd9c-474e-88cc-1fd02e73696c-home\") pod \"test-div-v2-kserve-5f6fdb6b77-8jssg\" (UID: \"34166acf-fd9c-474e-88cc-1fd02e73696c\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:41:36.703627 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.703621    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34166acf-fd9c-474e-88cc-1fd02e73696c-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-8jssg\" (UID: \"34166acf-fd9c-474e-88cc-1fd02e73696c\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:41:36.703786 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.703646    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34166acf-fd9c-474e-88cc-1fd02e73696c-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-8jssg\" (UID: \"34166acf-fd9c-474e-88cc-1fd02e73696c\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:41:36.804784 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.804752    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34166acf-fd9c-474e-88cc-1fd02e73696c-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-8jssg\" (UID: \"34166acf-fd9c-474e-88cc-1fd02e73696c\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:41:36.804977 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.804799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34166acf-fd9c-474e-88cc-1fd02e73696c-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-8jssg\" (UID: \"34166acf-fd9c-474e-88cc-1fd02e73696c\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:41:36.804977 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.804826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkc2z\" (UniqueName: \"kubernetes.io/projected/34166acf-fd9c-474e-88cc-1fd02e73696c-kube-api-access-pkc2z\") pod \"test-div-v2-kserve-5f6fdb6b77-8jssg\" (UID: \"34166acf-fd9c-474e-88cc-1fd02e73696c\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:41:36.804977 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.804882    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34166acf-fd9c-474e-88cc-1fd02e73696c-home\") pod \"test-div-v2-kserve-5f6fdb6b77-8jssg\" (UID: \"34166acf-fd9c-474e-88cc-1fd02e73696c\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:41:36.804977 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.804923    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34166acf-fd9c-474e-88cc-1fd02e73696c-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-8jssg\" (UID: \"34166acf-fd9c-474e-88cc-1fd02e73696c\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:41:36.804977 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.804952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34166acf-fd9c-474e-88cc-1fd02e73696c-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-8jssg\" (UID: \"34166acf-fd9c-474e-88cc-1fd02e73696c\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:41:36.805257 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.805231    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34166acf-fd9c-474e-88cc-1fd02e73696c-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-8jssg\" (UID: \"34166acf-fd9c-474e-88cc-1fd02e73696c\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:41:36.805351 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.805325    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34166acf-fd9c-474e-88cc-1fd02e73696c-home\") pod \"test-div-v2-kserve-5f6fdb6b77-8jssg\" (UID: \"34166acf-fd9c-474e-88cc-1fd02e73696c\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:41:36.805425 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.805335    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34166acf-fd9c-474e-88cc-1fd02e73696c-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-8jssg\" (UID: \"34166acf-fd9c-474e-88cc-1fd02e73696c\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:41:36.807099 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.807077    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34166acf-fd9c-474e-88cc-1fd02e73696c-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-8jssg\" (UID: \"34166acf-fd9c-474e-88cc-1fd02e73696c\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:41:36.807405 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.807390    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34166acf-fd9c-474e-88cc-1fd02e73696c-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-8jssg\" (UID: \"34166acf-fd9c-474e-88cc-1fd02e73696c\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:41:36.812944 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.812922    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkc2z\" (UniqueName: \"kubernetes.io/projected/34166acf-fd9c-474e-88cc-1fd02e73696c-kube-api-access-pkc2z\") pod \"test-div-v2-kserve-5f6fdb6b77-8jssg\" (UID: \"34166acf-fd9c-474e-88cc-1fd02e73696c\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:41:36.959859 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:36.959766    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:41:37.235401 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:37.235353    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"]
Sep 09 20:41:37.237852 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:41:37.237824    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34166acf_fd9c_474e_88cc_1fd02e73696c.slice/crio-9e86c20a1f62afa0540349e3cd486494e18cb9b5ec1d1cd2b395a78117b8c485 WatchSource:0}: Error finding container 9e86c20a1f62afa0540349e3cd486494e18cb9b5ec1d1cd2b395a78117b8c485: Status 404 returned error can't find the container with id 9e86c20a1f62afa0540349e3cd486494e18cb9b5ec1d1cd2b395a78117b8c485
Sep 09 20:41:37.506996 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:37.506911    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg" event={"ID":"34166acf-fd9c-474e-88cc-1fd02e73696c","Type":"ContainerStarted","Data":"e5693d880a330e6a5602450806cc8ffc5995043ccf4409b736b77405e8a4e9ff"}
Sep 09 20:41:37.506996 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:37.506968    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg" event={"ID":"34166acf-fd9c-474e-88cc-1fd02e73696c","Type":"ContainerStarted","Data":"9e86c20a1f62afa0540349e3cd486494e18cb9b5ec1d1cd2b395a78117b8c485"}
Sep 09 20:41:37.524979 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:37.524928    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg" podStartSLOduration=1.524912666 podStartE2EDuration="1.524912666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:41:37.523689532 +0000 UTC m=+1839.528866524" watchObservedRunningTime="2026-09-09 20:41:37.524912666 +0000 UTC m=+1839.530089662"
Sep 09 20:41:37.960865 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:37.960777    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:41:46.960577 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:46.960538    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:41:46.972944 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:46.972921    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:41:47.568375 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:47.568344    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:41:50.467588 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:50.467553    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt"]
Sep 09 20:41:50.471404 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:50.471379    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt"
Sep 09 20:41:50.475028 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:50.474989    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 09 20:41:50.475028 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:50.475018    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 09 20:41:50.475232 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:50.475043    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 09 20:41:50.475232 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:50.475091    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-hgrhh\""
Sep 09 20:41:50.475326 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:50.475306    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-spczr\""
Sep 09 20:41:50.479220 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:50.479200    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt"]
Sep 09 20:41:50.531655 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:50.531620    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8qx5\" (UniqueName: \"kubernetes.io/projected/b62c9031-c0bd-4d76-a455-00a322453fe7-kube-api-access-c8qx5\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched47hlt\" (UID: \"b62c9031-c0bd-4d76-a455-00a322453fe7\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt"
Sep 09 20:41:50.531817 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:50.531720    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b62c9031-c0bd-4d76-a455-00a322453fe7-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched47hlt\" (UID: \"b62c9031-c0bd-4d76-a455-00a322453fe7\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt"
Sep 09 20:41:50.632314 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:50.632281    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8qx5\" (UniqueName: \"kubernetes.io/projected/b62c9031-c0bd-4d76-a455-00a322453fe7-kube-api-access-c8qx5\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched47hlt\" (UID: \"b62c9031-c0bd-4d76-a455-00a322453fe7\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt"
Sep 09 20:41:50.632529 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:50.632351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b62c9031-c0bd-4d76-a455-00a322453fe7-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched47hlt\" (UID: \"b62c9031-c0bd-4d76-a455-00a322453fe7\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt"
Sep 09 20:41:50.634902 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:50.634870    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b62c9031-c0bd-4d76-a455-00a322453fe7-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched47hlt\" (UID: \"b62c9031-c0bd-4d76-a455-00a322453fe7\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt"
Sep 09 20:41:50.642860 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:50.642825    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8qx5\" (UniqueName: \"kubernetes.io/projected/b62c9031-c0bd-4d76-a455-00a322453fe7-kube-api-access-c8qx5\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched47hlt\" (UID: \"b62c9031-c0bd-4d76-a455-00a322453fe7\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt"
Sep 09 20:41:50.785901 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:50.785868    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt"
Sep 09 20:41:51.065036 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:51.065010    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt"]
Sep 09 20:41:51.066980 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:41:51.066952    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb62c9031_c0bd_4d76_a455_00a322453fe7.slice/crio-b474938eec404d90af10ad9001a1b195a9f70f7e46f44a5e1d0a3850bc8a95b1 WatchSource:0}: Error finding container b474938eec404d90af10ad9001a1b195a9f70f7e46f44a5e1d0a3850bc8a95b1: Status 404 returned error can't find the container with id b474938eec404d90af10ad9001a1b195a9f70f7e46f44a5e1d0a3850bc8a95b1
Sep 09 20:41:51.581903 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:51.581861    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt" event={"ID":"b62c9031-c0bd-4d76-a455-00a322453fe7","Type":"ContainerStarted","Data":"2be349dfb0bb3f28ebb87d936eb15174a4c19ba7582125dd3587bf96d787b874"}
Sep 09 20:41:51.582411 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:51.581918    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt" event={"ID":"b62c9031-c0bd-4d76-a455-00a322453fe7","Type":"ContainerStarted","Data":"b474938eec404d90af10ad9001a1b195a9f70f7e46f44a5e1d0a3850bc8a95b1"}
Sep 09 20:41:51.582411 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:51.582026    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt"
Sep 09 20:41:51.602757 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:41:51.602688    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt" podStartSLOduration=1.6026674079999998 podStartE2EDuration="1.602667408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:41:51.598785984 +0000 UTC m=+1853.603962983" watchObservedRunningTime="2026-09-09 20:41:51.602667408 +0000 UTC m=+1853.607844404"
Sep 09 20:42:06.979197 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:06.979156    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"]
Sep 09 20:42:06.988344 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:06.988304    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"]
Sep 09 20:42:22.001409 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.001320    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx" podUID="9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8" containerName="main" containerID="cri-o://013a4dd34bd7b54166ead0e721f00de95036d517d3d4c6db260dc9017ef8154d" gracePeriod=45
Sep 09 20:42:22.014619 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.014490    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg" podUID="34166acf-fd9c-474e-88cc-1fd02e73696c" containerName="main" containerID="cri-o://e5693d880a330e6a5602450806cc8ffc5995043ccf4409b736b77405e8a4e9ff" gracePeriod=45
Sep 09 20:42:22.279171 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.279143    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:42:22.281230 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.281212    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:42:22.317489 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.317430    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34166acf-fd9c-474e-88cc-1fd02e73696c-model-cache\") pod \"34166acf-fd9c-474e-88cc-1fd02e73696c\" (UID: \"34166acf-fd9c-474e-88cc-1fd02e73696c\") "
Sep 09 20:42:22.317663 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.317509    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-home\") pod \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\" (UID: \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\") "
Sep 09 20:42:22.317663 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.317549    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-tmp-dir\") pod \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\" (UID: \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\") "
Sep 09 20:42:22.317663 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.317579    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34166acf-fd9c-474e-88cc-1fd02e73696c-tmp-dir\") pod \"34166acf-fd9c-474e-88cc-1fd02e73696c\" (UID: \"34166acf-fd9c-474e-88cc-1fd02e73696c\") "
Sep 09 20:42:22.317663 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.317614    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-dshm\") pod \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\" (UID: \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\") "
Sep 09 20:42:22.317663 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.317662    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34166acf-fd9c-474e-88cc-1fd02e73696c-home\") pod \"34166acf-fd9c-474e-88cc-1fd02e73696c\" (UID: \"34166acf-fd9c-474e-88cc-1fd02e73696c\") "
Sep 09 20:42:22.318021 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.317696    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34166acf-fd9c-474e-88cc-1fd02e73696c-model-cache" (OuterVolumeSpecName: "model-cache") pod "34166acf-fd9c-474e-88cc-1fd02e73696c" (UID: "34166acf-fd9c-474e-88cc-1fd02e73696c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:42:22.318021 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.317792    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8" (UID: "9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:42:22.318021 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.317814    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-home" (OuterVolumeSpecName: "home") pod "9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8" (UID: "9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:42:22.318021 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.317864    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34166acf-fd9c-474e-88cc-1fd02e73696c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "34166acf-fd9c-474e-88cc-1fd02e73696c" (UID: "34166acf-fd9c-474e-88cc-1fd02e73696c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:42:22.318021 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.318009    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34166acf-fd9c-474e-88cc-1fd02e73696c-home" (OuterVolumeSpecName: "home") pod "34166acf-fd9c-474e-88cc-1fd02e73696c" (UID: "34166acf-fd9c-474e-88cc-1fd02e73696c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:42:22.318249 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.318090    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34166acf-fd9c-474e-88cc-1fd02e73696c-tls-certs\") pod \"34166acf-fd9c-474e-88cc-1fd02e73696c\" (UID: \"34166acf-fd9c-474e-88cc-1fd02e73696c\") "
Sep 09 20:42:22.318249 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.318135    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34166acf-fd9c-474e-88cc-1fd02e73696c-dshm\") pod \"34166acf-fd9c-474e-88cc-1fd02e73696c\" (UID: \"34166acf-fd9c-474e-88cc-1fd02e73696c\") "
Sep 09 20:42:22.318249 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.318166    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-tls-certs\") pod \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\" (UID: \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\") "
Sep 09 20:42:22.318249 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.318203    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgv6l\" (UniqueName: \"kubernetes.io/projected/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-kube-api-access-pgv6l\") pod \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\" (UID: \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\") "
Sep 09 20:42:22.318249 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.318236    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkc2z\" (UniqueName: \"kubernetes.io/projected/34166acf-fd9c-474e-88cc-1fd02e73696c-kube-api-access-pkc2z\") pod \"34166acf-fd9c-474e-88cc-1fd02e73696c\" (UID: \"34166acf-fd9c-474e-88cc-1fd02e73696c\") "
Sep 09 20:42:22.318494 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.318262    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-model-cache\") pod \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\" (UID: \"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8\") "
Sep 09 20:42:22.321998 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.319264    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-model-cache" (OuterVolumeSpecName: "model-cache") pod "9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8" (UID: "9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:42:22.321998 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.319501    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:42:22.321998 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.319523    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34166acf-fd9c-474e-88cc-1fd02e73696c-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:42:22.321998 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.319547    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:42:22.321998 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.319562    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:42:22.321998 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.319574    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34166acf-fd9c-474e-88cc-1fd02e73696c-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:42:22.321998 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.319587    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34166acf-fd9c-474e-88cc-1fd02e73696c-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:42:22.322471 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.322021    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-dshm" (OuterVolumeSpecName: "dshm") pod "9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8" (UID: "9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:42:22.322857 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.322830    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34166acf-fd9c-474e-88cc-1fd02e73696c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34166acf-fd9c-474e-88cc-1fd02e73696c" (UID: "34166acf-fd9c-474e-88cc-1fd02e73696c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:42:22.323082 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.323041    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-kube-api-access-pgv6l" (OuterVolumeSpecName: "kube-api-access-pgv6l") pod "9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8" (UID: "9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8"). InnerVolumeSpecName "kube-api-access-pgv6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:42:22.328591 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.328561    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8" (UID: "9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:42:22.329094 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.329073    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34166acf-fd9c-474e-88cc-1fd02e73696c-dshm" (OuterVolumeSpecName: "dshm") pod "34166acf-fd9c-474e-88cc-1fd02e73696c" (UID: "34166acf-fd9c-474e-88cc-1fd02e73696c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:42:22.329495 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.329453    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34166acf-fd9c-474e-88cc-1fd02e73696c-kube-api-access-pkc2z" (OuterVolumeSpecName: "kube-api-access-pkc2z") pod "34166acf-fd9c-474e-88cc-1fd02e73696c" (UID: "34166acf-fd9c-474e-88cc-1fd02e73696c"). InnerVolumeSpecName "kube-api-access-pkc2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:42:22.420199 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.420161    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34166acf-fd9c-474e-88cc-1fd02e73696c-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:42:22.420337 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.420213    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:42:22.420337 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.420226    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgv6l\" (UniqueName: \"kubernetes.io/projected/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-kube-api-access-pgv6l\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:42:22.420337 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.420236    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkc2z\" (UniqueName: \"kubernetes.io/projected/34166acf-fd9c-474e-88cc-1fd02e73696c-kube-api-access-pkc2z\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:42:22.420337 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.420247    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:42:22.420337 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.420258    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34166acf-fd9c-474e-88cc-1fd02e73696c-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:42:22.589606 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.589519    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt"
Sep 09 20:42:22.734825 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.734795    2581 generic.go:356] "Generic (PLEG): container finished" podID="9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8" containerID="013a4dd34bd7b54166ead0e721f00de95036d517d3d4c6db260dc9017ef8154d" exitCode=0
Sep 09 20:42:22.734994 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.734855    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"
Sep 09 20:42:22.734994 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.734888    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx" event={"ID":"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8","Type":"ContainerDied","Data":"013a4dd34bd7b54166ead0e721f00de95036d517d3d4c6db260dc9017ef8154d"}
Sep 09 20:42:22.734994 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.734943    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx" event={"ID":"9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8","Type":"ContainerDied","Data":"f3f08da599d07e8b41d44be65afb307cbf2766b4ff81f95a2d861b7f6c994ba2"}
Sep 09 20:42:22.734994 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.734975    2581 scope.go:117] "RemoveContainer" containerID="013a4dd34bd7b54166ead0e721f00de95036d517d3d4c6db260dc9017ef8154d"
Sep 09 20:42:22.736483 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.736443    2581 generic.go:356] "Generic (PLEG): container finished" podID="34166acf-fd9c-474e-88cc-1fd02e73696c" containerID="e5693d880a330e6a5602450806cc8ffc5995043ccf4409b736b77405e8a4e9ff" exitCode=0
Sep 09 20:42:22.736607 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.736496    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg" event={"ID":"34166acf-fd9c-474e-88cc-1fd02e73696c","Type":"ContainerDied","Data":"e5693d880a330e6a5602450806cc8ffc5995043ccf4409b736b77405e8a4e9ff"}
Sep 09 20:42:22.736607 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.736522    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"
Sep 09 20:42:22.736607 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.736536    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg" event={"ID":"34166acf-fd9c-474e-88cc-1fd02e73696c","Type":"ContainerDied","Data":"9e86c20a1f62afa0540349e3cd486494e18cb9b5ec1d1cd2b395a78117b8c485"}
Sep 09 20:42:22.743685 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.743667    2581 scope.go:117] "RemoveContainer" containerID="013a4dd34bd7b54166ead0e721f00de95036d517d3d4c6db260dc9017ef8154d"
Sep 09 20:42:22.743909 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:42:22.743893    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"013a4dd34bd7b54166ead0e721f00de95036d517d3d4c6db260dc9017ef8154d\": container with ID starting with 013a4dd34bd7b54166ead0e721f00de95036d517d3d4c6db260dc9017ef8154d not found: ID does not exist" containerID="013a4dd34bd7b54166ead0e721f00de95036d517d3d4c6db260dc9017ef8154d"
Sep 09 20:42:22.743960 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.743921    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013a4dd34bd7b54166ead0e721f00de95036d517d3d4c6db260dc9017ef8154d"} err="failed to get container status \"013a4dd34bd7b54166ead0e721f00de95036d517d3d4c6db260dc9017ef8154d\": rpc error: code = NotFound desc = could not find container \"013a4dd34bd7b54166ead0e721f00de95036d517d3d4c6db260dc9017ef8154d\": container with ID starting with 013a4dd34bd7b54166ead0e721f00de95036d517d3d4c6db260dc9017ef8154d not found: ID does not exist"
Sep 09 20:42:22.743960 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.743940    2581 scope.go:117] "RemoveContainer" containerID="e5693d880a330e6a5602450806cc8ffc5995043ccf4409b736b77405e8a4e9ff"
Sep 09 20:42:22.752047 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.752031    2581 scope.go:117] "RemoveContainer" containerID="e5693d880a330e6a5602450806cc8ffc5995043ccf4409b736b77405e8a4e9ff"
Sep 09 20:42:22.752316 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:42:22.752296    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5693d880a330e6a5602450806cc8ffc5995043ccf4409b736b77405e8a4e9ff\": container with ID starting with e5693d880a330e6a5602450806cc8ffc5995043ccf4409b736b77405e8a4e9ff not found: ID does not exist" containerID="e5693d880a330e6a5602450806cc8ffc5995043ccf4409b736b77405e8a4e9ff"
Sep 09 20:42:22.752400 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.752322    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5693d880a330e6a5602450806cc8ffc5995043ccf4409b736b77405e8a4e9ff"} err="failed to get container status \"e5693d880a330e6a5602450806cc8ffc5995043ccf4409b736b77405e8a4e9ff\": rpc error: code = NotFound desc = could not find container \"e5693d880a330e6a5602450806cc8ffc5995043ccf4409b736b77405e8a4e9ff\": container with ID starting with e5693d880a330e6a5602450806cc8ffc5995043ccf4409b736b77405e8a4e9ff not found: ID does not exist"
Sep 09 20:42:22.753273 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.753255    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"]
Sep 09 20:42:22.757270 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.757245    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-898wx"]
Sep 09 20:42:22.772419 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.772395    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"]
Sep 09 20:42:22.776739 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:22.776718    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8jssg"]
Sep 09 20:42:24.634909 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:24.634876    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34166acf-fd9c-474e-88cc-1fd02e73696c" path="/var/lib/kubelet/pods/34166acf-fd9c-474e-88cc-1fd02e73696c/volumes"
Sep 09 20:42:24.635480 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:24.635442    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8" path="/var/lib/kubelet/pods/9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8/volumes"
Sep 09 20:42:47.448616 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.448580    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"]
Sep 09 20:42:47.449203 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.448913    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34166acf-fd9c-474e-88cc-1fd02e73696c" containerName="main"
Sep 09 20:42:47.449203 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.448925    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="34166acf-fd9c-474e-88cc-1fd02e73696c" containerName="main"
Sep 09 20:42:47.449203 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.448944    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8" containerName="main"
Sep 09 20:42:47.449203 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.448950    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8" containerName="main"
Sep 09 20:42:47.449203 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.449027    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="34166acf-fd9c-474e-88cc-1fd02e73696c" containerName="main"
Sep 09 20:42:47.449203 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.449037    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cf0fbde-b44f-4e7d-9dc1-e75548f65eb8" containerName="main"
Sep 09 20:42:47.451945 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.451924    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:42:47.454358 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.454339    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Sep 09 20:42:47.455338 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.455316    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Sep 09 20:42:47.455435 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.455414    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 09 20:42:47.455509 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.455416    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-fwvl4\""
Sep 09 20:42:47.463969 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.463942    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"]
Sep 09 20:42:47.528584 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.528553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae7dd23-d3be-43b4-942b-eaab7ece883f-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-4djfw\" (UID: \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:42:47.528745 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.528601    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae7dd23-d3be-43b4-942b-eaab7ece883f-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-4djfw\" (UID: \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:42:47.528745 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.528620    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae7dd23-d3be-43b4-942b-eaab7ece883f-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-4djfw\" (UID: \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:42:47.528745 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.528688    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m62vm\" (UniqueName: \"kubernetes.io/projected/6ae7dd23-d3be-43b4-942b-eaab7ece883f-kube-api-access-m62vm\") pod \"test-div-v2-kserve-5f6fdb6b77-4djfw\" (UID: \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:42:47.528855 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.528749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae7dd23-d3be-43b4-942b-eaab7ece883f-home\") pod \"test-div-v2-kserve-5f6fdb6b77-4djfw\" (UID: \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:42:47.528855 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.528764    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae7dd23-d3be-43b4-942b-eaab7ece883f-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-4djfw\" (UID: \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:42:47.629586 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.629554    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae7dd23-d3be-43b4-942b-eaab7ece883f-home\") pod \"test-div-v2-kserve-5f6fdb6b77-4djfw\" (UID: \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:42:47.629586 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.629586    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae7dd23-d3be-43b4-942b-eaab7ece883f-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-4djfw\" (UID: \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:42:47.629803 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.629627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae7dd23-d3be-43b4-942b-eaab7ece883f-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-4djfw\" (UID: \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:42:47.629803 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.629668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae7dd23-d3be-43b4-942b-eaab7ece883f-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-4djfw\" (UID: \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:42:47.629803 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.629685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae7dd23-d3be-43b4-942b-eaab7ece883f-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-4djfw\" (UID: \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:42:47.629803 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.629727    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m62vm\" (UniqueName: \"kubernetes.io/projected/6ae7dd23-d3be-43b4-942b-eaab7ece883f-kube-api-access-m62vm\") pod \"test-div-v2-kserve-5f6fdb6b77-4djfw\" (UID: \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:42:47.630094 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.630071    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae7dd23-d3be-43b4-942b-eaab7ece883f-home\") pod \"test-div-v2-kserve-5f6fdb6b77-4djfw\" (UID: \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:42:47.630171 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.630153    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae7dd23-d3be-43b4-942b-eaab7ece883f-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-4djfw\" (UID: \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:42:47.630226 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.630203    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae7dd23-d3be-43b4-942b-eaab7ece883f-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-4djfw\" (UID: \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:42:47.631979 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.631956    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae7dd23-d3be-43b4-942b-eaab7ece883f-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-4djfw\" (UID: \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:42:47.632132 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.632114    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae7dd23-d3be-43b4-942b-eaab7ece883f-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-4djfw\" (UID: \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:42:47.637808 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.637783    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m62vm\" (UniqueName: \"kubernetes.io/projected/6ae7dd23-d3be-43b4-942b-eaab7ece883f-kube-api-access-m62vm\") pod \"test-div-v2-kserve-5f6fdb6b77-4djfw\" (UID: \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:42:47.764032 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:47.763999    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:42:48.038835 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:48.038808    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"]
Sep 09 20:42:48.041586 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:42:48.041557    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae7dd23_d3be_43b4_942b_eaab7ece883f.slice/crio-fcbfb7eaad40627600bbbefaa627f0076a0a4d0964da6de65b2c749f7525b844 WatchSource:0}: Error finding container fcbfb7eaad40627600bbbefaa627f0076a0a4d0964da6de65b2c749f7525b844: Status 404 returned error can't find the container with id fcbfb7eaad40627600bbbefaa627f0076a0a4d0964da6de65b2c749f7525b844
Sep 09 20:42:48.875453 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:48.875414    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw" event={"ID":"6ae7dd23-d3be-43b4-942b-eaab7ece883f","Type":"ContainerStarted","Data":"ac9695401c89a4dc4d27aecf96ef59f86b4a4bf0ab7bb9ce003965cff186a15e"}
Sep 09 20:42:48.875851 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:48.875498    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw" event={"ID":"6ae7dd23-d3be-43b4-942b-eaab7ece883f","Type":"ContainerStarted","Data":"fcbfb7eaad40627600bbbefaa627f0076a0a4d0964da6de65b2c749f7525b844"}
Sep 09 20:42:48.893320 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:48.893264    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw" podStartSLOduration=1.893242818 podStartE2EDuration="1.893242818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:42:48.892631296 +0000 UTC m=+1910.897808290" watchObservedRunningTime="2026-09-09 20:42:48.893242818 +0000 UTC m=+1910.898419813"
Sep 09 20:42:49.765179 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:49.765139    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:42:57.764723 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:57.764684    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:42:57.777054 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:57.777025    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:42:57.932507 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:42:57.932454    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:43:30.407474 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:30.407428    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt"]
Sep 09 20:43:32.588924 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:32.588882    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt" podUID="b62c9031-c0bd-4d76-a455-00a322453fe7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:43:37.146055 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:37.146021    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"]
Sep 09 20:43:42.589071 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:42.589031    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt" podUID="b62c9031-c0bd-4d76-a455-00a322453fe7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:43:45.427532 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:45.427493    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt" podUID="b62c9031-c0bd-4d76-a455-00a322453fe7" containerName="main" containerID="cri-o://2be349dfb0bb3f28ebb87d936eb15174a4c19ba7582125dd3587bf96d787b874" gracePeriod=45
Sep 09 20:43:52.166641 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:52.166557    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw" podUID="6ae7dd23-d3be-43b4-942b-eaab7ece883f" containerName="main" containerID="cri-o://ac9695401c89a4dc4d27aecf96ef59f86b4a4bf0ab7bb9ce003965cff186a15e" gracePeriod=45
Sep 09 20:43:52.411568 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:52.411544    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:43:52.469775 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:52.469705    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae7dd23-d3be-43b4-942b-eaab7ece883f-dshm\") pod \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\" (UID: \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\") "
Sep 09 20:43:52.469775 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:52.469774    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae7dd23-d3be-43b4-942b-eaab7ece883f-model-cache\") pod \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\" (UID: \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\") "
Sep 09 20:43:52.469936 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:52.469791    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae7dd23-d3be-43b4-942b-eaab7ece883f-home\") pod \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\" (UID: \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\") "
Sep 09 20:43:52.469936 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:52.469816    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m62vm\" (UniqueName: \"kubernetes.io/projected/6ae7dd23-d3be-43b4-942b-eaab7ece883f-kube-api-access-m62vm\") pod \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\" (UID: \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\") "
Sep 09 20:43:52.469936 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:52.469865    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae7dd23-d3be-43b4-942b-eaab7ece883f-tls-certs\") pod \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\" (UID: \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\") "
Sep 09 20:43:52.469936 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:52.469914    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae7dd23-d3be-43b4-942b-eaab7ece883f-tmp-dir\") pod \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\" (UID: \"6ae7dd23-d3be-43b4-942b-eaab7ece883f\") "
Sep 09 20:43:52.470147 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:52.469982    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae7dd23-d3be-43b4-942b-eaab7ece883f-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ae7dd23-d3be-43b4-942b-eaab7ece883f" (UID: "6ae7dd23-d3be-43b4-942b-eaab7ece883f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:43:52.470147 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:52.470007    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae7dd23-d3be-43b4-942b-eaab7ece883f-home" (OuterVolumeSpecName: "home") pod "6ae7dd23-d3be-43b4-942b-eaab7ece883f" (UID: "6ae7dd23-d3be-43b4-942b-eaab7ece883f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:43:52.470261 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:52.470182    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae7dd23-d3be-43b4-942b-eaab7ece883f-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:43:52.470261 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:52.470195    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae7dd23-d3be-43b4-942b-eaab7ece883f-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:43:52.470261 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:52.470206    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae7dd23-d3be-43b4-942b-eaab7ece883f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ae7dd23-d3be-43b4-942b-eaab7ece883f" (UID: "6ae7dd23-d3be-43b4-942b-eaab7ece883f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:43:52.471917 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:52.471894    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae7dd23-d3be-43b4-942b-eaab7ece883f-dshm" (OuterVolumeSpecName: "dshm") pod "6ae7dd23-d3be-43b4-942b-eaab7ece883f" (UID: "6ae7dd23-d3be-43b4-942b-eaab7ece883f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:43:52.472035 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:52.472002    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ae7dd23-d3be-43b4-942b-eaab7ece883f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ae7dd23-d3be-43b4-942b-eaab7ece883f" (UID: "6ae7dd23-d3be-43b4-942b-eaab7ece883f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:43:52.472089 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:52.472044    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae7dd23-d3be-43b4-942b-eaab7ece883f-kube-api-access-m62vm" (OuterVolumeSpecName: "kube-api-access-m62vm") pod "6ae7dd23-d3be-43b4-942b-eaab7ece883f" (UID: "6ae7dd23-d3be-43b4-942b-eaab7ece883f"). InnerVolumeSpecName "kube-api-access-m62vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:43:52.570898 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:52.570862    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m62vm\" (UniqueName: \"kubernetes.io/projected/6ae7dd23-d3be-43b4-942b-eaab7ece883f-kube-api-access-m62vm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:43:52.570898 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:52.570891    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae7dd23-d3be-43b4-942b-eaab7ece883f-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:43:52.570898 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:52.570901    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae7dd23-d3be-43b4-942b-eaab7ece883f-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:43:52.570898 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:52.570909    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae7dd23-d3be-43b4-942b-eaab7ece883f-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:43:52.589043 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:52.589013    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt" podUID="b62c9031-c0bd-4d76-a455-00a322453fe7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:43:52.589164 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:52.589115    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt"
Sep 09 20:43:53.211872 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:53.211836    2581 generic.go:356] "Generic (PLEG): container finished" podID="6ae7dd23-d3be-43b4-942b-eaab7ece883f" containerID="ac9695401c89a4dc4d27aecf96ef59f86b4a4bf0ab7bb9ce003965cff186a15e" exitCode=0
Sep 09 20:43:53.212343 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:53.211883    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"
Sep 09 20:43:53.212343 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:53.211893    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw" event={"ID":"6ae7dd23-d3be-43b4-942b-eaab7ece883f","Type":"ContainerDied","Data":"ac9695401c89a4dc4d27aecf96ef59f86b4a4bf0ab7bb9ce003965cff186a15e"}
Sep 09 20:43:53.212343 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:53.211940    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw" event={"ID":"6ae7dd23-d3be-43b4-942b-eaab7ece883f","Type":"ContainerDied","Data":"fcbfb7eaad40627600bbbefaa627f0076a0a4d0964da6de65b2c749f7525b844"}
Sep 09 20:43:53.212343 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:53.211963    2581 scope.go:117] "RemoveContainer" containerID="ac9695401c89a4dc4d27aecf96ef59f86b4a4bf0ab7bb9ce003965cff186a15e"
Sep 09 20:43:53.220447 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:53.220432    2581 scope.go:117] "RemoveContainer" containerID="ac9695401c89a4dc4d27aecf96ef59f86b4a4bf0ab7bb9ce003965cff186a15e"
Sep 09 20:43:53.220766 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:43:53.220747    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac9695401c89a4dc4d27aecf96ef59f86b4a4bf0ab7bb9ce003965cff186a15e\": container with ID starting with ac9695401c89a4dc4d27aecf96ef59f86b4a4bf0ab7bb9ce003965cff186a15e not found: ID does not exist" containerID="ac9695401c89a4dc4d27aecf96ef59f86b4a4bf0ab7bb9ce003965cff186a15e"
Sep 09 20:43:53.220839 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:53.220772    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac9695401c89a4dc4d27aecf96ef59f86b4a4bf0ab7bb9ce003965cff186a15e"} err="failed to get container status \"ac9695401c89a4dc4d27aecf96ef59f86b4a4bf0ab7bb9ce003965cff186a15e\": rpc error: code = NotFound desc = could not find container \"ac9695401c89a4dc4d27aecf96ef59f86b4a4bf0ab7bb9ce003965cff186a15e\": container with ID starting with ac9695401c89a4dc4d27aecf96ef59f86b4a4bf0ab7bb9ce003965cff186a15e not found: ID does not exist"
Sep 09 20:43:53.230055 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:53.230033    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"]
Sep 09 20:43:53.237137 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:53.237115    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-4djfw"]
Sep 09 20:43:54.635705 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:43:54.635672    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ae7dd23-d3be-43b4-942b-eaab7ece883f" path="/var/lib/kubelet/pods/6ae7dd23-d3be-43b4-942b-eaab7ece883f/volumes"
Sep 09 20:44:02.589346 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:02.589309    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt" podUID="b62c9031-c0bd-4d76-a455-00a322453fe7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:44:12.589726 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:12.589681    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt" podUID="b62c9031-c0bd-4d76-a455-00a322453fe7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:44:15.684501 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:15.684454    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt"
Sep 09 20:44:15.863215 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:15.863185    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8qx5\" (UniqueName: \"kubernetes.io/projected/b62c9031-c0bd-4d76-a455-00a322453fe7-kube-api-access-c8qx5\") pod \"b62c9031-c0bd-4d76-a455-00a322453fe7\" (UID: \"b62c9031-c0bd-4d76-a455-00a322453fe7\") "
Sep 09 20:44:15.863428 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:15.863234    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b62c9031-c0bd-4d76-a455-00a322453fe7-tls-certs\") pod \"b62c9031-c0bd-4d76-a455-00a322453fe7\" (UID: \"b62c9031-c0bd-4d76-a455-00a322453fe7\") "
Sep 09 20:44:15.865434 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:15.865397    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b62c9031-c0bd-4d76-a455-00a322453fe7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b62c9031-c0bd-4d76-a455-00a322453fe7" (UID: "b62c9031-c0bd-4d76-a455-00a322453fe7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:44:15.865591 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:15.865444    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b62c9031-c0bd-4d76-a455-00a322453fe7-kube-api-access-c8qx5" (OuterVolumeSpecName: "kube-api-access-c8qx5") pod "b62c9031-c0bd-4d76-a455-00a322453fe7" (UID: "b62c9031-c0bd-4d76-a455-00a322453fe7"). InnerVolumeSpecName "kube-api-access-c8qx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:44:15.964594 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:15.964552    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b62c9031-c0bd-4d76-a455-00a322453fe7-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:44:15.964594 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:15.964592    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8qx5\" (UniqueName: \"kubernetes.io/projected/b62c9031-c0bd-4d76-a455-00a322453fe7-kube-api-access-c8qx5\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:44:16.328782 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:16.328750    2581 generic.go:356] "Generic (PLEG): container finished" podID="b62c9031-c0bd-4d76-a455-00a322453fe7" containerID="2be349dfb0bb3f28ebb87d936eb15174a4c19ba7582125dd3587bf96d787b874" exitCode=0
Sep 09 20:44:16.328958 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:16.328796    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt"
Sep 09 20:44:16.328958 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:16.328839    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt" event={"ID":"b62c9031-c0bd-4d76-a455-00a322453fe7","Type":"ContainerDied","Data":"2be349dfb0bb3f28ebb87d936eb15174a4c19ba7582125dd3587bf96d787b874"}
Sep 09 20:44:16.328958 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:16.328898    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt" event={"ID":"b62c9031-c0bd-4d76-a455-00a322453fe7","Type":"ContainerDied","Data":"b474938eec404d90af10ad9001a1b195a9f70f7e46f44a5e1d0a3850bc8a95b1"}
Sep 09 20:44:16.328958 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:16.328921    2581 scope.go:117] "RemoveContainer" containerID="2be349dfb0bb3f28ebb87d936eb15174a4c19ba7582125dd3587bf96d787b874"
Sep 09 20:44:16.337874 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:16.337857    2581 scope.go:117] "RemoveContainer" containerID="2be349dfb0bb3f28ebb87d936eb15174a4c19ba7582125dd3587bf96d787b874"
Sep 09 20:44:16.338117 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:44:16.338098    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be349dfb0bb3f28ebb87d936eb15174a4c19ba7582125dd3587bf96d787b874\": container with ID starting with 2be349dfb0bb3f28ebb87d936eb15174a4c19ba7582125dd3587bf96d787b874 not found: ID does not exist" containerID="2be349dfb0bb3f28ebb87d936eb15174a4c19ba7582125dd3587bf96d787b874"
Sep 09 20:44:16.338185 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:16.338128    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be349dfb0bb3f28ebb87d936eb15174a4c19ba7582125dd3587bf96d787b874"} err="failed to get container status \"2be349dfb0bb3f28ebb87d936eb15174a4c19ba7582125dd3587bf96d787b874\": rpc error: code = NotFound desc = could not find container \"2be349dfb0bb3f28ebb87d936eb15174a4c19ba7582125dd3587bf96d787b874\": container with ID starting with 2be349dfb0bb3f28ebb87d936eb15174a4c19ba7582125dd3587bf96d787b874 not found: ID does not exist"
Sep 09 20:44:16.351026 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:16.351002    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt"]
Sep 09 20:44:16.357330 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:16.357311    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched47hlt"]
Sep 09 20:44:16.634893 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:16.634817    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b62c9031-c0bd-4d76-a455-00a322453fe7" path="/var/lib/kubelet/pods/b62c9031-c0bd-4d76-a455-00a322453fe7/volumes"
Sep 09 20:44:38.577362 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.577331    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"]
Sep 09 20:44:38.577858 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.577703    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b62c9031-c0bd-4d76-a455-00a322453fe7" containerName="main"
Sep 09 20:44:38.577858 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.577716    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b62c9031-c0bd-4d76-a455-00a322453fe7" containerName="main"
Sep 09 20:44:38.577858 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.577724    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ae7dd23-d3be-43b4-942b-eaab7ece883f" containerName="main"
Sep 09 20:44:38.577858 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.577730    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ae7dd23-d3be-43b4-942b-eaab7ece883f" containerName="main"
Sep 09 20:44:38.577858 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.577784    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b62c9031-c0bd-4d76-a455-00a322453fe7" containerName="main"
Sep 09 20:44:38.577858 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.577792    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ae7dd23-d3be-43b4-942b-eaab7ece883f" containerName="main"
Sep 09 20:44:38.580824 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.580808    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.584568 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.584527    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 09 20:44:38.584568 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.584535    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-pngj7\""
Sep 09 20:44:38.584568 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.584571    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:44:38.584843 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.584603    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 09 20:44:38.584843 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.584538    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 09 20:44:38.584843 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.584723    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 09 20:44:38.592189 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.592168    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"]
Sep 09 20:44:38.645830 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.645805    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-dshm\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-7nsql\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.645973 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.645833    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1987ac26-6193-4bce-9407-bbbee45f65d5-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-7nsql\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.645973 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.645864    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-7nsql\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.645973 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.645888    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1987ac26-6193-4bce-9407-bbbee45f65d5-tls-certs\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-7nsql\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.645973 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.645918    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-model-cache\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-7nsql\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.645973 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.645935    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkvbt\" (UniqueName: \"kubernetes.io/projected/1987ac26-6193-4bce-9407-bbbee45f65d5-kube-api-access-qkvbt\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-7nsql\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.645973 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.645960    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-7nsql\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.646169 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.646019    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-home\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-7nsql\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.747328 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.747289    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-home\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-7nsql\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.747328 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.747333    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-dshm\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-7nsql\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.747619 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.747350    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1987ac26-6193-4bce-9407-bbbee45f65d5-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-7nsql\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.747619 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.747490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-7nsql\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.747619 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.747527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1987ac26-6193-4bce-9407-bbbee45f65d5-tls-certs\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-7nsql\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.747619 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.747560    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-model-cache\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-7nsql\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.747836 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.747680    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkvbt\" (UniqueName: \"kubernetes.io/projected/1987ac26-6193-4bce-9407-bbbee45f65d5-kube-api-access-qkvbt\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-7nsql\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.747836 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.747730    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-7nsql\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.747836 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.747775    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-home\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-7nsql\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.747989 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.747932    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-model-cache\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-7nsql\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.747989 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.747981    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-7nsql\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.748099 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.748078    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-7nsql\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.748187 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.748169    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1987ac26-6193-4bce-9407-bbbee45f65d5-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-7nsql\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.749817 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.749794    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-dshm\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-7nsql\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.750045 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.750031    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1987ac26-6193-4bce-9407-bbbee45f65d5-tls-certs\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-7nsql\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.756669 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.756650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkvbt\" (UniqueName: \"kubernetes.io/projected/1987ac26-6193-4bce-9407-bbbee45f65d5-kube-api-access-qkvbt\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-7nsql\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.818285 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.818251    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x"]
Sep 09 20:44:38.822926 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.822906    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x"
Sep 09 20:44:38.832389 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.832364    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x"]
Sep 09 20:44:38.849034 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.849006    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc4911f-e4c6-45be-a337-2bc9348ff190-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x\" (UID: \"3dc4911f-e4c6-45be-a337-2bc9348ff190\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x"
Sep 09 20:44:38.849140 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.849063    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dc4911f-e4c6-45be-a337-2bc9348ff190-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x\" (UID: \"3dc4911f-e4c6-45be-a337-2bc9348ff190\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x"
Sep 09 20:44:38.849140 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.849084    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dc4911f-e4c6-45be-a337-2bc9348ff190-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x\" (UID: \"3dc4911f-e4c6-45be-a337-2bc9348ff190\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x"
Sep 09 20:44:38.849140 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.849115    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dc4911f-e4c6-45be-a337-2bc9348ff190-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x\" (UID: \"3dc4911f-e4c6-45be-a337-2bc9348ff190\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x"
Sep 09 20:44:38.891760 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.891730    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:38.950059 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.950025    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc4911f-e4c6-45be-a337-2bc9348ff190-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x\" (UID: \"3dc4911f-e4c6-45be-a337-2bc9348ff190\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x"
Sep 09 20:44:38.950196 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.950132    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dc4911f-e4c6-45be-a337-2bc9348ff190-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x\" (UID: \"3dc4911f-e4c6-45be-a337-2bc9348ff190\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x"
Sep 09 20:44:38.950196 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.950172    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dc4911f-e4c6-45be-a337-2bc9348ff190-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x\" (UID: \"3dc4911f-e4c6-45be-a337-2bc9348ff190\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x"
Sep 09 20:44:38.950315 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.950241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dc4911f-e4c6-45be-a337-2bc9348ff190-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x\" (UID: \"3dc4911f-e4c6-45be-a337-2bc9348ff190\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x"
Sep 09 20:44:38.950514 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.950494    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc4911f-e4c6-45be-a337-2bc9348ff190-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x\" (UID: \"3dc4911f-e4c6-45be-a337-2bc9348ff190\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x"
Sep 09 20:44:38.950618 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.950597    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dc4911f-e4c6-45be-a337-2bc9348ff190-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x\" (UID: \"3dc4911f-e4c6-45be-a337-2bc9348ff190\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x"
Sep 09 20:44:38.951024 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.951001    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dc4911f-e4c6-45be-a337-2bc9348ff190-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x\" (UID: \"3dc4911f-e4c6-45be-a337-2bc9348ff190\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x"
Sep 09 20:44:38.953575 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:38.953534    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dc4911f-e4c6-45be-a337-2bc9348ff190-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x\" (UID: \"3dc4911f-e4c6-45be-a337-2bc9348ff190\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x"
Sep 09 20:44:39.134811 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:39.134731    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x"
Sep 09 20:44:39.171680 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:39.171650    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"]
Sep 09 20:44:39.173673 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:44:39.173643    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1987ac26_6193_4bce_9407_bbbee45f65d5.slice/crio-fbb1e9502e3bfae173b8b452e9a722dd10e59315ca54d663b16400dd6a2757eb WatchSource:0}: Error finding container fbb1e9502e3bfae173b8b452e9a722dd10e59315ca54d663b16400dd6a2757eb: Status 404 returned error can't find the container with id fbb1e9502e3bfae173b8b452e9a722dd10e59315ca54d663b16400dd6a2757eb
Sep 09 20:44:39.410702 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:39.410674    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x"]
Sep 09 20:44:39.412502 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:44:39.412394    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc4911f_e4c6_45be_a337_2bc9348ff190.slice/crio-c656b48a8d48e717597020f48c53de66453543c41f168e08731743d69388f8de WatchSource:0}: Error finding container c656b48a8d48e717597020f48c53de66453543c41f168e08731743d69388f8de: Status 404 returned error can't find the container with id c656b48a8d48e717597020f48c53de66453543c41f168e08731743d69388f8de
Sep 09 20:44:39.443222 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:39.443192    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x" event={"ID":"3dc4911f-e4c6-45be-a337-2bc9348ff190","Type":"ContainerStarted","Data":"c656b48a8d48e717597020f48c53de66453543c41f168e08731743d69388f8de"}
Sep 09 20:44:39.444747 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:39.444722    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql" event={"ID":"1987ac26-6193-4bce-9407-bbbee45f65d5","Type":"ContainerStarted","Data":"dce507a1e9e34800ff32742b0fa5b0ce31993585eb1ad1b10aff01d932a3350e"}
Sep 09 20:44:39.444837 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:39.444776    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql" event={"ID":"1987ac26-6193-4bce-9407-bbbee45f65d5","Type":"ContainerStarted","Data":"fbb1e9502e3bfae173b8b452e9a722dd10e59315ca54d663b16400dd6a2757eb"}
Sep 09 20:44:40.453897 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:40.453855    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x" event={"ID":"3dc4911f-e4c6-45be-a337-2bc9348ff190","Type":"ContainerStarted","Data":"42adea08d32766af1843f9c19caef5d27fbc91e14717931d76a3b2974d934293"}
Sep 09 20:44:42.468340 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:42.468304    2581 generic.go:356] "Generic (PLEG): container finished" podID="1987ac26-6193-4bce-9407-bbbee45f65d5" containerID="dce507a1e9e34800ff32742b0fa5b0ce31993585eb1ad1b10aff01d932a3350e" exitCode=0
Sep 09 20:44:42.468810 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:42.468379    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql" event={"ID":"1987ac26-6193-4bce-9407-bbbee45f65d5","Type":"ContainerDied","Data":"dce507a1e9e34800ff32742b0fa5b0ce31993585eb1ad1b10aff01d932a3350e"}
Sep 09 20:44:42.470119 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:42.470098    2581 generic.go:356] "Generic (PLEG): container finished" podID="3dc4911f-e4c6-45be-a337-2bc9348ff190" containerID="42adea08d32766af1843f9c19caef5d27fbc91e14717931d76a3b2974d934293" exitCode=0
Sep 09 20:44:42.470226 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:42.470118    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x" event={"ID":"3dc4911f-e4c6-45be-a337-2bc9348ff190","Type":"ContainerDied","Data":"42adea08d32766af1843f9c19caef5d27fbc91e14717931d76a3b2974d934293"}
Sep 09 20:44:42.471104 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:42.471088    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:44:43.478126 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:43.478085    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql" event={"ID":"1987ac26-6193-4bce-9407-bbbee45f65d5","Type":"ContainerStarted","Data":"b027a0ccdf434e9dc145a3e7eef6471de340d07bc84265fcca37f8536aa6aae2"}
Sep 09 20:44:43.505370 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:43.505325    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql" podStartSLOduration=2.843497986 podStartE2EDuration="5.505306854s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.661808868s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:44:43.504872439 +0000 UTC m=+2025.510049434" watchObservedRunningTime="2026-09-09 20:44:43.505306854 +0000 UTC m=+2025.510483850"
Sep 09 20:44:43.892033 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:43.891949    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:48.892315 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:48.892278    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:48.911059 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:48.911023    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:44:49.536827 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:44:49.536750    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:45:29.773098 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:45:29.772158    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x" event={"ID":"3dc4911f-e4c6-45be-a337-2bc9348ff190","Type":"ContainerStarted","Data":"c4f3f8a70a99a9eb27e26ca3250a2a4c0489545d29f889181b5e52ca24b45723"}
Sep 09 20:45:29.773098 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:45:29.773040    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x"
Sep 09 20:45:29.793414 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:45:29.793365    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x" podStartSLOduration=2.310053287 podStartE2EDuration="51.793352258s" totalImagesPullingTime="46.867630997s" totalInitContainerRuntime="2.615667974s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:45:29.790429217 +0000 UTC m=+2071.795606212" watchObservedRunningTime="2026-09-09 20:45:29.793352258 +0000 UTC m=+2071.798529252"
Sep 09 20:45:58.600750 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:45:58.600723    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 20:45:59.051246 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:45:58.603361    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 20:45:59.051246 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:45:58.603419    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 20:45:59.051246 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:45:58.606256    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 20:46:00.789731 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:00.789696    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x"
Sep 09 20:46:33.613826 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:33.613797    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x_3dc4911f-e4c6-45be-a337-2bc9348ff190/main/0.log"
Sep 09 20:46:45.697000 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:45.696962    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x"]
Sep 09 20:46:45.697869 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:45.697834    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x" podUID="3dc4911f-e4c6-45be-a337-2bc9348ff190" containerName="main" containerID="cri-o://c4f3f8a70a99a9eb27e26ca3250a2a4c0489545d29f889181b5e52ca24b45723" gracePeriod=30
Sep 09 20:46:45.705639 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:45.705616    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"]
Sep 09 20:46:45.714054 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:46:45.714019    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:46:45.714054 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:46:45.714054 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:46:45.714054 ip-10-0-134-84 kubenswrapper[2581]:  > containerID="b027a0ccdf434e9dc145a3e7eef6471de340d07bc84265fcca37f8536aa6aae2" cmd=["/bin/sleep","15"]
Sep 09 20:46:45.714255 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:45.714074    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql" message="" err=<
Sep 09 20:46:45.714255 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:46:45.714255 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:46:45.714255 ip-10-0-134-84 kubenswrapper[2581]:  >
Sep 09 20:46:45.714255 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:46:45.714087    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:46:45.714255 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:46:45.714255 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:46:45.714255 ip-10-0-134-84 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql" podUID="1987ac26-6193-4bce-9407-bbbee45f65d5" containerName="main" containerID="cri-o://b027a0ccdf434e9dc145a3e7eef6471de340d07bc84265fcca37f8536aa6aae2"
Sep 09 20:46:45.714255 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:45.714118    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql" podUID="1987ac26-6193-4bce-9407-bbbee45f65d5" containerName="main" containerID="cri-o://b027a0ccdf434e9dc145a3e7eef6471de340d07bc84265fcca37f8536aa6aae2" gracePeriod=60
Sep 09 20:46:46.164014 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.163982    2581 generic.go:356] "Generic (PLEG): container finished" podID="1987ac26-6193-4bce-9407-bbbee45f65d5" containerID="b027a0ccdf434e9dc145a3e7eef6471de340d07bc84265fcca37f8536aa6aae2" exitCode=0
Sep 09 20:46:46.164139 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.164056    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql" event={"ID":"1987ac26-6193-4bce-9407-bbbee45f65d5","Type":"ContainerDied","Data":"b027a0ccdf434e9dc145a3e7eef6471de340d07bc84265fcca37f8536aa6aae2"}
Sep 09 20:46:46.351936 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.351913    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x"
Sep 09 20:46:46.473264 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.473240    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc4911f-e4c6-45be-a337-2bc9348ff190-tmp-dir\") pod \"3dc4911f-e4c6-45be-a337-2bc9348ff190\" (UID: \"3dc4911f-e4c6-45be-a337-2bc9348ff190\") "
Sep 09 20:46:46.473379 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.473270    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dc4911f-e4c6-45be-a337-2bc9348ff190-tls-certs\") pod \"3dc4911f-e4c6-45be-a337-2bc9348ff190\" (UID: \"3dc4911f-e4c6-45be-a337-2bc9348ff190\") "
Sep 09 20:46:46.473379 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.473305    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dc4911f-e4c6-45be-a337-2bc9348ff190-cabundle-cert\") pod \"3dc4911f-e4c6-45be-a337-2bc9348ff190\" (UID: \"3dc4911f-e4c6-45be-a337-2bc9348ff190\") "
Sep 09 20:46:46.473515 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.473381    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dc4911f-e4c6-45be-a337-2bc9348ff190-kserve-provision-location\") pod \"3dc4911f-e4c6-45be-a337-2bc9348ff190\" (UID: \"3dc4911f-e4c6-45be-a337-2bc9348ff190\") "
Sep 09 20:46:46.473719 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.473695    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dc4911f-e4c6-45be-a337-2bc9348ff190-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3dc4911f-e4c6-45be-a337-2bc9348ff190" (UID: "3dc4911f-e4c6-45be-a337-2bc9348ff190"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:46.473777 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.473693    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dc4911f-e4c6-45be-a337-2bc9348ff190-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3dc4911f-e4c6-45be-a337-2bc9348ff190" (UID: "3dc4911f-e4c6-45be-a337-2bc9348ff190"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:46:46.474172 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.474158    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:46:46.475357 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.475332    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dc4911f-e4c6-45be-a337-2bc9348ff190-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3dc4911f-e4c6-45be-a337-2bc9348ff190" (UID: "3dc4911f-e4c6-45be-a337-2bc9348ff190"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:46:46.494327 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.494298    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dc4911f-e4c6-45be-a337-2bc9348ff190-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3dc4911f-e4c6-45be-a337-2bc9348ff190" (UID: "3dc4911f-e4c6-45be-a337-2bc9348ff190"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:46.574071 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.574041    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1987ac26-6193-4bce-9407-bbbee45f65d5-tls-certs\") pod \"1987ac26-6193-4bce-9407-bbbee45f65d5\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") "
Sep 09 20:46:46.574248 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.574077    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-home\") pod \"1987ac26-6193-4bce-9407-bbbee45f65d5\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") "
Sep 09 20:46:46.574248 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.574111    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-dshm\") pod \"1987ac26-6193-4bce-9407-bbbee45f65d5\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") "
Sep 09 20:46:46.574248 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.574131    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-model-cache\") pod \"1987ac26-6193-4bce-9407-bbbee45f65d5\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") "
Sep 09 20:46:46.574248 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.574150    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-kserve-provision-location\") pod \"1987ac26-6193-4bce-9407-bbbee45f65d5\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") "
Sep 09 20:46:46.574248 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.574196    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1987ac26-6193-4bce-9407-bbbee45f65d5-cabundle-cert\") pod \"1987ac26-6193-4bce-9407-bbbee45f65d5\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") "
Sep 09 20:46:46.574248 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.574227    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-tmp-dir\") pod \"1987ac26-6193-4bce-9407-bbbee45f65d5\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") "
Sep 09 20:46:46.574587 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.574253    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkvbt\" (UniqueName: \"kubernetes.io/projected/1987ac26-6193-4bce-9407-bbbee45f65d5-kube-api-access-qkvbt\") pod \"1987ac26-6193-4bce-9407-bbbee45f65d5\" (UID: \"1987ac26-6193-4bce-9407-bbbee45f65d5\") "
Sep 09 20:46:46.574587 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.574374    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-home" (OuterVolumeSpecName: "home") pod "1987ac26-6193-4bce-9407-bbbee45f65d5" (UID: "1987ac26-6193-4bce-9407-bbbee45f65d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:46.574587 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.574426    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "1987ac26-6193-4bce-9407-bbbee45f65d5" (UID: "1987ac26-6193-4bce-9407-bbbee45f65d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:46.574703 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.574598    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:46:46.574703 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.574621    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:46:46.574703 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.574635    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc4911f-e4c6-45be-a337-2bc9348ff190-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:46:46.574703 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.574647    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dc4911f-e4c6-45be-a337-2bc9348ff190-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:46:46.574703 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.574658    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dc4911f-e4c6-45be-a337-2bc9348ff190-cabundle-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:46:46.574703 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.574671    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dc4911f-e4c6-45be-a337-2bc9348ff190-kserve-provision-location\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:46:46.574703 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.574686    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1987ac26-6193-4bce-9407-bbbee45f65d5" (UID: "1987ac26-6193-4bce-9407-bbbee45f65d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:46.574964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.574741    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1987ac26-6193-4bce-9407-bbbee45f65d5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1987ac26-6193-4bce-9407-bbbee45f65d5" (UID: "1987ac26-6193-4bce-9407-bbbee45f65d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:46:46.576304 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.576280    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1987ac26-6193-4bce-9407-bbbee45f65d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1987ac26-6193-4bce-9407-bbbee45f65d5" (UID: "1987ac26-6193-4bce-9407-bbbee45f65d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:46:46.576711 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.576684    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-dshm" (OuterVolumeSpecName: "dshm") pod "1987ac26-6193-4bce-9407-bbbee45f65d5" (UID: "1987ac26-6193-4bce-9407-bbbee45f65d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:46.576828 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.576733    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1987ac26-6193-4bce-9407-bbbee45f65d5-kube-api-access-qkvbt" (OuterVolumeSpecName: "kube-api-access-qkvbt") pod "1987ac26-6193-4bce-9407-bbbee45f65d5" (UID: "1987ac26-6193-4bce-9407-bbbee45f65d5"). InnerVolumeSpecName "kube-api-access-qkvbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:46:46.593534 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.593511    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1987ac26-6193-4bce-9407-bbbee45f65d5" (UID: "1987ac26-6193-4bce-9407-bbbee45f65d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:46.675341 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.675304    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1987ac26-6193-4bce-9407-bbbee45f65d5-cabundle-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:46:46.675341 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.675329    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:46:46.675341 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.675338    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkvbt\" (UniqueName: \"kubernetes.io/projected/1987ac26-6193-4bce-9407-bbbee45f65d5-kube-api-access-qkvbt\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:46:46.675341 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.675347    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1987ac26-6193-4bce-9407-bbbee45f65d5-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:46:46.675685 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.675357    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:46:46.675685 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:46.675365    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1987ac26-6193-4bce-9407-bbbee45f65d5-kserve-provision-location\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:46:47.171067 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:47.170980    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"
Sep 09 20:46:47.171067 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:47.171015    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql" event={"ID":"1987ac26-6193-4bce-9407-bbbee45f65d5","Type":"ContainerDied","Data":"fbb1e9502e3bfae173b8b452e9a722dd10e59315ca54d663b16400dd6a2757eb"}
Sep 09 20:46:47.171553 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:47.171092    2581 scope.go:117] "RemoveContainer" containerID="b027a0ccdf434e9dc145a3e7eef6471de340d07bc84265fcca37f8536aa6aae2"
Sep 09 20:46:47.172927 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:47.172892    2581 generic.go:356] "Generic (PLEG): container finished" podID="3dc4911f-e4c6-45be-a337-2bc9348ff190" containerID="c4f3f8a70a99a9eb27e26ca3250a2a4c0489545d29f889181b5e52ca24b45723" exitCode=0
Sep 09 20:46:47.173039 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:47.172953    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x"
Sep 09 20:46:47.173039 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:47.172955    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x" event={"ID":"3dc4911f-e4c6-45be-a337-2bc9348ff190","Type":"ContainerDied","Data":"c4f3f8a70a99a9eb27e26ca3250a2a4c0489545d29f889181b5e52ca24b45723"}
Sep 09 20:46:47.173172 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:47.173078    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x" event={"ID":"3dc4911f-e4c6-45be-a337-2bc9348ff190","Type":"ContainerDied","Data":"c656b48a8d48e717597020f48c53de66453543c41f168e08731743d69388f8de"}
Sep 09 20:46:47.181028 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:47.181009    2581 scope.go:117] "RemoveContainer" containerID="dce507a1e9e34800ff32742b0fa5b0ce31993585eb1ad1b10aff01d932a3350e"
Sep 09 20:46:47.189068 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:47.189045    2581 scope.go:117] "RemoveContainer" containerID="c4f3f8a70a99a9eb27e26ca3250a2a4c0489545d29f889181b5e52ca24b45723"
Sep 09 20:46:47.191774 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:47.191752    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"]
Sep 09 20:46:47.196906 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:47.196877    2581 scope.go:117] "RemoveContainer" containerID="42adea08d32766af1843f9c19caef5d27fbc91e14717931d76a3b2974d934293"
Sep 09 20:46:47.199171 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:47.199149    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-7nsql"]
Sep 09 20:46:47.204449 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:47.204430    2581 scope.go:117] "RemoveContainer" containerID="c4f3f8a70a99a9eb27e26ca3250a2a4c0489545d29f889181b5e52ca24b45723"
Sep 09 20:46:47.204736 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:46:47.204718    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f3f8a70a99a9eb27e26ca3250a2a4c0489545d29f889181b5e52ca24b45723\": container with ID starting with c4f3f8a70a99a9eb27e26ca3250a2a4c0489545d29f889181b5e52ca24b45723 not found: ID does not exist" containerID="c4f3f8a70a99a9eb27e26ca3250a2a4c0489545d29f889181b5e52ca24b45723"
Sep 09 20:46:47.204782 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:47.204744    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f3f8a70a99a9eb27e26ca3250a2a4c0489545d29f889181b5e52ca24b45723"} err="failed to get container status \"c4f3f8a70a99a9eb27e26ca3250a2a4c0489545d29f889181b5e52ca24b45723\": rpc error: code = NotFound desc = could not find container \"c4f3f8a70a99a9eb27e26ca3250a2a4c0489545d29f889181b5e52ca24b45723\": container with ID starting with c4f3f8a70a99a9eb27e26ca3250a2a4c0489545d29f889181b5e52ca24b45723 not found: ID does not exist"
Sep 09 20:46:47.204782 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:47.204760    2581 scope.go:117] "RemoveContainer" containerID="42adea08d32766af1843f9c19caef5d27fbc91e14717931d76a3b2974d934293"
Sep 09 20:46:47.205007 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:46:47.204989    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42adea08d32766af1843f9c19caef5d27fbc91e14717931d76a3b2974d934293\": container with ID starting with 42adea08d32766af1843f9c19caef5d27fbc91e14717931d76a3b2974d934293 not found: ID does not exist" containerID="42adea08d32766af1843f9c19caef5d27fbc91e14717931d76a3b2974d934293"
Sep 09 20:46:47.205048 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:47.205013    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42adea08d32766af1843f9c19caef5d27fbc91e14717931d76a3b2974d934293"} err="failed to get container status \"42adea08d32766af1843f9c19caef5d27fbc91e14717931d76a3b2974d934293\": rpc error: code = NotFound desc = could not find container \"42adea08d32766af1843f9c19caef5d27fbc91e14717931d76a3b2974d934293\": container with ID starting with 42adea08d32766af1843f9c19caef5d27fbc91e14717931d76a3b2974d934293 not found: ID does not exist"
Sep 09 20:46:47.215468 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:47.215436    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x"]
Sep 09 20:46:47.220859 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:47.220837    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-86db9f67ff-fbh4x"]
Sep 09 20:46:48.637644 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:48.637607    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1987ac26-6193-4bce-9407-bbbee45f65d5" path="/var/lib/kubelet/pods/1987ac26-6193-4bce-9407-bbbee45f65d5/volumes"
Sep 09 20:46:48.638229 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:46:48.638207    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dc4911f-e4c6-45be-a337-2bc9348ff190" path="/var/lib/kubelet/pods/3dc4911f-e4c6-45be-a337-2bc9348ff190/volumes"
Sep 09 20:47:56.570955 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.570919    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"]
Sep 09 20:47:56.571448 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.571224    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1987ac26-6193-4bce-9407-bbbee45f65d5" containerName="main"
Sep 09 20:47:56.571448 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.571235    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1987ac26-6193-4bce-9407-bbbee45f65d5" containerName="main"
Sep 09 20:47:56.571448 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.571243    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dc4911f-e4c6-45be-a337-2bc9348ff190" containerName="main"
Sep 09 20:47:56.571448 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.571250    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dc4911f-e4c6-45be-a337-2bc9348ff190" containerName="main"
Sep 09 20:47:56.571448 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.571266    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dc4911f-e4c6-45be-a337-2bc9348ff190" containerName="storage-initializer"
Sep 09 20:47:56.571448 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.571272    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dc4911f-e4c6-45be-a337-2bc9348ff190" containerName="storage-initializer"
Sep 09 20:47:56.571448 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.571281    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1987ac26-6193-4bce-9407-bbbee45f65d5" containerName="storage-initializer"
Sep 09 20:47:56.571448 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.571287    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1987ac26-6193-4bce-9407-bbbee45f65d5" containerName="storage-initializer"
Sep 09 20:47:56.571448 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.571339    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dc4911f-e4c6-45be-a337-2bc9348ff190" containerName="main"
Sep 09 20:47:56.571448 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.571349    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1987ac26-6193-4bce-9407-bbbee45f65d5" containerName="main"
Sep 09 20:47:56.574757 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.574738    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.577181 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.577156    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 09 20:47:56.578265 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.578225    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-nfjpv\""
Sep 09 20:47:56.578265 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.578246    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:47:56.578428 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.578260    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 09 20:47:56.578428 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.578339    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 09 20:47:56.578428 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.578339    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 09 20:47:56.588617 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.588598    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"]
Sep 09 20:47:56.651721 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.651689    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-dshm\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-67drk\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.651898 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.651748    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-67drk\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.651898 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.651783    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed11f1f5-64b3-49c8-be6b-44e45516cca9-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-67drk\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.651898 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.651827    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-home\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-67drk\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.651898 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.651853    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncqrp\" (UniqueName: \"kubernetes.io/projected/ed11f1f5-64b3-49c8-be6b-44e45516cca9-kube-api-access-ncqrp\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-67drk\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.651898 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.651884    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-model-cache\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-67drk\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.652080 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.651949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-67drk\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.652080 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.651983    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed11f1f5-64b3-49c8-be6b-44e45516cca9-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-67drk\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.752907 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.752870    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-model-cache\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-67drk\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.752907 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.752910    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-67drk\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.753129 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.752933    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed11f1f5-64b3-49c8-be6b-44e45516cca9-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-67drk\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.753129 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.752959    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-dshm\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-67drk\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.753129 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.752993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-67drk\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.753129 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.753012    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed11f1f5-64b3-49c8-be6b-44e45516cca9-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-67drk\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.753129 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.753040    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-home\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-67drk\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.753129 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.753065    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncqrp\" (UniqueName: \"kubernetes.io/projected/ed11f1f5-64b3-49c8-be6b-44e45516cca9-kube-api-access-ncqrp\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-67drk\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.753450 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.753330    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-model-cache\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-67drk\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.753450 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.753348    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-67drk\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.753450 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.753438    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-home\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-67drk\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.753613 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.753537    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-67drk\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.753727 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.753708    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed11f1f5-64b3-49c8-be6b-44e45516cca9-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-67drk\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.755382 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.755361    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-dshm\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-67drk\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.755574 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.755556    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed11f1f5-64b3-49c8-be6b-44e45516cca9-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-67drk\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.761474 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.761441    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncqrp\" (UniqueName: \"kubernetes.io/projected/ed11f1f5-64b3-49c8-be6b-44e45516cca9-kube-api-access-ncqrp\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-67drk\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.848871 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.848796    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4"]
Sep 09 20:47:56.853093 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.853072    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4"
Sep 09 20:47:56.853638 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.853607    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4\" (UID: \"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4"
Sep 09 20:47:56.853759 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.853663    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4\" (UID: \"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4"
Sep 09 20:47:56.853759 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.853729    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4\" (UID: \"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4"
Sep 09 20:47:56.853855 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.853779    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4\" (UID: \"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4"
Sep 09 20:47:56.862435 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.862409    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4"]
Sep 09 20:47:56.884837 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.884808    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:47:56.954409 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.954371    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4\" (UID: \"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4"
Sep 09 20:47:56.954568 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.954423    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4\" (UID: \"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4"
Sep 09 20:47:56.954568 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.954498    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4\" (UID: \"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4"
Sep 09 20:47:56.954668 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.954634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4\" (UID: \"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4"
Sep 09 20:47:56.954841 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.954819    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4\" (UID: \"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4"
Sep 09 20:47:56.954841 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.954835    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4\" (UID: \"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4"
Sep 09 20:47:56.955271 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.955250    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4\" (UID: \"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4"
Sep 09 20:47:56.957007 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:56.956978    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4\" (UID: \"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4"
Sep 09 20:47:57.160037 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:57.160009    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"]
Sep 09 20:47:57.161507 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:47:57.161481    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded11f1f5_64b3_49c8_be6b_44e45516cca9.slice/crio-83b64be493b995ec7c21a4e703d2ff703c15764e2a82207f59d19e66bc24a660 WatchSource:0}: Error finding container 83b64be493b995ec7c21a4e703d2ff703c15764e2a82207f59d19e66bc24a660: Status 404 returned error can't find the container with id 83b64be493b995ec7c21a4e703d2ff703c15764e2a82207f59d19e66bc24a660
Sep 09 20:47:57.166261 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:57.166238    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4"
Sep 09 20:47:57.442921 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:57.442893    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4"]
Sep 09 20:47:57.445071 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:47:57.445045    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5bf75d_ce1e_4e3f_b3e2_f6c49979545b.slice/crio-6f04c3e5b5e1de2850e3cacc0e777a824cec2486b93db3f145b0faef39afdcc4 WatchSource:0}: Error finding container 6f04c3e5b5e1de2850e3cacc0e777a824cec2486b93db3f145b0faef39afdcc4: Status 404 returned error can't find the container with id 6f04c3e5b5e1de2850e3cacc0e777a824cec2486b93db3f145b0faef39afdcc4
Sep 09 20:47:57.520829 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:57.520800    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4" event={"ID":"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b","Type":"ContainerStarted","Data":"bdb3bd4f55996b958df8779a128fbe80d6468f6ec2c14939a4bc20c8205abec2"}
Sep 09 20:47:57.520993 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:57.520857    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4" event={"ID":"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b","Type":"ContainerStarted","Data":"6f04c3e5b5e1de2850e3cacc0e777a824cec2486b93db3f145b0faef39afdcc4"}
Sep 09 20:47:57.522596 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:57.522567    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk" event={"ID":"ed11f1f5-64b3-49c8-be6b-44e45516cca9","Type":"ContainerStarted","Data":"dc23c450bf9613815c1f517b1953c37aeee0eaeaf58ffcbdb95aa327f7e561dc"}
Sep 09 20:47:57.522707 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:47:57.522626    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk" event={"ID":"ed11f1f5-64b3-49c8-be6b-44e45516cca9","Type":"ContainerStarted","Data":"83b64be493b995ec7c21a4e703d2ff703c15764e2a82207f59d19e66bc24a660"}
Sep 09 20:48:00.546332 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:48:00.546296    2581 generic.go:356] "Generic (PLEG): container finished" podID="ed11f1f5-64b3-49c8-be6b-44e45516cca9" containerID="dc23c450bf9613815c1f517b1953c37aeee0eaeaf58ffcbdb95aa327f7e561dc" exitCode=0
Sep 09 20:48:00.546766 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:48:00.546369    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk" event={"ID":"ed11f1f5-64b3-49c8-be6b-44e45516cca9","Type":"ContainerDied","Data":"dc23c450bf9613815c1f517b1953c37aeee0eaeaf58ffcbdb95aa327f7e561dc"}
Sep 09 20:48:00.548074 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:48:00.548054    2581 generic.go:356] "Generic (PLEG): container finished" podID="4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b" containerID="bdb3bd4f55996b958df8779a128fbe80d6468f6ec2c14939a4bc20c8205abec2" exitCode=0
Sep 09 20:48:00.548163 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:48:00.548093    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4" event={"ID":"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b","Type":"ContainerDied","Data":"bdb3bd4f55996b958df8779a128fbe80d6468f6ec2c14939a4bc20c8205abec2"}
Sep 09 20:48:01.556010 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:48:01.555973    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4" event={"ID":"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b","Type":"ContainerStarted","Data":"3b1c6458dd0ec04bc03b3b565e15ebf137f5ea6d44dcf190d1ddbd9bf60e586d"}
Sep 09 20:48:01.556485 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:48:01.556272    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4"
Sep 09 20:48:01.558126 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:48:01.558101    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk" event={"ID":"ed11f1f5-64b3-49c8-be6b-44e45516cca9","Type":"ContainerStarted","Data":"f0e4dd36f8e931d6a76e086e5d11712f8db112e9c3b86229c23b267ea3036ccb"}
Sep 09 20:48:01.575374 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:48:01.575328    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4" podStartSLOduration=2.949117613 podStartE2EDuration="5.575311278s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.626193665s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:01.573712196 +0000 UTC m=+2223.578889194" watchObservedRunningTime="2026-09-09 20:48:01.575311278 +0000 UTC m=+2223.580488272"
Sep 09 20:48:01.595676 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:48:01.595627    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk" podStartSLOduration=3.024397572 podStartE2EDuration="5.595612307s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.571214735s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:01.593853189 +0000 UTC m=+2223.599030185" watchObservedRunningTime="2026-09-09 20:48:01.595612307 +0000 UTC m=+2223.600789302"
Sep 09 20:48:01.885148 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:48:01.885060    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:48:06.885568 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:48:06.885527    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:48:06.898284 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:48:06.898255    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:48:07.600501 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:48:07.600470    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:48:32.575681 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:48:32.575589    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4"
Sep 09 20:48:54.582774 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:48:54.582738    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4_4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b/main/0.log"
Sep 09 20:49:29.098740 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.098704    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"]
Sep 09 20:49:29.101005 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.100979    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4"]
Sep 09 20:49:29.101210 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.101172    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4" podUID="4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b" containerName="main" containerID="cri-o://3b1c6458dd0ec04bc03b3b565e15ebf137f5ea6d44dcf190d1ddbd9bf60e586d" gracePeriod=30
Sep 09 20:49:29.109274 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:49:29.109241    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:49:29.109274 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:49:29.109274 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:49:29.109274 ip-10-0-134-84 kubenswrapper[2581]:  > containerID="f0e4dd36f8e931d6a76e086e5d11712f8db112e9c3b86229c23b267ea3036ccb" cmd=["/bin/sleep","15"]
Sep 09 20:49:29.109500 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.109295    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk" message="" err=<
Sep 09 20:49:29.109500 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:49:29.109500 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:49:29.109500 ip-10-0-134-84 kubenswrapper[2581]:  >
Sep 09 20:49:29.109500 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:49:29.109307    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:49:29.109500 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:49:29.109500 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:49:29.109500 ip-10-0-134-84 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk" podUID="ed11f1f5-64b3-49c8-be6b-44e45516cca9" containerName="main" containerID="cri-o://f0e4dd36f8e931d6a76e086e5d11712f8db112e9c3b86229c23b267ea3036ccb"
Sep 09 20:49:29.109500 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.109341    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk" podUID="ed11f1f5-64b3-49c8-be6b-44e45516cca9" containerName="main" containerID="cri-o://f0e4dd36f8e931d6a76e086e5d11712f8db112e9c3b86229c23b267ea3036ccb" gracePeriod=60
Sep 09 20:49:29.473107 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.473082    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:49:29.559713 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.559678    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-dshm\") pod \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") "
Sep 09 20:49:29.559871 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.559725    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-home\") pod \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") "
Sep 09 20:49:29.559871 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.559743    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-tmp-dir\") pod \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") "
Sep 09 20:49:29.559871 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.559785    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed11f1f5-64b3-49c8-be6b-44e45516cca9-tls-certs\") pod \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") "
Sep 09 20:49:29.559871 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.559814    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncqrp\" (UniqueName: \"kubernetes.io/projected/ed11f1f5-64b3-49c8-be6b-44e45516cca9-kube-api-access-ncqrp\") pod \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") "
Sep 09 20:49:29.559871 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.559833    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-kserve-provision-location\") pod \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") "
Sep 09 20:49:29.559871 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.559859    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-model-cache\") pod \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") "
Sep 09 20:49:29.560205 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.559906    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed11f1f5-64b3-49c8-be6b-44e45516cca9-cabundle-cert\") pod \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\" (UID: \"ed11f1f5-64b3-49c8-be6b-44e45516cca9\") "
Sep 09 20:49:29.560205 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.559941    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-home" (OuterVolumeSpecName: "home") pod "ed11f1f5-64b3-49c8-be6b-44e45516cca9" (UID: "ed11f1f5-64b3-49c8-be6b-44e45516cca9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:29.560205 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.560052    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ed11f1f5-64b3-49c8-be6b-44e45516cca9" (UID: "ed11f1f5-64b3-49c8-be6b-44e45516cca9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:29.560205 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.560193    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:49:29.560407 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.560192    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-model-cache" (OuterVolumeSpecName: "model-cache") pod "ed11f1f5-64b3-49c8-be6b-44e45516cca9" (UID: "ed11f1f5-64b3-49c8-be6b-44e45516cca9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:29.560407 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.560235    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:49:29.560407 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.560366    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed11f1f5-64b3-49c8-be6b-44e45516cca9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ed11f1f5-64b3-49c8-be6b-44e45516cca9" (UID: "ed11f1f5-64b3-49c8-be6b-44e45516cca9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:49:29.562130 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.562101    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-dshm" (OuterVolumeSpecName: "dshm") pod "ed11f1f5-64b3-49c8-be6b-44e45516cca9" (UID: "ed11f1f5-64b3-49c8-be6b-44e45516cca9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:29.562130 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.562107    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed11f1f5-64b3-49c8-be6b-44e45516cca9-kube-api-access-ncqrp" (OuterVolumeSpecName: "kube-api-access-ncqrp") pod "ed11f1f5-64b3-49c8-be6b-44e45516cca9" (UID: "ed11f1f5-64b3-49c8-be6b-44e45516cca9"). InnerVolumeSpecName "kube-api-access-ncqrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:49:29.562260 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.562164    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed11f1f5-64b3-49c8-be6b-44e45516cca9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ed11f1f5-64b3-49c8-be6b-44e45516cca9" (UID: "ed11f1f5-64b3-49c8-be6b-44e45516cca9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:49:29.580903 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.580878    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ed11f1f5-64b3-49c8-be6b-44e45516cca9" (UID: "ed11f1f5-64b3-49c8-be6b-44e45516cca9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:29.660885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.660810    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:49:29.660885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.660837    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed11f1f5-64b3-49c8-be6b-44e45516cca9-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:49:29.660885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.660847    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncqrp\" (UniqueName: \"kubernetes.io/projected/ed11f1f5-64b3-49c8-be6b-44e45516cca9-kube-api-access-ncqrp\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:49:29.660885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.660856    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-kserve-provision-location\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:49:29.660885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.660864    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed11f1f5-64b3-49c8-be6b-44e45516cca9-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:49:29.660885 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.660873    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed11f1f5-64b3-49c8-be6b-44e45516cca9-cabundle-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:49:29.742915 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.742886    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4"
Sep 09 20:49:29.861698 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.861664    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b-kserve-provision-location\") pod \"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b\" (UID: \"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b\") "
Sep 09 20:49:29.861900 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.861730    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b-tmp-dir\") pod \"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b\" (UID: \"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b\") "
Sep 09 20:49:29.861900 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.861756    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b-tls-certs\") pod \"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b\" (UID: \"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b\") "
Sep 09 20:49:29.861900 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.861806    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b-cabundle-cert\") pod \"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b\" (UID: \"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b\") "
Sep 09 20:49:29.862190 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.862154    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b" (UID: "4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:29.862296 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.862208    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b" (UID: "4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:49:29.863884 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.863863    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b" (UID: "4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:49:29.883933 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.883908    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b" (UID: "4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:29.962442 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.962361    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:49:29.962442 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.962391    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b-cabundle-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:49:29.962442 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.962401    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b-kserve-provision-location\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:49:29.962442 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:29.962409    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:49:30.017763 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.017728    2581 generic.go:356] "Generic (PLEG): container finished" podID="4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b" containerID="3b1c6458dd0ec04bc03b3b565e15ebf137f5ea6d44dcf190d1ddbd9bf60e586d" exitCode=0
Sep 09 20:49:30.017929 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.017781    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4"
Sep 09 20:49:30.017929 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.017776    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4" event={"ID":"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b","Type":"ContainerDied","Data":"3b1c6458dd0ec04bc03b3b565e15ebf137f5ea6d44dcf190d1ddbd9bf60e586d"}
Sep 09 20:49:30.017929 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.017901    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4" event={"ID":"4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b","Type":"ContainerDied","Data":"6f04c3e5b5e1de2850e3cacc0e777a824cec2486b93db3f145b0faef39afdcc4"}
Sep 09 20:49:30.018054 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.017930    2581 scope.go:117] "RemoveContainer" containerID="3b1c6458dd0ec04bc03b3b565e15ebf137f5ea6d44dcf190d1ddbd9bf60e586d"
Sep 09 20:49:30.019610 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.019585    2581 generic.go:356] "Generic (PLEG): container finished" podID="ed11f1f5-64b3-49c8-be6b-44e45516cca9" containerID="f0e4dd36f8e931d6a76e086e5d11712f8db112e9c3b86229c23b267ea3036ccb" exitCode=0
Sep 09 20:49:30.019712 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.019636    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"
Sep 09 20:49:30.019712 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.019652    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk" event={"ID":"ed11f1f5-64b3-49c8-be6b-44e45516cca9","Type":"ContainerDied","Data":"f0e4dd36f8e931d6a76e086e5d11712f8db112e9c3b86229c23b267ea3036ccb"}
Sep 09 20:49:30.019712 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.019696    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk" event={"ID":"ed11f1f5-64b3-49c8-be6b-44e45516cca9","Type":"ContainerDied","Data":"83b64be493b995ec7c21a4e703d2ff703c15764e2a82207f59d19e66bc24a660"}
Sep 09 20:49:30.027730 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.027701    2581 scope.go:117] "RemoveContainer" containerID="bdb3bd4f55996b958df8779a128fbe80d6468f6ec2c14939a4bc20c8205abec2"
Sep 09 20:49:30.039216 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.039191    2581 scope.go:117] "RemoveContainer" containerID="3b1c6458dd0ec04bc03b3b565e15ebf137f5ea6d44dcf190d1ddbd9bf60e586d"
Sep 09 20:49:30.039494 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:49:30.039474    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b1c6458dd0ec04bc03b3b565e15ebf137f5ea6d44dcf190d1ddbd9bf60e586d\": container with ID starting with 3b1c6458dd0ec04bc03b3b565e15ebf137f5ea6d44dcf190d1ddbd9bf60e586d not found: ID does not exist" containerID="3b1c6458dd0ec04bc03b3b565e15ebf137f5ea6d44dcf190d1ddbd9bf60e586d"
Sep 09 20:49:30.039558 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.039503    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b1c6458dd0ec04bc03b3b565e15ebf137f5ea6d44dcf190d1ddbd9bf60e586d"} err="failed to get container status \"3b1c6458dd0ec04bc03b3b565e15ebf137f5ea6d44dcf190d1ddbd9bf60e586d\": rpc error: code = NotFound desc = could not find container \"3b1c6458dd0ec04bc03b3b565e15ebf137f5ea6d44dcf190d1ddbd9bf60e586d\": container with ID starting with 3b1c6458dd0ec04bc03b3b565e15ebf137f5ea6d44dcf190d1ddbd9bf60e586d not found: ID does not exist"
Sep 09 20:49:30.039558 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.039518    2581 scope.go:117] "RemoveContainer" containerID="bdb3bd4f55996b958df8779a128fbe80d6468f6ec2c14939a4bc20c8205abec2"
Sep 09 20:49:30.039765 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:49:30.039744    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdb3bd4f55996b958df8779a128fbe80d6468f6ec2c14939a4bc20c8205abec2\": container with ID starting with bdb3bd4f55996b958df8779a128fbe80d6468f6ec2c14939a4bc20c8205abec2 not found: ID does not exist" containerID="bdb3bd4f55996b958df8779a128fbe80d6468f6ec2c14939a4bc20c8205abec2"
Sep 09 20:49:30.039813 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.039772    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb3bd4f55996b958df8779a128fbe80d6468f6ec2c14939a4bc20c8205abec2"} err="failed to get container status \"bdb3bd4f55996b958df8779a128fbe80d6468f6ec2c14939a4bc20c8205abec2\": rpc error: code = NotFound desc = could not find container \"bdb3bd4f55996b958df8779a128fbe80d6468f6ec2c14939a4bc20c8205abec2\": container with ID starting with bdb3bd4f55996b958df8779a128fbe80d6468f6ec2c14939a4bc20c8205abec2 not found: ID does not exist"
Sep 09 20:49:30.039813 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.039787    2581 scope.go:117] "RemoveContainer" containerID="f0e4dd36f8e931d6a76e086e5d11712f8db112e9c3b86229c23b267ea3036ccb"
Sep 09 20:49:30.046542 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.046520    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4"]
Sep 09 20:49:30.048751 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.048605    2581 scope.go:117] "RemoveContainer" containerID="dc23c450bf9613815c1f517b1953c37aeee0eaeaf58ffcbdb95aa327f7e561dc"
Sep 09 20:49:30.050748 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.050727    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-ddc485f5c-m5ct4"]
Sep 09 20:49:30.056658 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.056643    2581 scope.go:117] "RemoveContainer" containerID="f0e4dd36f8e931d6a76e086e5d11712f8db112e9c3b86229c23b267ea3036ccb"
Sep 09 20:49:30.056943 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:49:30.056925    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0e4dd36f8e931d6a76e086e5d11712f8db112e9c3b86229c23b267ea3036ccb\": container with ID starting with f0e4dd36f8e931d6a76e086e5d11712f8db112e9c3b86229c23b267ea3036ccb not found: ID does not exist" containerID="f0e4dd36f8e931d6a76e086e5d11712f8db112e9c3b86229c23b267ea3036ccb"
Sep 09 20:49:30.056997 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.056952    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e4dd36f8e931d6a76e086e5d11712f8db112e9c3b86229c23b267ea3036ccb"} err="failed to get container status \"f0e4dd36f8e931d6a76e086e5d11712f8db112e9c3b86229c23b267ea3036ccb\": rpc error: code = NotFound desc = could not find container \"f0e4dd36f8e931d6a76e086e5d11712f8db112e9c3b86229c23b267ea3036ccb\": container with ID starting with f0e4dd36f8e931d6a76e086e5d11712f8db112e9c3b86229c23b267ea3036ccb not found: ID does not exist"
Sep 09 20:49:30.056997 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.056968    2581 scope.go:117] "RemoveContainer" containerID="dc23c450bf9613815c1f517b1953c37aeee0eaeaf58ffcbdb95aa327f7e561dc"
Sep 09 20:49:30.057206 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:49:30.057193    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc23c450bf9613815c1f517b1953c37aeee0eaeaf58ffcbdb95aa327f7e561dc\": container with ID starting with dc23c450bf9613815c1f517b1953c37aeee0eaeaf58ffcbdb95aa327f7e561dc not found: ID does not exist" containerID="dc23c450bf9613815c1f517b1953c37aeee0eaeaf58ffcbdb95aa327f7e561dc"
Sep 09 20:49:30.057246 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.057209    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc23c450bf9613815c1f517b1953c37aeee0eaeaf58ffcbdb95aa327f7e561dc"} err="failed to get container status \"dc23c450bf9613815c1f517b1953c37aeee0eaeaf58ffcbdb95aa327f7e561dc\": rpc error: code = NotFound desc = could not find container \"dc23c450bf9613815c1f517b1953c37aeee0eaeaf58ffcbdb95aa327f7e561dc\": container with ID starting with dc23c450bf9613815c1f517b1953c37aeee0eaeaf58ffcbdb95aa327f7e561dc not found: ID does not exist"
Sep 09 20:49:30.063479 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.063447    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"]
Sep 09 20:49:30.067616 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.067597    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-67drk"]
Sep 09 20:49:30.634908 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.634876    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b" path="/var/lib/kubelet/pods/4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b/volumes"
Sep 09 20:49:30.635351 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:49:30.635337    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed11f1f5-64b3-49c8-be6b-44e45516cca9" path="/var/lib/kubelet/pods/ed11f1f5-64b3-49c8-be6b-44e45516cca9/volumes"
Sep 09 20:50:58.054352 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.054320    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"]
Sep 09 20:50:58.054835 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.054664    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed11f1f5-64b3-49c8-be6b-44e45516cca9" containerName="main"
Sep 09 20:50:58.054835 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.054677    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed11f1f5-64b3-49c8-be6b-44e45516cca9" containerName="main"
Sep 09 20:50:58.054835 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.054686    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b" containerName="main"
Sep 09 20:50:58.054835 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.054693    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b" containerName="main"
Sep 09 20:50:58.054835 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.054702    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b" containerName="storage-initializer"
Sep 09 20:50:58.054835 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.054708    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b" containerName="storage-initializer"
Sep 09 20:50:58.054835 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.054722    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed11f1f5-64b3-49c8-be6b-44e45516cca9" containerName="storage-initializer"
Sep 09 20:50:58.054835 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.054727    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed11f1f5-64b3-49c8-be6b-44e45516cca9" containerName="storage-initializer"
Sep 09 20:50:58.054835 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.054780    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed11f1f5-64b3-49c8-be6b-44e45516cca9" containerName="main"
Sep 09 20:50:58.054835 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.054789    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e5bf75d-ce1e-4e3f-b3e2-f6c49979545b" containerName="main"
Sep 09 20:50:58.057934 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.057914    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.060365 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.060343    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:50:58.060490 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.060436    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 09 20:50:58.060490 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.060443    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-k776l\""
Sep 09 20:50:58.061538 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.061515    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 09 20:50:58.061538 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.061529    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 09 20:50:58.061668 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.061527    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 09 20:50:58.070312 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.070286    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"]
Sep 09 20:50:58.079367 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.079339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-dshm\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-dncbj\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.079451 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.079410    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-model-cache\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-dncbj\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.079518 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.079467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-tls-certs\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-dncbj\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.079571 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.079534    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-dncbj\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.079615 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.079572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svzkk\" (UniqueName: \"kubernetes.io/projected/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-kube-api-access-svzkk\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-dncbj\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.079615 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.079606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-tmp-dir\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-dncbj\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.079713 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.079641    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-home\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-dncbj\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.079713 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.079677    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-dncbj\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.180273 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.180243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-tls-certs\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-dncbj\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.180437 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.180288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-dncbj\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.180437 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.180309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svzkk\" (UniqueName: \"kubernetes.io/projected/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-kube-api-access-svzkk\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-dncbj\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.180437 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.180422    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-tmp-dir\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-dncbj\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.180661 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.180487    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-home\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-dncbj\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.180661 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.180527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-dncbj\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.180661 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.180571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-dshm\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-dncbj\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.180661 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.180612    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-model-cache\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-dncbj\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.180876 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.180846    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-tmp-dir\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-dncbj\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.180935 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.180892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-dncbj\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.180998 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.180957    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-home\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-dncbj\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.180998 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.180984    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-model-cache\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-dncbj\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.181162 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.181141    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-dncbj\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.182783 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.182756    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-dshm\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-dncbj\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.182878 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.182822    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-tls-certs\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-dncbj\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.189783 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.189765    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svzkk\" (UniqueName: \"kubernetes.io/projected/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-kube-api-access-svzkk\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-dncbj\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.275999 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.275969    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg"]
Sep 09 20:50:58.280118 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.280094    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg"
Sep 09 20:50:58.290084 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.290014    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg"]
Sep 09 20:50:58.369211 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.369136    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:50:58.382780 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.382747    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4265eaa5-c13d-4241-9f2c-81e678b34570-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg\" (UID: \"4265eaa5-c13d-4241-9f2c-81e678b34570\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg"
Sep 09 20:50:58.382908 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.382792    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4265eaa5-c13d-4241-9f2c-81e678b34570-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg\" (UID: \"4265eaa5-c13d-4241-9f2c-81e678b34570\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg"
Sep 09 20:50:58.382992 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.382937    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4265eaa5-c13d-4241-9f2c-81e678b34570-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg\" (UID: \"4265eaa5-c13d-4241-9f2c-81e678b34570\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg"
Sep 09 20:50:58.383062 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.383001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4265eaa5-c13d-4241-9f2c-81e678b34570-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg\" (UID: \"4265eaa5-c13d-4241-9f2c-81e678b34570\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg"
Sep 09 20:50:58.484368 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.484334    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4265eaa5-c13d-4241-9f2c-81e678b34570-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg\" (UID: \"4265eaa5-c13d-4241-9f2c-81e678b34570\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg"
Sep 09 20:50:58.484564 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.484390    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4265eaa5-c13d-4241-9f2c-81e678b34570-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg\" (UID: \"4265eaa5-c13d-4241-9f2c-81e678b34570\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg"
Sep 09 20:50:58.484564 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.484446    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4265eaa5-c13d-4241-9f2c-81e678b34570-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg\" (UID: \"4265eaa5-c13d-4241-9f2c-81e678b34570\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg"
Sep 09 20:50:58.484564 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.484488    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4265eaa5-c13d-4241-9f2c-81e678b34570-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg\" (UID: \"4265eaa5-c13d-4241-9f2c-81e678b34570\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg"
Sep 09 20:50:58.484830 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.484805    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4265eaa5-c13d-4241-9f2c-81e678b34570-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg\" (UID: \"4265eaa5-c13d-4241-9f2c-81e678b34570\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg"
Sep 09 20:50:58.484903 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.484876    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4265eaa5-c13d-4241-9f2c-81e678b34570-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg\" (UID: \"4265eaa5-c13d-4241-9f2c-81e678b34570\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg"
Sep 09 20:50:58.485124 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.485098    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4265eaa5-c13d-4241-9f2c-81e678b34570-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg\" (UID: \"4265eaa5-c13d-4241-9f2c-81e678b34570\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg"
Sep 09 20:50:58.487030 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.487009    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4265eaa5-c13d-4241-9f2c-81e678b34570-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg\" (UID: \"4265eaa5-c13d-4241-9f2c-81e678b34570\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg"
Sep 09 20:50:58.592773 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.592734    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg"
Sep 09 20:50:58.638627 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.638556    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 20:50:58.641629 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.641594    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 20:50:58.644880 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.644856    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 20:50:58.647347 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.647320    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"]
Sep 09 20:50:58.663754 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.663690    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 20:50:58.667518 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:50:58.667489    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5b27ca_edb7_4cb6_86f4_9f75bacfaf71.slice/crio-2d5d2ebe086acfb06ae385e9510229d5289dc11b0af1eda8cfbfcff5574ecf93 WatchSource:0}: Error finding container 2d5d2ebe086acfb06ae385e9510229d5289dc11b0af1eda8cfbfcff5574ecf93: Status 404 returned error can't find the container with id 2d5d2ebe086acfb06ae385e9510229d5289dc11b0af1eda8cfbfcff5574ecf93
Sep 09 20:50:58.671767 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.671713    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 09 20:50:58.875840 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:58.875813    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg"]
Sep 09 20:50:58.877315 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:50:58.877288    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4265eaa5_c13d_4241_9f2c_81e678b34570.slice/crio-712cd217b7d853a21917c1fceacda9419e6fa974309d492c5b0ef0ed1b5b8725 WatchSource:0}: Error finding container 712cd217b7d853a21917c1fceacda9419e6fa974309d492c5b0ef0ed1b5b8725: Status 404 returned error can't find the container with id 712cd217b7d853a21917c1fceacda9419e6fa974309d492c5b0ef0ed1b5b8725
Sep 09 20:50:59.461661 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:59.461399    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg" event={"ID":"4265eaa5-c13d-4241-9f2c-81e678b34570","Type":"ContainerStarted","Data":"3c6462abcba436751b08cc35dd9acd06dc1a188dfbbf2c33c431e77b11b670ca"}
Sep 09 20:50:59.461661 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:59.461492    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg" event={"ID":"4265eaa5-c13d-4241-9f2c-81e678b34570","Type":"ContainerStarted","Data":"712cd217b7d853a21917c1fceacda9419e6fa974309d492c5b0ef0ed1b5b8725"}
Sep 09 20:50:59.463250 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:59.463223    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj" event={"ID":"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71","Type":"ContainerStarted","Data":"6753294d83c5bfdf58161951dbc8c19efff64ed2e7d6d4268e8e4c3a05ac4d9b"}
Sep 09 20:50:59.463393 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:50:59.463273    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj" event={"ID":"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71","Type":"ContainerStarted","Data":"2d5d2ebe086acfb06ae385e9510229d5289dc11b0af1eda8cfbfcff5574ecf93"}
Sep 09 20:51:01.476629 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:01.476589    2581 generic.go:356] "Generic (PLEG): container finished" podID="fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71" containerID="6753294d83c5bfdf58161951dbc8c19efff64ed2e7d6d4268e8e4c3a05ac4d9b" exitCode=0
Sep 09 20:51:01.477066 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:01.476648    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj" event={"ID":"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71","Type":"ContainerDied","Data":"6753294d83c5bfdf58161951dbc8c19efff64ed2e7d6d4268e8e4c3a05ac4d9b"}
Sep 09 20:51:02.483265 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:02.483230    2581 generic.go:356] "Generic (PLEG): container finished" podID="4265eaa5-c13d-4241-9f2c-81e678b34570" containerID="3c6462abcba436751b08cc35dd9acd06dc1a188dfbbf2c33c431e77b11b670ca" exitCode=0
Sep 09 20:51:02.483768 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:02.483308    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg" event={"ID":"4265eaa5-c13d-4241-9f2c-81e678b34570","Type":"ContainerDied","Data":"3c6462abcba436751b08cc35dd9acd06dc1a188dfbbf2c33c431e77b11b670ca"}
Sep 09 20:51:02.485392 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:02.485370    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj" event={"ID":"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71","Type":"ContainerStarted","Data":"4e1fe2e56b57c971de79d0d8d657b071852193fe20809098a9279920af37fac2"}
Sep 09 20:51:02.517781 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:02.517729    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj" podStartSLOduration=1.839019075 podStartE2EDuration="4.517711014s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.678691939s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:51:02.516605947 +0000 UTC m=+2404.521782943" watchObservedRunningTime="2026-09-09 20:51:02.517711014 +0000 UTC m=+2404.522888008"
Sep 09 20:51:03.370061 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:03.370023    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:51:03.491883 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:03.491846    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg" event={"ID":"4265eaa5-c13d-4241-9f2c-81e678b34570","Type":"ContainerStarted","Data":"ff306cdf1c6825ac65c872c300628caae23be7ab180d8e82c4905a8f85511f31"}
Sep 09 20:51:03.518340 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:03.518286    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg" podStartSLOduration=2.851086598 podStartE2EDuration="5.518272631s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.667186033s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:51:03.515787547 +0000 UTC m=+2405.520964546" watchObservedRunningTime="2026-09-09 20:51:03.518272631 +0000 UTC m=+2405.523449625"
Sep 09 20:51:08.370001 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:08.369958    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:51:08.382424 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:08.382388    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:51:08.530305 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:08.530275    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:51:13.492895 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:13.492854    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg"
Sep 09 20:51:33.504926 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:33.504841    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg"
Sep 09 20:51:37.559302 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:37.559267    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg_4265eaa5-c13d-4241-9f2c-81e678b34570/main/0.log"
Sep 09 20:51:53.772398 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:53.772364    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"]
Sep 09 20:51:53.778954 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:53.778926    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg"]
Sep 09 20:51:53.779235 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:53.779166    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg" podUID="4265eaa5-c13d-4241-9f2c-81e678b34570" containerName="main" containerID="cri-o://ff306cdf1c6825ac65c872c300628caae23be7ab180d8e82c4905a8f85511f31" gracePeriod=30
Sep 09 20:51:53.782127 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:51:53.782089    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:51:53.782127 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:51:53.782127 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:51:53.782127 ip-10-0-134-84 kubenswrapper[2581]:  > containerID="4e1fe2e56b57c971de79d0d8d657b071852193fe20809098a9279920af37fac2" cmd=["/bin/sleep","15"]
Sep 09 20:51:53.782386 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:53.782144    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj" message="" err=<
Sep 09 20:51:53.782386 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:51:53.782386 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:51:53.782386 ip-10-0-134-84 kubenswrapper[2581]:  >
Sep 09 20:51:53.782386 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:51:53.782161    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:51:53.782386 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:51:53.782386 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 20:51:53.782386 ip-10-0-134-84 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj" podUID="fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71" containerName="main" containerID="cri-o://4e1fe2e56b57c971de79d0d8d657b071852193fe20809098a9279920af37fac2"
Sep 09 20:51:53.782386 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:53.782213    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj" podUID="fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71" containerName="main" containerID="cri-o://4e1fe2e56b57c971de79d0d8d657b071852193fe20809098a9279920af37fac2" gracePeriod=60
Sep 09 20:51:54.133335 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.133308    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:51:54.170440 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.170408    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-kserve-provision-location\") pod \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") "
Sep 09 20:51:54.170655 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.170476    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-dshm\") pod \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") "
Sep 09 20:51:54.170655 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.170504    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-tls-certs\") pod \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") "
Sep 09 20:51:54.170655 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.170533    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-home\") pod \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") "
Sep 09 20:51:54.170655 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.170563    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-cabundle-cert\") pod \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") "
Sep 09 20:51:54.170655 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.170603    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svzkk\" (UniqueName: \"kubernetes.io/projected/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-kube-api-access-svzkk\") pod \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") "
Sep 09 20:51:54.170655 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.170637    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-tmp-dir\") pod \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") "
Sep 09 20:51:54.170964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.170670    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-model-cache\") pod \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\" (UID: \"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71\") "
Sep 09 20:51:54.170964 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.170879    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71" (UID: "fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:51:54.171421 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.171358    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71" (UID: "fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:51:54.171570 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.171426    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71" (UID: "fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:51:54.171672 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.171651    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-home" (OuterVolumeSpecName: "home") pod "fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71" (UID: "fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:51:54.173288 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.173253    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-kube-api-access-svzkk" (OuterVolumeSpecName: "kube-api-access-svzkk") pod "fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71" (UID: "fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71"). InnerVolumeSpecName "kube-api-access-svzkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:51:54.173409 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.173330    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71" (UID: "fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:51:54.173409 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.173371    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-dshm" (OuterVolumeSpecName: "dshm") pod "fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71" (UID: "fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:51:54.204925 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.204874    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71" (UID: "fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:51:54.271703 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.271669    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-kserve-provision-location\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:51:54.271703 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.271701    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:51:54.271878 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.271714    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:51:54.271878 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.271725    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:51:54.271878 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.271736    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-cabundle-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:51:54.271878 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.271746    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svzkk\" (UniqueName: \"kubernetes.io/projected/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-kube-api-access-svzkk\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:51:54.271878 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.271757    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:51:54.271878 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.271770    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:51:54.419077 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.419048    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg"
Sep 09 20:51:54.472497 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.472444    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4265eaa5-c13d-4241-9f2c-81e678b34570-tls-certs\") pod \"4265eaa5-c13d-4241-9f2c-81e678b34570\" (UID: \"4265eaa5-c13d-4241-9f2c-81e678b34570\") "
Sep 09 20:51:54.472674 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.472583    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4265eaa5-c13d-4241-9f2c-81e678b34570-cabundle-cert\") pod \"4265eaa5-c13d-4241-9f2c-81e678b34570\" (UID: \"4265eaa5-c13d-4241-9f2c-81e678b34570\") "
Sep 09 20:51:54.472674 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.472623    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4265eaa5-c13d-4241-9f2c-81e678b34570-tmp-dir\") pod \"4265eaa5-c13d-4241-9f2c-81e678b34570\" (UID: \"4265eaa5-c13d-4241-9f2c-81e678b34570\") "
Sep 09 20:51:54.472674 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.472664    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4265eaa5-c13d-4241-9f2c-81e678b34570-kserve-provision-location\") pod \"4265eaa5-c13d-4241-9f2c-81e678b34570\" (UID: \"4265eaa5-c13d-4241-9f2c-81e678b34570\") "
Sep 09 20:51:54.472995 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.472947    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4265eaa5-c13d-4241-9f2c-81e678b34570-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4265eaa5-c13d-4241-9f2c-81e678b34570" (UID: "4265eaa5-c13d-4241-9f2c-81e678b34570"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:51:54.473125 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.473031    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4265eaa5-c13d-4241-9f2c-81e678b34570-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4265eaa5-c13d-4241-9f2c-81e678b34570" (UID: "4265eaa5-c13d-4241-9f2c-81e678b34570"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:51:54.475179 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.475149    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4265eaa5-c13d-4241-9f2c-81e678b34570-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4265eaa5-c13d-4241-9f2c-81e678b34570" (UID: "4265eaa5-c13d-4241-9f2c-81e678b34570"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:51:54.495547 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.495504    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4265eaa5-c13d-4241-9f2c-81e678b34570-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4265eaa5-c13d-4241-9f2c-81e678b34570" (UID: "4265eaa5-c13d-4241-9f2c-81e678b34570"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:51:54.573536 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.573493    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4265eaa5-c13d-4241-9f2c-81e678b34570-cabundle-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:51:54.573536 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.573535    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4265eaa5-c13d-4241-9f2c-81e678b34570-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:51:54.573536 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.573550    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4265eaa5-c13d-4241-9f2c-81e678b34570-kserve-provision-location\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:51:54.573854 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.573562    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4265eaa5-c13d-4241-9f2c-81e678b34570-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:51:54.676598 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:51:54.676554    2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5b27ca_edb7_4cb6_86f4_9f75bacfaf71.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4265eaa5_c13d_4241_9f2c_81e678b34570.slice\": RecentStats: unable to find data in memory cache]"
Sep 09 20:51:54.758185 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.758149    2581 generic.go:356] "Generic (PLEG): container finished" podID="4265eaa5-c13d-4241-9f2c-81e678b34570" containerID="ff306cdf1c6825ac65c872c300628caae23be7ab180d8e82c4905a8f85511f31" exitCode=0
Sep 09 20:51:54.758382 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.758227    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg"
Sep 09 20:51:54.758382 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.758226    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg" event={"ID":"4265eaa5-c13d-4241-9f2c-81e678b34570","Type":"ContainerDied","Data":"ff306cdf1c6825ac65c872c300628caae23be7ab180d8e82c4905a8f85511f31"}
Sep 09 20:51:54.758382 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.758323    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg" event={"ID":"4265eaa5-c13d-4241-9f2c-81e678b34570","Type":"ContainerDied","Data":"712cd217b7d853a21917c1fceacda9419e6fa974309d492c5b0ef0ed1b5b8725"}
Sep 09 20:51:54.758382 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.758354    2581 scope.go:117] "RemoveContainer" containerID="ff306cdf1c6825ac65c872c300628caae23be7ab180d8e82c4905a8f85511f31"
Sep 09 20:51:54.760111 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.760091    2581 generic.go:356] "Generic (PLEG): container finished" podID="fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71" containerID="4e1fe2e56b57c971de79d0d8d657b071852193fe20809098a9279920af37fac2" exitCode=0
Sep 09 20:51:54.760229 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.760153    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"
Sep 09 20:51:54.760229 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.760160    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj" event={"ID":"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71","Type":"ContainerDied","Data":"4e1fe2e56b57c971de79d0d8d657b071852193fe20809098a9279920af37fac2"}
Sep 09 20:51:54.760229 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.760204    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj" event={"ID":"fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71","Type":"ContainerDied","Data":"2d5d2ebe086acfb06ae385e9510229d5289dc11b0af1eda8cfbfcff5574ecf93"}
Sep 09 20:51:54.767405 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.767387    2581 scope.go:117] "RemoveContainer" containerID="3c6462abcba436751b08cc35dd9acd06dc1a188dfbbf2c33c431e77b11b670ca"
Sep 09 20:51:54.775419 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.775403    2581 scope.go:117] "RemoveContainer" containerID="ff306cdf1c6825ac65c872c300628caae23be7ab180d8e82c4905a8f85511f31"
Sep 09 20:51:54.775764 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:51:54.775709    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff306cdf1c6825ac65c872c300628caae23be7ab180d8e82c4905a8f85511f31\": container with ID starting with ff306cdf1c6825ac65c872c300628caae23be7ab180d8e82c4905a8f85511f31 not found: ID does not exist" containerID="ff306cdf1c6825ac65c872c300628caae23be7ab180d8e82c4905a8f85511f31"
Sep 09 20:51:54.775764 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.775733    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff306cdf1c6825ac65c872c300628caae23be7ab180d8e82c4905a8f85511f31"} err="failed to get container status \"ff306cdf1c6825ac65c872c300628caae23be7ab180d8e82c4905a8f85511f31\": rpc error: code = NotFound desc = could not find container \"ff306cdf1c6825ac65c872c300628caae23be7ab180d8e82c4905a8f85511f31\": container with ID starting with ff306cdf1c6825ac65c872c300628caae23be7ab180d8e82c4905a8f85511f31 not found: ID does not exist"
Sep 09 20:51:54.775764 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.775748    2581 scope.go:117] "RemoveContainer" containerID="3c6462abcba436751b08cc35dd9acd06dc1a188dfbbf2c33c431e77b11b670ca"
Sep 09 20:51:54.776020 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:51:54.776002    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c6462abcba436751b08cc35dd9acd06dc1a188dfbbf2c33c431e77b11b670ca\": container with ID starting with 3c6462abcba436751b08cc35dd9acd06dc1a188dfbbf2c33c431e77b11b670ca not found: ID does not exist" containerID="3c6462abcba436751b08cc35dd9acd06dc1a188dfbbf2c33c431e77b11b670ca"
Sep 09 20:51:54.776068 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.776025    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c6462abcba436751b08cc35dd9acd06dc1a188dfbbf2c33c431e77b11b670ca"} err="failed to get container status \"3c6462abcba436751b08cc35dd9acd06dc1a188dfbbf2c33c431e77b11b670ca\": rpc error: code = NotFound desc = could not find container \"3c6462abcba436751b08cc35dd9acd06dc1a188dfbbf2c33c431e77b11b670ca\": container with ID starting with 3c6462abcba436751b08cc35dd9acd06dc1a188dfbbf2c33c431e77b11b670ca not found: ID does not exist"
Sep 09 20:51:54.776068 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.776041    2581 scope.go:117] "RemoveContainer" containerID="4e1fe2e56b57c971de79d0d8d657b071852193fe20809098a9279920af37fac2"
Sep 09 20:51:54.779501 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.779483    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg"]
Sep 09 20:51:54.785084 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.784906    2581 scope.go:117] "RemoveContainer" containerID="6753294d83c5bfdf58161951dbc8c19efff64ed2e7d6d4268e8e4c3a05ac4d9b"
Sep 09 20:51:54.786116 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.786098    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c96cd4fdf-8pmkg"]
Sep 09 20:51:54.792816 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.792800    2581 scope.go:117] "RemoveContainer" containerID="4e1fe2e56b57c971de79d0d8d657b071852193fe20809098a9279920af37fac2"
Sep 09 20:51:54.793080 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:51:54.793063    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e1fe2e56b57c971de79d0d8d657b071852193fe20809098a9279920af37fac2\": container with ID starting with 4e1fe2e56b57c971de79d0d8d657b071852193fe20809098a9279920af37fac2 not found: ID does not exist" containerID="4e1fe2e56b57c971de79d0d8d657b071852193fe20809098a9279920af37fac2"
Sep 09 20:51:54.793124 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.793086    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e1fe2e56b57c971de79d0d8d657b071852193fe20809098a9279920af37fac2"} err="failed to get container status \"4e1fe2e56b57c971de79d0d8d657b071852193fe20809098a9279920af37fac2\": rpc error: code = NotFound desc = could not find container \"4e1fe2e56b57c971de79d0d8d657b071852193fe20809098a9279920af37fac2\": container with ID starting with 4e1fe2e56b57c971de79d0d8d657b071852193fe20809098a9279920af37fac2 not found: ID does not exist"
Sep 09 20:51:54.793124 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.793100    2581 scope.go:117] "RemoveContainer" containerID="6753294d83c5bfdf58161951dbc8c19efff64ed2e7d6d4268e8e4c3a05ac4d9b"
Sep 09 20:51:54.793320 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:51:54.793303    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6753294d83c5bfdf58161951dbc8c19efff64ed2e7d6d4268e8e4c3a05ac4d9b\": container with ID starting with 6753294d83c5bfdf58161951dbc8c19efff64ed2e7d6d4268e8e4c3a05ac4d9b not found: ID does not exist" containerID="6753294d83c5bfdf58161951dbc8c19efff64ed2e7d6d4268e8e4c3a05ac4d9b"
Sep 09 20:51:54.793361 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.793327    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6753294d83c5bfdf58161951dbc8c19efff64ed2e7d6d4268e8e4c3a05ac4d9b"} err="failed to get container status \"6753294d83c5bfdf58161951dbc8c19efff64ed2e7d6d4268e8e4c3a05ac4d9b\": rpc error: code = NotFound desc = could not find container \"6753294d83c5bfdf58161951dbc8c19efff64ed2e7d6d4268e8e4c3a05ac4d9b\": container with ID starting with 6753294d83c5bfdf58161951dbc8c19efff64ed2e7d6d4268e8e4c3a05ac4d9b not found: ID does not exist"
Sep 09 20:51:54.798002 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.797983    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"]
Sep 09 20:51:54.803387 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:54.803369    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-dncbj"]
Sep 09 20:51:56.635673 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:56.635642    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4265eaa5-c13d-4241-9f2c-81e678b34570" path="/var/lib/kubelet/pods/4265eaa5-c13d-4241-9f2c-81e678b34570/volumes"
Sep 09 20:51:56.636108 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:51:56.636095    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71" path="/var/lib/kubelet/pods/fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71/volumes"
Sep 09 20:53:06.414865 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.414832    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg"]
Sep 09 20:53:06.415332 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.415154    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4265eaa5-c13d-4241-9f2c-81e678b34570" containerName="storage-initializer"
Sep 09 20:53:06.415332 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.415165    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4265eaa5-c13d-4241-9f2c-81e678b34570" containerName="storage-initializer"
Sep 09 20:53:06.415332 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.415176    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71" containerName="main"
Sep 09 20:53:06.415332 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.415181    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71" containerName="main"
Sep 09 20:53:06.415332 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.415195    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71" containerName="storage-initializer"
Sep 09 20:53:06.415332 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.415201    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71" containerName="storage-initializer"
Sep 09 20:53:06.415332 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.415212    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4265eaa5-c13d-4241-9f2c-81e678b34570" containerName="main"
Sep 09 20:53:06.415332 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.415217    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4265eaa5-c13d-4241-9f2c-81e678b34570" containerName="main"
Sep 09 20:53:06.415332 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.415272    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4265eaa5-c13d-4241-9f2c-81e678b34570" containerName="main"
Sep 09 20:53:06.415332 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.415282    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe5b27ca-edb7-4cb6-86f4-9f75bacfaf71" containerName="main"
Sep 09 20:53:06.418044 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.418025    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg"
Sep 09 20:53:06.420432 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.420413    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 09 20:53:06.421903 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.421716    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-dv7j8\""
Sep 09 20:53:06.422163 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.422137    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-lvpmw\""
Sep 09 20:53:06.422289 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.422248    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 09 20:53:06.422683 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.421881    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 09 20:53:06.435920 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.435896    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg"]
Sep 09 20:53:06.454242 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.454211    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/996caf85-035a-4cdc-a11e-e5062e1eafc1-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg\" (UID: \"996caf85-035a-4cdc-a11e-e5062e1eafc1\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg"
Sep 09 20:53:06.454376 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.454244    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s47fh\" (UniqueName: \"kubernetes.io/projected/996caf85-035a-4cdc-a11e-e5062e1eafc1-kube-api-access-s47fh\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg\" (UID: \"996caf85-035a-4cdc-a11e-e5062e1eafc1\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg"
Sep 09 20:53:06.555662 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.555622    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/996caf85-035a-4cdc-a11e-e5062e1eafc1-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg\" (UID: \"996caf85-035a-4cdc-a11e-e5062e1eafc1\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg"
Sep 09 20:53:06.555662 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.555663    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s47fh\" (UniqueName: \"kubernetes.io/projected/996caf85-035a-4cdc-a11e-e5062e1eafc1-kube-api-access-s47fh\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg\" (UID: \"996caf85-035a-4cdc-a11e-e5062e1eafc1\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg"
Sep 09 20:53:06.558215 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.558189    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/996caf85-035a-4cdc-a11e-e5062e1eafc1-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg\" (UID: \"996caf85-035a-4cdc-a11e-e5062e1eafc1\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg"
Sep 09 20:53:06.565448 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.565422    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s47fh\" (UniqueName: \"kubernetes.io/projected/996caf85-035a-4cdc-a11e-e5062e1eafc1-kube-api-access-s47fh\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg\" (UID: \"996caf85-035a-4cdc-a11e-e5062e1eafc1\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg"
Sep 09 20:53:06.734891 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:06.734857    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg"
Sep 09 20:53:07.007653 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:07.007611    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg"]
Sep 09 20:53:07.009861 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:53:07.009834    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996caf85_035a_4cdc_a11e_e5062e1eafc1.slice/crio-190fe8a0b203a3dcbb31dd3a6b9ecfe6b91e7da8d936f578a721aef22cb04260 WatchSource:0}: Error finding container 190fe8a0b203a3dcbb31dd3a6b9ecfe6b91e7da8d936f578a721aef22cb04260: Status 404 returned error can't find the container with id 190fe8a0b203a3dcbb31dd3a6b9ecfe6b91e7da8d936f578a721aef22cb04260
Sep 09 20:53:07.121107 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:07.120851    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg" event={"ID":"996caf85-035a-4cdc-a11e-e5062e1eafc1","Type":"ContainerStarted","Data":"2762fbf15f65973061bc6023270f7f16bd5da7003d7ff1d6c1440ff386aa3a8d"}
Sep 09 20:53:07.121107 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:07.120918    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg"
Sep 09 20:53:07.121107 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:07.120943    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg" event={"ID":"996caf85-035a-4cdc-a11e-e5062e1eafc1","Type":"ContainerStarted","Data":"190fe8a0b203a3dcbb31dd3a6b9ecfe6b91e7da8d936f578a721aef22cb04260"}
Sep 09 20:53:07.138827 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:07.138711    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg" podStartSLOduration=1.13869114 podStartE2EDuration="1.13869114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:07.138170678 +0000 UTC m=+2529.143347690" watchObservedRunningTime="2026-09-09 20:53:07.13869114 +0000 UTC m=+2529.143868136"
Sep 09 20:53:38.129156 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:38.129123    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg"
Sep 09 20:53:39.735443 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:39.735410    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg"]
Sep 09 20:53:48.128447 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:48.128408    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg" podUID="996caf85-035a-4cdc-a11e-e5062e1eafc1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:53:54.754934 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:54.754887    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg" podUID="996caf85-035a-4cdc-a11e-e5062e1eafc1" containerName="main" containerID="cri-o://2762fbf15f65973061bc6023270f7f16bd5da7003d7ff1d6c1440ff386aa3a8d" gracePeriod=45
Sep 09 20:53:58.128583 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:53:58.128549    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg" podUID="996caf85-035a-4cdc-a11e-e5062e1eafc1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:54:08.128731 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:08.128685    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg" podUID="996caf85-035a-4cdc-a11e-e5062e1eafc1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:54:08.129187 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:08.128815    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg"
Sep 09 20:54:18.128778 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:18.128735    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg" podUID="996caf85-035a-4cdc-a11e-e5062e1eafc1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:54:24.994637 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:24.994612    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg"
Sep 09 20:54:25.117035 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:25.116944    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/996caf85-035a-4cdc-a11e-e5062e1eafc1-tls-certs\") pod \"996caf85-035a-4cdc-a11e-e5062e1eafc1\" (UID: \"996caf85-035a-4cdc-a11e-e5062e1eafc1\") "
Sep 09 20:54:25.117197 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:25.117058    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s47fh\" (UniqueName: \"kubernetes.io/projected/996caf85-035a-4cdc-a11e-e5062e1eafc1-kube-api-access-s47fh\") pod \"996caf85-035a-4cdc-a11e-e5062e1eafc1\" (UID: \"996caf85-035a-4cdc-a11e-e5062e1eafc1\") "
Sep 09 20:54:25.119098 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:25.119057    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/996caf85-035a-4cdc-a11e-e5062e1eafc1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "996caf85-035a-4cdc-a11e-e5062e1eafc1" (UID: "996caf85-035a-4cdc-a11e-e5062e1eafc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:25.119211 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:25.119169    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996caf85-035a-4cdc-a11e-e5062e1eafc1-kube-api-access-s47fh" (OuterVolumeSpecName: "kube-api-access-s47fh") pod "996caf85-035a-4cdc-a11e-e5062e1eafc1" (UID: "996caf85-035a-4cdc-a11e-e5062e1eafc1"). InnerVolumeSpecName "kube-api-access-s47fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:25.217793 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:25.217762    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s47fh\" (UniqueName: \"kubernetes.io/projected/996caf85-035a-4cdc-a11e-e5062e1eafc1-kube-api-access-s47fh\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:54:25.217793 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:25.217786    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/996caf85-035a-4cdc-a11e-e5062e1eafc1-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:54:25.503815 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:25.503785    2581 generic.go:356] "Generic (PLEG): container finished" podID="996caf85-035a-4cdc-a11e-e5062e1eafc1" containerID="2762fbf15f65973061bc6023270f7f16bd5da7003d7ff1d6c1440ff386aa3a8d" exitCode=0
Sep 09 20:54:25.503979 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:25.503832    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg"
Sep 09 20:54:25.503979 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:25.503846    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg" event={"ID":"996caf85-035a-4cdc-a11e-e5062e1eafc1","Type":"ContainerDied","Data":"2762fbf15f65973061bc6023270f7f16bd5da7003d7ff1d6c1440ff386aa3a8d"}
Sep 09 20:54:25.503979 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:25.503886    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg" event={"ID":"996caf85-035a-4cdc-a11e-e5062e1eafc1","Type":"ContainerDied","Data":"190fe8a0b203a3dcbb31dd3a6b9ecfe6b91e7da8d936f578a721aef22cb04260"}
Sep 09 20:54:25.503979 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:25.503907    2581 scope.go:117] "RemoveContainer" containerID="2762fbf15f65973061bc6023270f7f16bd5da7003d7ff1d6c1440ff386aa3a8d"
Sep 09 20:54:25.512747 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:25.512732    2581 scope.go:117] "RemoveContainer" containerID="2762fbf15f65973061bc6023270f7f16bd5da7003d7ff1d6c1440ff386aa3a8d"
Sep 09 20:54:25.513013 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:54:25.512994    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2762fbf15f65973061bc6023270f7f16bd5da7003d7ff1d6c1440ff386aa3a8d\": container with ID starting with 2762fbf15f65973061bc6023270f7f16bd5da7003d7ff1d6c1440ff386aa3a8d not found: ID does not exist" containerID="2762fbf15f65973061bc6023270f7f16bd5da7003d7ff1d6c1440ff386aa3a8d"
Sep 09 20:54:25.513057 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:25.513024    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2762fbf15f65973061bc6023270f7f16bd5da7003d7ff1d6c1440ff386aa3a8d"} err="failed to get container status \"2762fbf15f65973061bc6023270f7f16bd5da7003d7ff1d6c1440ff386aa3a8d\": rpc error: code = NotFound desc = could not find container \"2762fbf15f65973061bc6023270f7f16bd5da7003d7ff1d6c1440ff386aa3a8d\": container with ID starting with 2762fbf15f65973061bc6023270f7f16bd5da7003d7ff1d6c1440ff386aa3a8d not found: ID does not exist"
Sep 09 20:54:25.525990 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:25.525963    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg"]
Sep 09 20:54:25.532410 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:25.532385    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schemgpdg"]
Sep 09 20:54:26.634967 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:26.634934    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="996caf85-035a-4cdc-a11e-e5062e1eafc1" path="/var/lib/kubelet/pods/996caf85-035a-4cdc-a11e-e5062e1eafc1/volumes"
Sep 09 20:54:41.639592 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:41.639553    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg"]
Sep 09 20:54:41.640143 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:41.640057    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="996caf85-035a-4cdc-a11e-e5062e1eafc1" containerName="main"
Sep 09 20:54:41.640143 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:41.640077    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="996caf85-035a-4cdc-a11e-e5062e1eafc1" containerName="main"
Sep 09 20:54:41.640257 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:41.640159    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="996caf85-035a-4cdc-a11e-e5062e1eafc1" containerName="main"
Sep 09 20:54:41.643143 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:41.643120    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg"
Sep 09 20:54:41.646689 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:41.646662    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-crlf8\""
Sep 09 20:54:41.646865 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:41.646661    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-j8rgt\""
Sep 09 20:54:41.646865 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:41.646734    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 09 20:54:41.646865 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:41.646743    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 09 20:54:41.646865 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:41.646663    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 09 20:54:41.651020 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:41.650997    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg"]
Sep 09 20:54:41.748396 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:41.748364    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8zkz\" (UniqueName: \"kubernetes.io/projected/c5df8c4d-abf1-435d-b1cc-0ed834174d76-kube-api-access-z8zkz\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg\" (UID: \"c5df8c4d-abf1-435d-b1cc-0ed834174d76\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg"
Sep 09 20:54:41.748562 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:41.748410    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5df8c4d-abf1-435d-b1cc-0ed834174d76-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg\" (UID: \"c5df8c4d-abf1-435d-b1cc-0ed834174d76\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg"
Sep 09 20:54:41.848900 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:41.848861    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8zkz\" (UniqueName: \"kubernetes.io/projected/c5df8c4d-abf1-435d-b1cc-0ed834174d76-kube-api-access-z8zkz\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg\" (UID: \"c5df8c4d-abf1-435d-b1cc-0ed834174d76\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg"
Sep 09 20:54:41.849051 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:41.848913    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5df8c4d-abf1-435d-b1cc-0ed834174d76-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg\" (UID: \"c5df8c4d-abf1-435d-b1cc-0ed834174d76\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg"
Sep 09 20:54:41.851399 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:41.851367    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5df8c4d-abf1-435d-b1cc-0ed834174d76-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg\" (UID: \"c5df8c4d-abf1-435d-b1cc-0ed834174d76\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg"
Sep 09 20:54:41.857947 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:41.857924    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8zkz\" (UniqueName: \"kubernetes.io/projected/c5df8c4d-abf1-435d-b1cc-0ed834174d76-kube-api-access-z8zkz\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg\" (UID: \"c5df8c4d-abf1-435d-b1cc-0ed834174d76\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg"
Sep 09 20:54:41.954999 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:41.954920    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg"
Sep 09 20:54:42.226353 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:42.226326    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg"]
Sep 09 20:54:42.228771 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:54:42.228736    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5df8c4d_abf1_435d_b1cc_0ed834174d76.slice/crio-0142fce8626886e2b15a4e6e1a63eac40ea6ec4e16c4b77213390deb1f372352 WatchSource:0}: Error finding container 0142fce8626886e2b15a4e6e1a63eac40ea6ec4e16c4b77213390deb1f372352: Status 404 returned error can't find the container with id 0142fce8626886e2b15a4e6e1a63eac40ea6ec4e16c4b77213390deb1f372352
Sep 09 20:54:42.591220 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:42.591184    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg" event={"ID":"c5df8c4d-abf1-435d-b1cc-0ed834174d76","Type":"ContainerStarted","Data":"e8aa0921dc5c13e60f3898b9141d5c9f43fd814d43a2a10ac1cba3cb653cda55"}
Sep 09 20:54:42.591389 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:42.591239    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg" event={"ID":"c5df8c4d-abf1-435d-b1cc-0ed834174d76","Type":"ContainerStarted","Data":"0142fce8626886e2b15a4e6e1a63eac40ea6ec4e16c4b77213390deb1f372352"}
Sep 09 20:54:42.591389 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:42.591275    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg"
Sep 09 20:54:42.608567 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:54:42.608507    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg" podStartSLOduration=1.6084871779999999 podStartE2EDuration="1.608487178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:42.607167993 +0000 UTC m=+2624.612345000" watchObservedRunningTime="2026-09-09 20:54:42.608487178 +0000 UTC m=+2624.613664175"
Sep 09 20:55:13.598003 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:55:13.597972    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg"
Sep 09 20:55:26.445954 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:55:26.445919    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz"]
Sep 09 20:55:26.449389 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:55:26.449372    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz"
Sep 09 20:55:26.451899 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:55:26.451873    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 09 20:55:26.452808 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:55:26.452792    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-jk2jc\""
Sep 09 20:55:26.458572 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:55:26.458551    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz"]
Sep 09 20:55:26.514487 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:55:26.514442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzwnl\" (UniqueName: \"kubernetes.io/projected/70a52c68-3291-4ce1-9e97-78a0620e0618-kube-api-access-lzwnl\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz\" (UID: \"70a52c68-3291-4ce1-9e97-78a0620e0618\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz"
Sep 09 20:55:26.514653 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:55:26.514508    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70a52c68-3291-4ce1-9e97-78a0620e0618-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz\" (UID: \"70a52c68-3291-4ce1-9e97-78a0620e0618\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz"
Sep 09 20:55:26.615809 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:55:26.615770    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzwnl\" (UniqueName: \"kubernetes.io/projected/70a52c68-3291-4ce1-9e97-78a0620e0618-kube-api-access-lzwnl\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz\" (UID: \"70a52c68-3291-4ce1-9e97-78a0620e0618\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz"
Sep 09 20:55:26.615985 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:55:26.615828    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70a52c68-3291-4ce1-9e97-78a0620e0618-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz\" (UID: \"70a52c68-3291-4ce1-9e97-78a0620e0618\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz"
Sep 09 20:55:26.618611 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:55:26.618585    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70a52c68-3291-4ce1-9e97-78a0620e0618-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz\" (UID: \"70a52c68-3291-4ce1-9e97-78a0620e0618\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz"
Sep 09 20:55:26.625472 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:55:26.625435    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzwnl\" (UniqueName: \"kubernetes.io/projected/70a52c68-3291-4ce1-9e97-78a0620e0618-kube-api-access-lzwnl\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz\" (UID: \"70a52c68-3291-4ce1-9e97-78a0620e0618\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz"
Sep 09 20:55:26.759742 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:55:26.759710    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz"
Sep 09 20:55:27.031929 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:55:27.031901    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz"]
Sep 09 20:55:27.033775 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:55:27.033739    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a52c68_3291_4ce1_9e97_78a0620e0618.slice/crio-28b2586b135fe0063c7f1619d0186c901621110bea727323e902fa1f78ab9534 WatchSource:0}: Error finding container 28b2586b135fe0063c7f1619d0186c901621110bea727323e902fa1f78ab9534: Status 404 returned error can't find the container with id 28b2586b135fe0063c7f1619d0186c901621110bea727323e902fa1f78ab9534
Sep 09 20:55:27.817058 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:55:27.817021    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz" event={"ID":"70a52c68-3291-4ce1-9e97-78a0620e0618","Type":"ContainerStarted","Data":"83afd7d0004a34b80b1bc17f22008d703fb030710cf1748f0beef42cf0ee4d2d"}
Sep 09 20:55:27.817448 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:55:27.817075    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz" event={"ID":"70a52c68-3291-4ce1-9e97-78a0620e0618","Type":"ContainerStarted","Data":"28b2586b135fe0063c7f1619d0186c901621110bea727323e902fa1f78ab9534"}
Sep 09 20:55:27.817448 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:55:27.817132    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz"
Sep 09 20:55:27.837442 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:55:27.837393    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz" podStartSLOduration=1.83737712 podStartE2EDuration="1.83737712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:27.835326826 +0000 UTC m=+2669.840503836" watchObservedRunningTime="2026-09-09 20:55:27.83737712 +0000 UTC m=+2669.842554114"
Sep 09 20:55:58.691748 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:55:58.691643    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 20:55:58.694475 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:55:58.694437    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 20:55:58.697871 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:55:58.697853    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 20:55:58.703481 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:55:58.703446    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 20:55:58.824674 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:55:58.824644    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz"
Sep 09 20:57:12.562654 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:57:12.562616    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz"]
Sep 09 20:57:18.824896 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:57:18.824851    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz" podUID="70a52c68-3291-4ce1-9e97-78a0620e0618" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:57:27.582722 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:57:27.582630    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz" podUID="70a52c68-3291-4ce1-9e97-78a0620e0618" containerName="main" containerID="cri-o://83afd7d0004a34b80b1bc17f22008d703fb030710cf1748f0beef42cf0ee4d2d" gracePeriod=45
Sep 09 20:57:28.824793 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:57:28.824750    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz" podUID="70a52c68-3291-4ce1-9e97-78a0620e0618" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:57:38.824652 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:57:38.824606    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz" podUID="70a52c68-3291-4ce1-9e97-78a0620e0618" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:57:38.825033 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:57:38.824724    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz"
Sep 09 20:57:48.824586 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:57:48.824546    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz" podUID="70a52c68-3291-4ce1-9e97-78a0620e0618" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:57:57.833142 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:57:57.833119    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz"
Sep 09 20:57:57.942836 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:57:57.942751    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70a52c68-3291-4ce1-9e97-78a0620e0618-tls-certs\") pod \"70a52c68-3291-4ce1-9e97-78a0620e0618\" (UID: \"70a52c68-3291-4ce1-9e97-78a0620e0618\") "
Sep 09 20:57:57.942836 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:57:57.942832    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzwnl\" (UniqueName: \"kubernetes.io/projected/70a52c68-3291-4ce1-9e97-78a0620e0618-kube-api-access-lzwnl\") pod \"70a52c68-3291-4ce1-9e97-78a0620e0618\" (UID: \"70a52c68-3291-4ce1-9e97-78a0620e0618\") "
Sep 09 20:57:57.945139 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:57:57.945111    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70a52c68-3291-4ce1-9e97-78a0620e0618-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70a52c68-3291-4ce1-9e97-78a0620e0618" (UID: "70a52c68-3291-4ce1-9e97-78a0620e0618"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:57:57.945266 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:57:57.945160    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a52c68-3291-4ce1-9e97-78a0620e0618-kube-api-access-lzwnl" (OuterVolumeSpecName: "kube-api-access-lzwnl") pod "70a52c68-3291-4ce1-9e97-78a0620e0618" (UID: "70a52c68-3291-4ce1-9e97-78a0620e0618"). InnerVolumeSpecName "kube-api-access-lzwnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:57:58.043861 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:57:58.043823    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzwnl\" (UniqueName: \"kubernetes.io/projected/70a52c68-3291-4ce1-9e97-78a0620e0618-kube-api-access-lzwnl\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:57:58.043861 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:57:58.043856    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70a52c68-3291-4ce1-9e97-78a0620e0618-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:57:58.574764 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:57:58.574730    2581 generic.go:356] "Generic (PLEG): container finished" podID="70a52c68-3291-4ce1-9e97-78a0620e0618" containerID="83afd7d0004a34b80b1bc17f22008d703fb030710cf1748f0beef42cf0ee4d2d" exitCode=0
Sep 09 20:57:58.574930 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:57:58.574797    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz"
Sep 09 20:57:58.574930 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:57:58.574815    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz" event={"ID":"70a52c68-3291-4ce1-9e97-78a0620e0618","Type":"ContainerDied","Data":"83afd7d0004a34b80b1bc17f22008d703fb030710cf1748f0beef42cf0ee4d2d"}
Sep 09 20:57:58.574930 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:57:58.574892    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz" event={"ID":"70a52c68-3291-4ce1-9e97-78a0620e0618","Type":"ContainerDied","Data":"28b2586b135fe0063c7f1619d0186c901621110bea727323e902fa1f78ab9534"}
Sep 09 20:57:58.574930 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:57:58.574914    2581 scope.go:117] "RemoveContainer" containerID="83afd7d0004a34b80b1bc17f22008d703fb030710cf1748f0beef42cf0ee4d2d"
Sep 09 20:57:58.583817 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:57:58.583801    2581 scope.go:117] "RemoveContainer" containerID="83afd7d0004a34b80b1bc17f22008d703fb030710cf1748f0beef42cf0ee4d2d"
Sep 09 20:57:58.584072 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:57:58.584056    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83afd7d0004a34b80b1bc17f22008d703fb030710cf1748f0beef42cf0ee4d2d\": container with ID starting with 83afd7d0004a34b80b1bc17f22008d703fb030710cf1748f0beef42cf0ee4d2d not found: ID does not exist" containerID="83afd7d0004a34b80b1bc17f22008d703fb030710cf1748f0beef42cf0ee4d2d"
Sep 09 20:57:58.584118 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:57:58.584079    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83afd7d0004a34b80b1bc17f22008d703fb030710cf1748f0beef42cf0ee4d2d"} err="failed to get container status \"83afd7d0004a34b80b1bc17f22008d703fb030710cf1748f0beef42cf0ee4d2d\": rpc error: code = NotFound desc = could not find container \"83afd7d0004a34b80b1bc17f22008d703fb030710cf1748f0beef42cf0ee4d2d\": container with ID starting with 83afd7d0004a34b80b1bc17f22008d703fb030710cf1748f0beef42cf0ee4d2d not found: ID does not exist"
Sep 09 20:57:58.597886 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:57:58.597861    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz"]
Sep 09 20:57:58.602638 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:57:58.602616    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche4h2tz"]
Sep 09 20:57:58.635291 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:57:58.635263    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70a52c68-3291-4ce1-9e97-78a0620e0618" path="/var/lib/kubelet/pods/70a52c68-3291-4ce1-9e97-78a0620e0618/volumes"
Sep 09 20:58:00.228056 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:00.228023    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg"]
Sep 09 20:58:03.597327 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:03.597289    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg" podUID="c5df8c4d-abf1-435d-b1cc-0ed834174d76" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:58:13.597164 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:13.597125    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg" podUID="c5df8c4d-abf1-435d-b1cc-0ed834174d76" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:58:15.249023 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:15.248947    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg" podUID="c5df8c4d-abf1-435d-b1cc-0ed834174d76" containerName="main" containerID="cri-o://e8aa0921dc5c13e60f3898b9141d5c9f43fd814d43a2a10ac1cba3cb653cda55" gracePeriod=45
Sep 09 20:58:23.598094 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:23.598058    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg" podUID="c5df8c4d-abf1-435d-b1cc-0ed834174d76" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:58:23.598594 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:23.598168    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg"
Sep 09 20:58:33.597923 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:33.597877    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg" podUID="c5df8c4d-abf1-435d-b1cc-0ed834174d76" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:58:43.597222 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:43.597186    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg" podUID="c5df8c4d-abf1-435d-b1cc-0ed834174d76" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:58:45.497347 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:45.497321    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg"
Sep 09 20:58:45.632358 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:45.632265    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5df8c4d-abf1-435d-b1cc-0ed834174d76-tls-certs\") pod \"c5df8c4d-abf1-435d-b1cc-0ed834174d76\" (UID: \"c5df8c4d-abf1-435d-b1cc-0ed834174d76\") "
Sep 09 20:58:45.632358 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:45.632331    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8zkz\" (UniqueName: \"kubernetes.io/projected/c5df8c4d-abf1-435d-b1cc-0ed834174d76-kube-api-access-z8zkz\") pod \"c5df8c4d-abf1-435d-b1cc-0ed834174d76\" (UID: \"c5df8c4d-abf1-435d-b1cc-0ed834174d76\") "
Sep 09 20:58:45.634443 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:45.634409    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5df8c4d-abf1-435d-b1cc-0ed834174d76-kube-api-access-z8zkz" (OuterVolumeSpecName: "kube-api-access-z8zkz") pod "c5df8c4d-abf1-435d-b1cc-0ed834174d76" (UID: "c5df8c4d-abf1-435d-b1cc-0ed834174d76"). InnerVolumeSpecName "kube-api-access-z8zkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:58:45.634443 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:45.634409    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5df8c4d-abf1-435d-b1cc-0ed834174d76-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c5df8c4d-abf1-435d-b1cc-0ed834174d76" (UID: "c5df8c4d-abf1-435d-b1cc-0ed834174d76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:58:45.733390 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:45.733356    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5df8c4d-abf1-435d-b1cc-0ed834174d76-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:58:45.733390 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:45.733385    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8zkz\" (UniqueName: \"kubernetes.io/projected/c5df8c4d-abf1-435d-b1cc-0ed834174d76-kube-api-access-z8zkz\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 20:58:45.805291 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:45.805259    2581 generic.go:356] "Generic (PLEG): container finished" podID="c5df8c4d-abf1-435d-b1cc-0ed834174d76" containerID="e8aa0921dc5c13e60f3898b9141d5c9f43fd814d43a2a10ac1cba3cb653cda55" exitCode=0
Sep 09 20:58:45.805428 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:45.805312    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg"
Sep 09 20:58:45.805428 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:45.805345    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg" event={"ID":"c5df8c4d-abf1-435d-b1cc-0ed834174d76","Type":"ContainerDied","Data":"e8aa0921dc5c13e60f3898b9141d5c9f43fd814d43a2a10ac1cba3cb653cda55"}
Sep 09 20:58:45.805428 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:45.805395    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg" event={"ID":"c5df8c4d-abf1-435d-b1cc-0ed834174d76","Type":"ContainerDied","Data":"0142fce8626886e2b15a4e6e1a63eac40ea6ec4e16c4b77213390deb1f372352"}
Sep 09 20:58:45.805428 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:45.805417    2581 scope.go:117] "RemoveContainer" containerID="e8aa0921dc5c13e60f3898b9141d5c9f43fd814d43a2a10ac1cba3cb653cda55"
Sep 09 20:58:45.814044 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:45.814022    2581 scope.go:117] "RemoveContainer" containerID="e8aa0921dc5c13e60f3898b9141d5c9f43fd814d43a2a10ac1cba3cb653cda55"
Sep 09 20:58:45.814315 ip-10-0-134-84 kubenswrapper[2581]: E0909 20:58:45.814296    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8aa0921dc5c13e60f3898b9141d5c9f43fd814d43a2a10ac1cba3cb653cda55\": container with ID starting with e8aa0921dc5c13e60f3898b9141d5c9f43fd814d43a2a10ac1cba3cb653cda55 not found: ID does not exist" containerID="e8aa0921dc5c13e60f3898b9141d5c9f43fd814d43a2a10ac1cba3cb653cda55"
Sep 09 20:58:45.814380 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:45.814321    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8aa0921dc5c13e60f3898b9141d5c9f43fd814d43a2a10ac1cba3cb653cda55"} err="failed to get container status \"e8aa0921dc5c13e60f3898b9141d5c9f43fd814d43a2a10ac1cba3cb653cda55\": rpc error: code = NotFound desc = could not find container \"e8aa0921dc5c13e60f3898b9141d5c9f43fd814d43a2a10ac1cba3cb653cda55\": container with ID starting with e8aa0921dc5c13e60f3898b9141d5c9f43fd814d43a2a10ac1cba3cb653cda55 not found: ID does not exist"
Sep 09 20:58:45.831432 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:45.831409    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg"]
Sep 09 20:58:45.838147 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:45.838128    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetqhdg"]
Sep 09 20:58:46.635965 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:46.635929    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5df8c4d-abf1-435d-b1cc-0ed834174d76" path="/var/lib/kubelet/pods/c5df8c4d-abf1-435d-b1cc-0ed834174d76/volumes"
Sep 09 20:58:54.694902 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:54.694824    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk"]
Sep 09 20:58:54.695346 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:54.695318    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70a52c68-3291-4ce1-9e97-78a0620e0618" containerName="main"
Sep 09 20:58:54.695346 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:54.695336    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="70a52c68-3291-4ce1-9e97-78a0620e0618" containerName="main"
Sep 09 20:58:54.695424 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:54.695370    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5df8c4d-abf1-435d-b1cc-0ed834174d76" containerName="main"
Sep 09 20:58:54.695424 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:54.695380    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5df8c4d-abf1-435d-b1cc-0ed834174d76" containerName="main"
Sep 09 20:58:54.695506 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:54.695485    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="70a52c68-3291-4ce1-9e97-78a0620e0618" containerName="main"
Sep 09 20:58:54.695506 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:54.695499    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5df8c4d-abf1-435d-b1cc-0ed834174d76" containerName="main"
Sep 09 20:58:54.700170 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:54.700148    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk"
Sep 09 20:58:54.703174 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:54.703070    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 09 20:58:54.703174 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:54.703097    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 09 20:58:54.703384 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:54.703360    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 09 20:58:54.703708 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:54.703688    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-bnhqh\""
Sep 09 20:58:54.704002 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:54.703975    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-zdmph\""
Sep 09 20:58:54.706512 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:54.706485    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk"]
Sep 09 20:58:54.807911 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:54.807877    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tp6z\" (UniqueName: \"kubernetes.io/projected/3250d655-294a-4aa8-8c71-3f8fdf9d913d-kube-api-access-2tp6z\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk\" (UID: \"3250d655-294a-4aa8-8c71-3f8fdf9d913d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk"
Sep 09 20:58:54.808096 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:54.807928    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3250d655-294a-4aa8-8c71-3f8fdf9d913d-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk\" (UID: \"3250d655-294a-4aa8-8c71-3f8fdf9d913d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk"
Sep 09 20:58:54.908678 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:54.908646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3250d655-294a-4aa8-8c71-3f8fdf9d913d-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk\" (UID: \"3250d655-294a-4aa8-8c71-3f8fdf9d913d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk"
Sep 09 20:58:54.908936 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:54.908736    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tp6z\" (UniqueName: \"kubernetes.io/projected/3250d655-294a-4aa8-8c71-3f8fdf9d913d-kube-api-access-2tp6z\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk\" (UID: \"3250d655-294a-4aa8-8c71-3f8fdf9d913d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk"
Sep 09 20:58:54.911300 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:54.911267    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3250d655-294a-4aa8-8c71-3f8fdf9d913d-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk\" (UID: \"3250d655-294a-4aa8-8c71-3f8fdf9d913d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk"
Sep 09 20:58:54.921370 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:54.921344    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tp6z\" (UniqueName: \"kubernetes.io/projected/3250d655-294a-4aa8-8c71-3f8fdf9d913d-kube-api-access-2tp6z\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk\" (UID: \"3250d655-294a-4aa8-8c71-3f8fdf9d913d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk"
Sep 09 20:58:55.011894 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:55.011862    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk"
Sep 09 20:58:55.286549 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:55.286521    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk"]
Sep 09 20:58:55.288341 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:58:55.288306    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3250d655_294a_4aa8_8c71_3f8fdf9d913d.slice/crio-c2342e57ae260855e3c265ac724d7f0190db5a5801253c9905d1796e3b57cc34 WatchSource:0}: Error finding container c2342e57ae260855e3c265ac724d7f0190db5a5801253c9905d1796e3b57cc34: Status 404 returned error can't find the container with id c2342e57ae260855e3c265ac724d7f0190db5a5801253c9905d1796e3b57cc34
Sep 09 20:58:55.857005 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:55.856966    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk" event={"ID":"3250d655-294a-4aa8-8c71-3f8fdf9d913d","Type":"ContainerStarted","Data":"85c7a136ce464b6f8700764aa53dbdfd546a7b693578b244276592364f469c3e"}
Sep 09 20:58:55.857441 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:55.857023    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk" event={"ID":"3250d655-294a-4aa8-8c71-3f8fdf9d913d","Type":"ContainerStarted","Data":"c2342e57ae260855e3c265ac724d7f0190db5a5801253c9905d1796e3b57cc34"}
Sep 09 20:58:55.857441 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:55.857095    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk"
Sep 09 20:58:55.874252 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:58:55.874200    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk" podStartSLOduration=1.874183039 podStartE2EDuration="1.874183039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:55.872840455 +0000 UTC m=+2877.878017462" watchObservedRunningTime="2026-09-09 20:58:55.874183039 +0000 UTC m=+2877.879360033"
Sep 09 20:59:26.864223 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:26.864186    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk"
Sep 09 20:59:34.681602 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:34.681567    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"]
Sep 09 20:59:34.685468 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:34.685435    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 20:59:34.687773 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:34.687753    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 09 20:59:34.694218 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:34.694196    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"]
Sep 09 20:59:34.716873 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:34.716837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm\" (UID: \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 20:59:34.717020 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:34.716972    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm\" (UID: \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 20:59:34.717127 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:34.717094    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm\" (UID: \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 20:59:34.717203 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:34.717181    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm\" (UID: \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 20:59:34.717267 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:34.717228    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxtq6\" (UniqueName: \"kubernetes.io/projected/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-kube-api-access-dxtq6\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm\" (UID: \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 20:59:34.717324 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:34.717275    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm\" (UID: \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 20:59:34.818571 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:34.818531    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm\" (UID: \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 20:59:34.818734 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:34.818594    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm\" (UID: \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 20:59:34.818734 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:34.818700    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxtq6\" (UniqueName: \"kubernetes.io/projected/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-kube-api-access-dxtq6\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm\" (UID: \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 20:59:34.818876 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:34.818746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm\" (UID: \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 20:59:34.818876 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:34.818789    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm\" (UID: \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 20:59:34.819026 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:34.818904    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm\" (UID: \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 20:59:34.819026 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:34.819002    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm\" (UID: \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 20:59:34.819305 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:34.819271    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm\" (UID: \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 20:59:34.819593 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:34.819568    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm\" (UID: \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 20:59:34.821640 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:34.821595    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm\" (UID: \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 20:59:34.821838 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:34.821820    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm\" (UID: \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 20:59:34.827677 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:34.827657    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxtq6\" (UniqueName: \"kubernetes.io/projected/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-kube-api-access-dxtq6\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm\" (UID: \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 20:59:34.997295 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:34.997266    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 20:59:35.271096 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:35.271069    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"]
Sep 09 20:59:35.273150 ip-10-0-134-84 kubenswrapper[2581]: W0909 20:59:35.273120    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c2119e_c2a0_48b5_9a78_720e3ef8e57d.slice/crio-ebc3ac4cb6f1d7a3050b46c85f718a5a2c5e1ab9e3f4523f6a25f5f3d3168e8d WatchSource:0}: Error finding container ebc3ac4cb6f1d7a3050b46c85f718a5a2c5e1ab9e3f4523f6a25f5f3d3168e8d: Status 404 returned error can't find the container with id ebc3ac4cb6f1d7a3050b46c85f718a5a2c5e1ab9e3f4523f6a25f5f3d3168e8d
Sep 09 20:59:36.064101 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:36.064063    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm" event={"ID":"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d","Type":"ContainerStarted","Data":"6519999d7904fe0e10a364c650a0b4e9a665593255e756378ba640bb29c8f068"}
Sep 09 20:59:36.064555 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:36.064123    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm" event={"ID":"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d","Type":"ContainerStarted","Data":"ebc3ac4cb6f1d7a3050b46c85f718a5a2c5e1ab9e3f4523f6a25f5f3d3168e8d"}
Sep 09 20:59:36.083529 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:36.083483    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm" podStartSLOduration=2.0834494 podStartE2EDuration="2.0834494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:59:36.080768792 +0000 UTC m=+2918.085945787" watchObservedRunningTime="2026-09-09 20:59:36.0834494 +0000 UTC m=+2918.088626395"
Sep 09 20:59:36.998105 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:36.998070    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 20:59:44.998278 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:44.998240    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 20:59:45.010743 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:45.010715    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 20:59:45.122041 ip-10-0-134-84 kubenswrapper[2581]: I0909 20:59:45.122013    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 21:00:09.286512 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:09.286476    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"]
Sep 09 21:00:09.297248 ip-10-0-134-84 kubenswrapper[2581]: E0909 21:00:09.297212    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:00:09.297248 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:00:09.297248 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 21:00:09.297248 ip-10-0-134-84 kubenswrapper[2581]:  > containerID="6519999d7904fe0e10a364c650a0b4e9a665593255e756378ba640bb29c8f068" cmd=["/bin/sleep","15"]
Sep 09 21:00:09.297522 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:09.297264    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm" message="" err=<
Sep 09 21:00:09.297522 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:00:09.297522 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 21:00:09.297522 ip-10-0-134-84 kubenswrapper[2581]:  >
Sep 09 21:00:09.297522 ip-10-0-134-84 kubenswrapper[2581]: E0909 21:00:09.297276    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:00:09.297522 ip-10-0-134-84 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:00:09.297522 ip-10-0-134-84 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 09 21:00:09.297522 ip-10-0-134-84 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm" podUID="a5c2119e-c2a0-48b5-9a78-720e3ef8e57d" containerName="main" containerID="cri-o://6519999d7904fe0e10a364c650a0b4e9a665593255e756378ba640bb29c8f068"
Sep 09 21:00:09.297522 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:09.297315    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm" podUID="a5c2119e-c2a0-48b5-9a78-720e3ef8e57d" containerName="main" containerID="cri-o://6519999d7904fe0e10a364c650a0b4e9a665593255e756378ba640bb29c8f068" gracePeriod=60
Sep 09 21:00:09.669610 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:09.669586    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 21:00:09.820638 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:09.820606    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-tmp-dir\") pod \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\" (UID: \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\") "
Sep 09 21:00:09.820814 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:09.820659    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-model-cache\") pod \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\" (UID: \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\") "
Sep 09 21:00:09.820814 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:09.820693    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-tls-certs\") pod \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\" (UID: \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\") "
Sep 09 21:00:09.820814 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:09.820720    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-dshm\") pod \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\" (UID: \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\") "
Sep 09 21:00:09.820814 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:09.820740    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-home\") pod \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\" (UID: \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\") "
Sep 09 21:00:09.820814 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:09.820764    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxtq6\" (UniqueName: \"kubernetes.io/projected/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-kube-api-access-dxtq6\") pod \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\" (UID: \"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d\") "
Sep 09 21:00:09.821095 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:09.820904    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5c2119e-c2a0-48b5-9a78-720e3ef8e57d" (UID: "a5c2119e-c2a0-48b5-9a78-720e3ef8e57d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:09.821095 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:09.820994    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-home" (OuterVolumeSpecName: "home") pod "a5c2119e-c2a0-48b5-9a78-720e3ef8e57d" (UID: "a5c2119e-c2a0-48b5-9a78-720e3ef8e57d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:09.821095 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:09.821001    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5c2119e-c2a0-48b5-9a78-720e3ef8e57d" (UID: "a5c2119e-c2a0-48b5-9a78-720e3ef8e57d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:09.822845 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:09.822816    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5c2119e-c2a0-48b5-9a78-720e3ef8e57d" (UID: "a5c2119e-c2a0-48b5-9a78-720e3ef8e57d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:00:09.822970 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:09.822856    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-dshm" (OuterVolumeSpecName: "dshm") pod "a5c2119e-c2a0-48b5-9a78-720e3ef8e57d" (UID: "a5c2119e-c2a0-48b5-9a78-720e3ef8e57d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:09.822970 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:09.822871    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-kube-api-access-dxtq6" (OuterVolumeSpecName: "kube-api-access-dxtq6") pod "a5c2119e-c2a0-48b5-9a78-720e3ef8e57d" (UID: "a5c2119e-c2a0-48b5-9a78-720e3ef8e57d"). InnerVolumeSpecName "kube-api-access-dxtq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:00:09.921635 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:09.921601    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:00:09.921635 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:09.921630    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:00:09.921635 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:09.921641    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:00:09.921864 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:09.921648    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:00:09.921864 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:09.921656    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:00:09.921864 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:09.921664    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxtq6\" (UniqueName: \"kubernetes.io/projected/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d-kube-api-access-dxtq6\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:00:10.254250 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:10.254214    2581 generic.go:356] "Generic (PLEG): container finished" podID="a5c2119e-c2a0-48b5-9a78-720e3ef8e57d" containerID="6519999d7904fe0e10a364c650a0b4e9a665593255e756378ba640bb29c8f068" exitCode=0
Sep 09 21:00:10.254250 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:10.254265    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"
Sep 09 21:00:10.254485 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:10.254279    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm" event={"ID":"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d","Type":"ContainerDied","Data":"6519999d7904fe0e10a364c650a0b4e9a665593255e756378ba640bb29c8f068"}
Sep 09 21:00:10.254485 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:10.254333    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm" event={"ID":"a5c2119e-c2a0-48b5-9a78-720e3ef8e57d","Type":"ContainerDied","Data":"ebc3ac4cb6f1d7a3050b46c85f718a5a2c5e1ab9e3f4523f6a25f5f3d3168e8d"}
Sep 09 21:00:10.254485 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:10.254363    2581 scope.go:117] "RemoveContainer" containerID="6519999d7904fe0e10a364c650a0b4e9a665593255e756378ba640bb29c8f068"
Sep 09 21:00:10.265926 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:10.265680    2581 scope.go:117] "RemoveContainer" containerID="6519999d7904fe0e10a364c650a0b4e9a665593255e756378ba640bb29c8f068"
Sep 09 21:00:10.266811 ip-10-0-134-84 kubenswrapper[2581]: E0909 21:00:10.266653    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6519999d7904fe0e10a364c650a0b4e9a665593255e756378ba640bb29c8f068\": container with ID starting with 6519999d7904fe0e10a364c650a0b4e9a665593255e756378ba640bb29c8f068 not found: ID does not exist" containerID="6519999d7904fe0e10a364c650a0b4e9a665593255e756378ba640bb29c8f068"
Sep 09 21:00:10.266811 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:10.266690    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6519999d7904fe0e10a364c650a0b4e9a665593255e756378ba640bb29c8f068"} err="failed to get container status \"6519999d7904fe0e10a364c650a0b4e9a665593255e756378ba640bb29c8f068\": rpc error: code = NotFound desc = could not find container \"6519999d7904fe0e10a364c650a0b4e9a665593255e756378ba640bb29c8f068\": container with ID starting with 6519999d7904fe0e10a364c650a0b4e9a665593255e756378ba640bb29c8f068 not found: ID does not exist"
Sep 09 21:00:10.280674 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:10.280649    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"]
Sep 09 21:00:10.285114 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:10.285087    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-5kzxm"]
Sep 09 21:00:10.637062 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:10.636983    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5c2119e-c2a0-48b5-9a78-720e3ef8e57d" path="/var/lib/kubelet/pods/a5c2119e-c2a0-48b5-9a78-720e3ef8e57d/volumes"
Sep 09 21:00:27.568728 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.568650    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"]
Sep 09 21:00:27.569168 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.568978    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5c2119e-c2a0-48b5-9a78-720e3ef8e57d" containerName="main"
Sep 09 21:00:27.569168 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.568989    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5c2119e-c2a0-48b5-9a78-720e3ef8e57d" containerName="main"
Sep 09 21:00:27.569168 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.569048    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5c2119e-c2a0-48b5-9a78-720e3ef8e57d" containerName="main"
Sep 09 21:00:27.573372 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.573348    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.575879 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.575849    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 09 21:00:27.576069 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.576044    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 09 21:00:27.576197 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.576086    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-p25sj\""
Sep 09 21:00:27.576381 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.576364    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 09 21:00:27.576494 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.576428    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:00:27.576705 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.576686    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 09 21:00:27.584195 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.584163    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"]
Sep 09 21:00:27.677064 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.677024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.677064 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.677069    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.677286 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.677101    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a8d555-c86b-4ada-99cc-f6925b2f42e3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.677286 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.677137    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.677286 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.677168    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75a8d555-c86b-4ada-99cc-f6925b2f42e3-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.677286 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.677186    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.677286 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.677206    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hwdk\" (UniqueName: \"kubernetes.io/projected/75a8d555-c86b-4ada-99cc-f6925b2f42e3-kube-api-access-5hwdk\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.677516 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.677276    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.778737 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.778694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.778941 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.778760    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75a8d555-c86b-4ada-99cc-f6925b2f42e3-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.778941 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.778789    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.778941 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.778820    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hwdk\" (UniqueName: \"kubernetes.io/projected/75a8d555-c86b-4ada-99cc-f6925b2f42e3-kube-api-access-5hwdk\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.778941 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.778866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.778941 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.778900    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.778941 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.778935    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.779376 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.778982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a8d555-c86b-4ada-99cc-f6925b2f42e3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.779376 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.779156    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.779376 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.779191    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.779376 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.779275    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.779569 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.779392    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.779569 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.779519    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75a8d555-c86b-4ada-99cc-f6925b2f42e3-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.781270 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.781246    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.781526 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.781507    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a8d555-c86b-4ada-99cc-f6925b2f42e3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.789208 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.789181    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hwdk\" (UniqueName: \"kubernetes.io/projected/75a8d555-c86b-4ada-99cc-f6925b2f42e3-kube-api-access-5hwdk\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:27.886586 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:27.886496    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:28.168132 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:28.168108    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"]
Sep 09 21:00:28.170164 ip-10-0-134-84 kubenswrapper[2581]: W0909 21:00:28.170133    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a8d555_c86b_4ada_99cc_f6925b2f42e3.slice/crio-4b275828a02ebffa62ec0ab1e66b331d4283e244504023d35f15775847011968 WatchSource:0}: Error finding container 4b275828a02ebffa62ec0ab1e66b331d4283e244504023d35f15775847011968: Status 404 returned error can't find the container with id 4b275828a02ebffa62ec0ab1e66b331d4283e244504023d35f15775847011968
Sep 09 21:00:28.352403 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:28.352365    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f" event={"ID":"75a8d555-c86b-4ada-99cc-f6925b2f42e3","Type":"ContainerStarted","Data":"de2b0b015d264bfbf339cee0b42130e6d336217d4d651a2d773d2edecdc9345c"}
Sep 09 21:00:28.352628 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:28.352435    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f" event={"ID":"75a8d555-c86b-4ada-99cc-f6925b2f42e3","Type":"ContainerStarted","Data":"4b275828a02ebffa62ec0ab1e66b331d4283e244504023d35f15775847011968"}
Sep 09 21:00:31.371101 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:31.371064    2581 generic.go:356] "Generic (PLEG): container finished" podID="75a8d555-c86b-4ada-99cc-f6925b2f42e3" containerID="de2b0b015d264bfbf339cee0b42130e6d336217d4d651a2d773d2edecdc9345c" exitCode=0
Sep 09 21:00:31.371101 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:31.371106    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f" event={"ID":"75a8d555-c86b-4ada-99cc-f6925b2f42e3","Type":"ContainerDied","Data":"de2b0b015d264bfbf339cee0b42130e6d336217d4d651a2d773d2edecdc9345c"}
Sep 09 21:00:32.378567 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:32.378531    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f" event={"ID":"75a8d555-c86b-4ada-99cc-f6925b2f42e3","Type":"ContainerStarted","Data":"73569eda6d722e667d6455aa2565d9a2050d8b40792dbf407980b4414cb1bdb4"}
Sep 09 21:00:32.406003 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:32.405939    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f" podStartSLOduration=2.991669063 podStartE2EDuration="5.405923213s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.41425415s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:32.402623737 +0000 UTC m=+2974.407800731" watchObservedRunningTime="2026-09-09 21:00:32.405923213 +0000 UTC m=+2974.411100208"
Sep 09 21:00:32.887473 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:32.887431    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:37.886854 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:37.886813    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:00:37.888503 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:37.888454    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f" podUID="75a8d555-c86b-4ada-99cc-f6925b2f42e3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 09 21:00:47.887644 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:47.887604    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f" podUID="75a8d555-c86b-4ada-99cc-f6925b2f42e3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 09 21:00:57.887063 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:57.887011    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f" podUID="75a8d555-c86b-4ada-99cc-f6925b2f42e3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 09 21:00:58.729962 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:58.729932    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 21:00:58.733300 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:58.733283    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 21:00:58.737496 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:58.737450    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 21:00:58.740626 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:58.740609    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 21:00:59.392083 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:00:59.392041    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk"]
Sep 09 21:01:06.864953 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:06.864912    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk" podUID="3250d655-294a-4aa8-8c71-3f8fdf9d913d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:01:07.887224 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:07.887176    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f" podUID="75a8d555-c86b-4ada-99cc-f6925b2f42e3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 09 21:01:14.413653 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:14.413608    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk" podUID="3250d655-294a-4aa8-8c71-3f8fdf9d913d" containerName="main" containerID="cri-o://85c7a136ce464b6f8700764aa53dbdfd546a7b693578b244276592364f469c3e" gracePeriod=45
Sep 09 21:01:16.864491 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:16.864439    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk" podUID="3250d655-294a-4aa8-8c71-3f8fdf9d913d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:01:17.887758 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:17.887710    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f" podUID="75a8d555-c86b-4ada-99cc-f6925b2f42e3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 09 21:01:26.864762 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:26.864720    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk" podUID="3250d655-294a-4aa8-8c71-3f8fdf9d913d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:01:26.865292 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:26.864817    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk"
Sep 09 21:01:27.887437 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:27.887390    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f" podUID="75a8d555-c86b-4ada-99cc-f6925b2f42e3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 09 21:01:36.865035 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:36.864997    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk" podUID="3250d655-294a-4aa8-8c71-3f8fdf9d913d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:01:37.887505 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:37.887444    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f" podUID="75a8d555-c86b-4ada-99cc-f6925b2f42e3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 09 21:01:44.661762 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:44.661738    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk"
Sep 09 21:01:44.731850 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:44.731822    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tp6z\" (UniqueName: \"kubernetes.io/projected/3250d655-294a-4aa8-8c71-3f8fdf9d913d-kube-api-access-2tp6z\") pod \"3250d655-294a-4aa8-8c71-3f8fdf9d913d\" (UID: \"3250d655-294a-4aa8-8c71-3f8fdf9d913d\") "
Sep 09 21:01:44.732016 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:44.731897    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3250d655-294a-4aa8-8c71-3f8fdf9d913d-tls-certs\") pod \"3250d655-294a-4aa8-8c71-3f8fdf9d913d\" (UID: \"3250d655-294a-4aa8-8c71-3f8fdf9d913d\") "
Sep 09 21:01:44.733956 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:44.733925    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3250d655-294a-4aa8-8c71-3f8fdf9d913d-kube-api-access-2tp6z" (OuterVolumeSpecName: "kube-api-access-2tp6z") pod "3250d655-294a-4aa8-8c71-3f8fdf9d913d" (UID: "3250d655-294a-4aa8-8c71-3f8fdf9d913d"). InnerVolumeSpecName "kube-api-access-2tp6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:01:44.734059 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:44.733996    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3250d655-294a-4aa8-8c71-3f8fdf9d913d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3250d655-294a-4aa8-8c71-3f8fdf9d913d" (UID: "3250d655-294a-4aa8-8c71-3f8fdf9d913d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:01:44.771815 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:44.771781    2581 generic.go:356] "Generic (PLEG): container finished" podID="3250d655-294a-4aa8-8c71-3f8fdf9d913d" containerID="85c7a136ce464b6f8700764aa53dbdfd546a7b693578b244276592364f469c3e" exitCode=0
Sep 09 21:01:44.771922 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:44.771824    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk"
Sep 09 21:01:44.771922 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:44.771853    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk" event={"ID":"3250d655-294a-4aa8-8c71-3f8fdf9d913d","Type":"ContainerDied","Data":"85c7a136ce464b6f8700764aa53dbdfd546a7b693578b244276592364f469c3e"}
Sep 09 21:01:44.771922 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:44.771918    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk" event={"ID":"3250d655-294a-4aa8-8c71-3f8fdf9d913d","Type":"ContainerDied","Data":"c2342e57ae260855e3c265ac724d7f0190db5a5801253c9905d1796e3b57cc34"}
Sep 09 21:01:44.772028 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:44.771946    2581 scope.go:117] "RemoveContainer" containerID="85c7a136ce464b6f8700764aa53dbdfd546a7b693578b244276592364f469c3e"
Sep 09 21:01:44.782967 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:44.782947    2581 scope.go:117] "RemoveContainer" containerID="85c7a136ce464b6f8700764aa53dbdfd546a7b693578b244276592364f469c3e"
Sep 09 21:01:44.783286 ip-10-0-134-84 kubenswrapper[2581]: E0909 21:01:44.783265    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85c7a136ce464b6f8700764aa53dbdfd546a7b693578b244276592364f469c3e\": container with ID starting with 85c7a136ce464b6f8700764aa53dbdfd546a7b693578b244276592364f469c3e not found: ID does not exist" containerID="85c7a136ce464b6f8700764aa53dbdfd546a7b693578b244276592364f469c3e"
Sep 09 21:01:44.783381 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:44.783293    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85c7a136ce464b6f8700764aa53dbdfd546a7b693578b244276592364f469c3e"} err="failed to get container status \"85c7a136ce464b6f8700764aa53dbdfd546a7b693578b244276592364f469c3e\": rpc error: code = NotFound desc = could not find container \"85c7a136ce464b6f8700764aa53dbdfd546a7b693578b244276592364f469c3e\": container with ID starting with 85c7a136ce464b6f8700764aa53dbdfd546a7b693578b244276592364f469c3e not found: ID does not exist"
Sep 09 21:01:44.797581 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:44.797557    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk"]
Sep 09 21:01:44.802657 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:44.802636    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9q4gk"]
Sep 09 21:01:44.833082 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:44.833060    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tp6z\" (UniqueName: \"kubernetes.io/projected/3250d655-294a-4aa8-8c71-3f8fdf9d913d-kube-api-access-2tp6z\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:01:44.833153 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:44.833082    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3250d655-294a-4aa8-8c71-3f8fdf9d913d-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:01:46.635583 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:46.635542    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3250d655-294a-4aa8-8c71-3f8fdf9d913d" path="/var/lib/kubelet/pods/3250d655-294a-4aa8-8c71-3f8fdf9d913d/volumes"
Sep 09 21:01:47.887225 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:47.887184    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f" podUID="75a8d555-c86b-4ada-99cc-f6925b2f42e3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 09 21:01:57.887746 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:01:57.887709    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f" podUID="75a8d555-c86b-4ada-99cc-f6925b2f42e3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 09 21:02:07.887649 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:07.887611    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f" podUID="75a8d555-c86b-4ada-99cc-f6925b2f42e3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 09 21:02:17.887882 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:17.887840    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f" podUID="75a8d555-c86b-4ada-99cc-f6925b2f42e3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 09 21:02:27.897742 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:27.897707    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:02:27.905654 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:27.905628    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:02:32.585618 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:32.585583    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"]
Sep 09 21:02:47.609608 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:47.609562    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f" podUID="75a8d555-c86b-4ada-99cc-f6925b2f42e3" containerName="main" containerID="cri-o://73569eda6d722e667d6455aa2565d9a2050d8b40792dbf407980b4414cb1bdb4" gracePeriod=45
Sep 09 21:02:47.899223 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:47.899122    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f" podUID="75a8d555-c86b-4ada-99cc-f6925b2f42e3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 09 21:02:48.279258 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.279235    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:02:48.389037 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.389001    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75a8d555-c86b-4ada-99cc-f6925b2f42e3-cabundle-cert\") pod \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") "
Sep 09 21:02:48.389217 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.389061    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-model-cache\") pod \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") "
Sep 09 21:02:48.389217 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.389085    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hwdk\" (UniqueName: \"kubernetes.io/projected/75a8d555-c86b-4ada-99cc-f6925b2f42e3-kube-api-access-5hwdk\") pod \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") "
Sep 09 21:02:48.389217 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.389105    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-dshm\") pod \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") "
Sep 09 21:02:48.389217 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.389142    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-tmp-dir\") pod \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") "
Sep 09 21:02:48.389217 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.389197    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a8d555-c86b-4ada-99cc-f6925b2f42e3-tls-certs\") pod \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") "
Sep 09 21:02:48.389537 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.389228    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-home\") pod \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") "
Sep 09 21:02:48.389537 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.389254    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-kserve-provision-location\") pod \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\" (UID: \"75a8d555-c86b-4ada-99cc-f6925b2f42e3\") "
Sep 09 21:02:48.389537 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.389423    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a8d555-c86b-4ada-99cc-f6925b2f42e3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "75a8d555-c86b-4ada-99cc-f6925b2f42e3" (UID: "75a8d555-c86b-4ada-99cc-f6925b2f42e3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:02:48.389698 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.389657    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75a8d555-c86b-4ada-99cc-f6925b2f42e3-cabundle-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:02:48.389960 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.389937    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-model-cache" (OuterVolumeSpecName: "model-cache") pod "75a8d555-c86b-4ada-99cc-f6925b2f42e3" (UID: "75a8d555-c86b-4ada-99cc-f6925b2f42e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:02:48.390048 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.390019    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-home" (OuterVolumeSpecName: "home") pod "75a8d555-c86b-4ada-99cc-f6925b2f42e3" (UID: "75a8d555-c86b-4ada-99cc-f6925b2f42e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:02:48.391514 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.391478    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a8d555-c86b-4ada-99cc-f6925b2f42e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75a8d555-c86b-4ada-99cc-f6925b2f42e3" (UID: "75a8d555-c86b-4ada-99cc-f6925b2f42e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:02:48.391628 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.391527    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-dshm" (OuterVolumeSpecName: "dshm") pod "75a8d555-c86b-4ada-99cc-f6925b2f42e3" (UID: "75a8d555-c86b-4ada-99cc-f6925b2f42e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:02:48.391678 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.391665    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a8d555-c86b-4ada-99cc-f6925b2f42e3-kube-api-access-5hwdk" (OuterVolumeSpecName: "kube-api-access-5hwdk") pod "75a8d555-c86b-4ada-99cc-f6925b2f42e3" (UID: "75a8d555-c86b-4ada-99cc-f6925b2f42e3"). InnerVolumeSpecName "kube-api-access-5hwdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:02:48.407978 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.407952    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "75a8d555-c86b-4ada-99cc-f6925b2f42e3" (UID: "75a8d555-c86b-4ada-99cc-f6925b2f42e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:02:48.412329 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.412307    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "75a8d555-c86b-4ada-99cc-f6925b2f42e3" (UID: "75a8d555-c86b-4ada-99cc-f6925b2f42e3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:02:48.490636 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.490613    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:02:48.490780 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.490638    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:02:48.490780 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.490652    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a8d555-c86b-4ada-99cc-f6925b2f42e3-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:02:48.490780 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.490665    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:02:48.490780 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.490677    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-kserve-provision-location\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:02:48.490780 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.490688    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a8d555-c86b-4ada-99cc-f6925b2f42e3-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:02:48.490780 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:48.490701    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hwdk\" (UniqueName: \"kubernetes.io/projected/75a8d555-c86b-4ada-99cc-f6925b2f42e3-kube-api-access-5hwdk\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:02:49.105303 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:49.105273    2581 generic.go:356] "Generic (PLEG): container finished" podID="75a8d555-c86b-4ada-99cc-f6925b2f42e3" containerID="73569eda6d722e667d6455aa2565d9a2050d8b40792dbf407980b4414cb1bdb4" exitCode=0
Sep 09 21:02:49.105744 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:49.105323    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"
Sep 09 21:02:49.105744 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:49.105350    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f" event={"ID":"75a8d555-c86b-4ada-99cc-f6925b2f42e3","Type":"ContainerDied","Data":"73569eda6d722e667d6455aa2565d9a2050d8b40792dbf407980b4414cb1bdb4"}
Sep 09 21:02:49.105744 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:49.105400    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f" event={"ID":"75a8d555-c86b-4ada-99cc-f6925b2f42e3","Type":"ContainerDied","Data":"4b275828a02ebffa62ec0ab1e66b331d4283e244504023d35f15775847011968"}
Sep 09 21:02:49.105744 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:49.105419    2581 scope.go:117] "RemoveContainer" containerID="73569eda6d722e667d6455aa2565d9a2050d8b40792dbf407980b4414cb1bdb4"
Sep 09 21:02:49.113959 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:49.113943    2581 scope.go:117] "RemoveContainer" containerID="de2b0b015d264bfbf339cee0b42130e6d336217d4d651a2d773d2edecdc9345c"
Sep 09 21:02:49.121759 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:49.121738    2581 scope.go:117] "RemoveContainer" containerID="73569eda6d722e667d6455aa2565d9a2050d8b40792dbf407980b4414cb1bdb4"
Sep 09 21:02:49.122017 ip-10-0-134-84 kubenswrapper[2581]: E0909 21:02:49.122000    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73569eda6d722e667d6455aa2565d9a2050d8b40792dbf407980b4414cb1bdb4\": container with ID starting with 73569eda6d722e667d6455aa2565d9a2050d8b40792dbf407980b4414cb1bdb4 not found: ID does not exist" containerID="73569eda6d722e667d6455aa2565d9a2050d8b40792dbf407980b4414cb1bdb4"
Sep 09 21:02:49.122065 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:49.122026    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73569eda6d722e667d6455aa2565d9a2050d8b40792dbf407980b4414cb1bdb4"} err="failed to get container status \"73569eda6d722e667d6455aa2565d9a2050d8b40792dbf407980b4414cb1bdb4\": rpc error: code = NotFound desc = could not find container \"73569eda6d722e667d6455aa2565d9a2050d8b40792dbf407980b4414cb1bdb4\": container with ID starting with 73569eda6d722e667d6455aa2565d9a2050d8b40792dbf407980b4414cb1bdb4 not found: ID does not exist"
Sep 09 21:02:49.122065 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:49.122041    2581 scope.go:117] "RemoveContainer" containerID="de2b0b015d264bfbf339cee0b42130e6d336217d4d651a2d773d2edecdc9345c"
Sep 09 21:02:49.122290 ip-10-0-134-84 kubenswrapper[2581]: E0909 21:02:49.122271    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2b0b015d264bfbf339cee0b42130e6d336217d4d651a2d773d2edecdc9345c\": container with ID starting with de2b0b015d264bfbf339cee0b42130e6d336217d4d651a2d773d2edecdc9345c not found: ID does not exist" containerID="de2b0b015d264bfbf339cee0b42130e6d336217d4d651a2d773d2edecdc9345c"
Sep 09 21:02:49.122352 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:49.122299    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2b0b015d264bfbf339cee0b42130e6d336217d4d651a2d773d2edecdc9345c"} err="failed to get container status \"de2b0b015d264bfbf339cee0b42130e6d336217d4d651a2d773d2edecdc9345c\": rpc error: code = NotFound desc = could not find container \"de2b0b015d264bfbf339cee0b42130e6d336217d4d651a2d773d2edecdc9345c\": container with ID starting with de2b0b015d264bfbf339cee0b42130e6d336217d4d651a2d773d2edecdc9345c not found: ID does not exist"
Sep 09 21:02:49.128263 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:49.128242    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"]
Sep 09 21:02:49.134095 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:49.134073    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76f5cbbd7clcw7f"]
Sep 09 21:02:50.635592 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:02:50.635560    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75a8d555-c86b-4ada-99cc-f6925b2f42e3" path="/var/lib/kubelet/pods/75a8d555-c86b-4ada-99cc-f6925b2f42e3/volumes"
Sep 09 21:03:32.178643 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.178551    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"]
Sep 09 21:03:32.179181 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.179118    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75a8d555-c86b-4ada-99cc-f6925b2f42e3" containerName="main"
Sep 09 21:03:32.179181 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.179137    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="75a8d555-c86b-4ada-99cc-f6925b2f42e3" containerName="main"
Sep 09 21:03:32.179181 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.179155    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3250d655-294a-4aa8-8c71-3f8fdf9d913d" containerName="main"
Sep 09 21:03:32.179181 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.179161    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3250d655-294a-4aa8-8c71-3f8fdf9d913d" containerName="main"
Sep 09 21:03:32.179181 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.179183    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75a8d555-c86b-4ada-99cc-f6925b2f42e3" containerName="storage-initializer"
Sep 09 21:03:32.179363 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.179190    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="75a8d555-c86b-4ada-99cc-f6925b2f42e3" containerName="storage-initializer"
Sep 09 21:03:32.179363 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.179247    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3250d655-294a-4aa8-8c71-3f8fdf9d913d" containerName="main"
Sep 09 21:03:32.179363 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.179258    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="75a8d555-c86b-4ada-99cc-f6925b2f42e3" containerName="main"
Sep 09 21:03:32.182539 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.182523    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:32.185630 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.185603    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-b9xtq\""
Sep 09 21:03:32.185741 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.185603    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 09 21:03:32.185741 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.185645    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 09 21:03:32.186392 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.186373    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 09 21:03:32.200825 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.200800    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"]
Sep 09 21:03:32.250625 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.250594    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:32.250625 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.250627    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bz95\" (UniqueName: \"kubernetes.io/projected/29750413-775b-4dbe-a8d0-00350a2878fc-kube-api-access-5bz95\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:32.250826 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.250657    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:32.250826 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.250673    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29750413-775b-4dbe-a8d0-00350a2878fc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:32.250826 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.250728    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:32.250826 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.250775    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:32.250826 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.250799    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:32.351519 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.351486    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:32.351718 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.351527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bz95\" (UniqueName: \"kubernetes.io/projected/29750413-775b-4dbe-a8d0-00350a2878fc-kube-api-access-5bz95\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:32.351718 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.351564    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:32.351718 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.351587    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29750413-775b-4dbe-a8d0-00350a2878fc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:32.351718 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.351634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:32.351718 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.351683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:32.351718 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.351718    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:32.352043 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.351942    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:32.352043 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.351968    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:32.352043 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.352032    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:32.352150 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.352118    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:32.353930 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.353899    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:32.354161 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.354145    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29750413-775b-4dbe-a8d0-00350a2878fc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:32.361106 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.361081    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bz95\" (UniqueName: \"kubernetes.io/projected/29750413-775b-4dbe-a8d0-00350a2878fc-kube-api-access-5bz95\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:32.493050 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.493017    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:32.772637 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.772604    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"]
Sep 09 21:03:32.819320 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:32.819302    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:03:33.319278 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:33.319236    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t" event={"ID":"29750413-775b-4dbe-a8d0-00350a2878fc","Type":"ContainerStarted","Data":"8e2aa60b1293357bc25ddc15a993001e41012af68302ceb20b9bf19120ce2c24"}
Sep 09 21:03:41.362986 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:41.362948    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t" event={"ID":"29750413-775b-4dbe-a8d0-00350a2878fc","Type":"ContainerStarted","Data":"83c895941236fd5a58490e0ee53ce744207be05b943e10203f916db92ac9a2d1"}
Sep 09 21:03:42.368925 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:42.368889    2581 generic.go:356] "Generic (PLEG): container finished" podID="29750413-775b-4dbe-a8d0-00350a2878fc" containerID="83c895941236fd5a58490e0ee53ce744207be05b943e10203f916db92ac9a2d1" exitCode=0
Sep 09 21:03:42.369295 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:42.368976    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t" event={"ID":"29750413-775b-4dbe-a8d0-00350a2878fc","Type":"ContainerDied","Data":"83c895941236fd5a58490e0ee53ce744207be05b943e10203f916db92ac9a2d1"}
Sep 09 21:03:43.376481 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:43.376363    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t" event={"ID":"29750413-775b-4dbe-a8d0-00350a2878fc","Type":"ContainerStarted","Data":"3a0b1605105c6a244e96a9f9fbdf8bd7391061c67f995a9404ec63e412c1eeda"}
Sep 09 21:03:43.376481 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:43.376432    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t" event={"ID":"29750413-775b-4dbe-a8d0-00350a2878fc","Type":"ContainerStarted","Data":"e7d8e6ce9077e1334b6ef6604dc99665271aac4a4d5eba0f7381e83a6c8a96cc"}
Sep 09 21:03:43.403805 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:43.403754    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t" podStartSLOduration=2.43567263 podStartE2EDuration="11.403725542s" totalImagesPullingTime="7.780789048s" totalInitContainerRuntime="1.187263864s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:03:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:03:43.400719541 +0000 UTC m=+3165.405896536" watchObservedRunningTime="2026-09-09 21:03:43.403725542 +0000 UTC m=+3165.408902537"
Sep 09 21:03:43.494169 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:43.494125    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:52.493530 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:52.493486    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:03:52.495248 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:03:52.495217    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t" podUID="29750413-775b-4dbe-a8d0-00350a2878fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 09 21:04:02.493853 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:04:02.493801    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t" podUID="29750413-775b-4dbe-a8d0-00350a2878fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 09 21:04:12.493763 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:04:12.493713    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t" podUID="29750413-775b-4dbe-a8d0-00350a2878fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 09 21:04:22.494143 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:04:22.494087    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t" podUID="29750413-775b-4dbe-a8d0-00350a2878fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 09 21:04:32.494168 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:04:32.494121    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t" podUID="29750413-775b-4dbe-a8d0-00350a2878fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 09 21:04:42.493651 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:04:42.493608    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t" podUID="29750413-775b-4dbe-a8d0-00350a2878fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 09 21:04:52.494364 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:04:52.494265    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t" podUID="29750413-775b-4dbe-a8d0-00350a2878fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 09 21:05:02.493996 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:05:02.493937    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t" podUID="29750413-775b-4dbe-a8d0-00350a2878fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 09 21:05:12.494218 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:05:12.494171    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t" podUID="29750413-775b-4dbe-a8d0-00350a2878fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 09 21:05:22.494481 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:05:22.494417    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t" podUID="29750413-775b-4dbe-a8d0-00350a2878fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 09 21:05:32.493928 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:05:32.493884    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t" podUID="29750413-775b-4dbe-a8d0-00350a2878fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 09 21:05:42.504308 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:05:42.504273    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:05:42.511942 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:05:42.511916    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:05:47.528545 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:05:47.528507    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"]
Sep 09 21:05:47.529007 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:05:47.528839    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t" podUID="29750413-775b-4dbe-a8d0-00350a2878fc" containerName="modelcar" containerID="cri-o://3a0b1605105c6a244e96a9f9fbdf8bd7391061c67f995a9404ec63e412c1eeda" gracePeriod=60
Sep 09 21:05:48.034357 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:05:48.034329    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t_29750413-775b-4dbe-a8d0-00350a2878fc/modelcar/0.log"
Sep 09 21:05:48.035190 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:05:48.035162    2581 generic.go:356] "Generic (PLEG): container finished" podID="29750413-775b-4dbe-a8d0-00350a2878fc" containerID="3a0b1605105c6a244e96a9f9fbdf8bd7391061c67f995a9404ec63e412c1eeda" exitCode=143
Sep 09 21:05:48.035296 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:05:48.035240    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t" event={"ID":"29750413-775b-4dbe-a8d0-00350a2878fc","Type":"ContainerDied","Data":"3a0b1605105c6a244e96a9f9fbdf8bd7391061c67f995a9404ec63e412c1eeda"}
Sep 09 21:05:58.748484 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:05:58.748359    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t_29750413-775b-4dbe-a8d0-00350a2878fc/modelcar/0.log"
Sep 09 21:05:58.757810 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:05:58.757789    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t_29750413-775b-4dbe-a8d0-00350a2878fc/modelcar/0.log"
Sep 09 21:05:58.765242 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:05:58.765222    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 21:05:58.768300 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:05:58.768280    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 21:05:58.773618 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:05:58.773594    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 21:05:58.776316 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:05:58.776299    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 21:06:02.553319 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:02.553273    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t" podUID="29750413-775b-4dbe-a8d0-00350a2878fc" containerName="main" containerID="cri-o://e7d8e6ce9077e1334b6ef6604dc99665271aac4a4d5eba0f7381e83a6c8a96cc" gracePeriod=45
Sep 09 21:06:03.113445 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.113420    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t_29750413-775b-4dbe-a8d0-00350a2878fc/modelcar/0.log"
Sep 09 21:06:03.114106 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.114082    2581 generic.go:356] "Generic (PLEG): container finished" podID="29750413-775b-4dbe-a8d0-00350a2878fc" containerID="e7d8e6ce9077e1334b6ef6604dc99665271aac4a4d5eba0f7381e83a6c8a96cc" exitCode=0
Sep 09 21:06:03.114202 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.114144    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t" event={"ID":"29750413-775b-4dbe-a8d0-00350a2878fc","Type":"ContainerDied","Data":"e7d8e6ce9077e1334b6ef6604dc99665271aac4a4d5eba0f7381e83a6c8a96cc"}
Sep 09 21:06:03.504896 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.504853    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t" podUID="29750413-775b-4dbe-a8d0-00350a2878fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 09 21:06:03.807302 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.807276    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t_29750413-775b-4dbe-a8d0-00350a2878fc/modelcar/0.log"
Sep 09 21:06:03.810366 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.810344    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:06:03.857089 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.857053    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-home\") pod \"29750413-775b-4dbe-a8d0-00350a2878fc\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") "
Sep 09 21:06:03.857089 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.857092    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bz95\" (UniqueName: \"kubernetes.io/projected/29750413-775b-4dbe-a8d0-00350a2878fc-kube-api-access-5bz95\") pod \"29750413-775b-4dbe-a8d0-00350a2878fc\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") "
Sep 09 21:06:03.857323 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.857154    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-dshm\") pod \"29750413-775b-4dbe-a8d0-00350a2878fc\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") "
Sep 09 21:06:03.857323 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.857182    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-model-cache\") pod \"29750413-775b-4dbe-a8d0-00350a2878fc\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") "
Sep 09 21:06:03.857323 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.857204    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29750413-775b-4dbe-a8d0-00350a2878fc-tls-certs\") pod \"29750413-775b-4dbe-a8d0-00350a2878fc\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") "
Sep 09 21:06:03.857323 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.857226    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-kserve-provision-location\") pod \"29750413-775b-4dbe-a8d0-00350a2878fc\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") "
Sep 09 21:06:03.857323 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.857258    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-tmp-dir\") pod \"29750413-775b-4dbe-a8d0-00350a2878fc\" (UID: \"29750413-775b-4dbe-a8d0-00350a2878fc\") "
Sep 09 21:06:03.857623 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.857566    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "29750413-775b-4dbe-a8d0-00350a2878fc" (UID: "29750413-775b-4dbe-a8d0-00350a2878fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:03.857779 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.857730    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "29750413-775b-4dbe-a8d0-00350a2878fc" (UID: "29750413-775b-4dbe-a8d0-00350a2878fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:03.857892 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.857789    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-home" (OuterVolumeSpecName: "home") pod "29750413-775b-4dbe-a8d0-00350a2878fc" (UID: "29750413-775b-4dbe-a8d0-00350a2878fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:03.859619 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.859594    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-dshm" (OuterVolumeSpecName: "dshm") pod "29750413-775b-4dbe-a8d0-00350a2878fc" (UID: "29750413-775b-4dbe-a8d0-00350a2878fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:03.859718 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.859695    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29750413-775b-4dbe-a8d0-00350a2878fc-kube-api-access-5bz95" (OuterVolumeSpecName: "kube-api-access-5bz95") pod "29750413-775b-4dbe-a8d0-00350a2878fc" (UID: "29750413-775b-4dbe-a8d0-00350a2878fc"). InnerVolumeSpecName "kube-api-access-5bz95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:06:03.859790 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.859733    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29750413-775b-4dbe-a8d0-00350a2878fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29750413-775b-4dbe-a8d0-00350a2878fc" (UID: "29750413-775b-4dbe-a8d0-00350a2878fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:06:03.869248 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.869225    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "29750413-775b-4dbe-a8d0-00350a2878fc" (UID: "29750413-775b-4dbe-a8d0-00350a2878fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:03.958379 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.958331    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:06:03.958379 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.958370    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:06:03.958379 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.958382    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29750413-775b-4dbe-a8d0-00350a2878fc-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:06:03.958379 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.958391    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-kserve-provision-location\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:06:03.958695 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.958400    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:06:03.958695 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.958408    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29750413-775b-4dbe-a8d0-00350a2878fc-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:06:03.958695 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:03.958416    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bz95\" (UniqueName: \"kubernetes.io/projected/29750413-775b-4dbe-a8d0-00350a2878fc-kube-api-access-5bz95\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:06:04.120537 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:04.120432    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t_29750413-775b-4dbe-a8d0-00350a2878fc/modelcar/0.log"
Sep 09 21:06:04.121192 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:04.121171    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"
Sep 09 21:06:04.121244 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:04.121182    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t" event={"ID":"29750413-775b-4dbe-a8d0-00350a2878fc","Type":"ContainerDied","Data":"8e2aa60b1293357bc25ddc15a993001e41012af68302ceb20b9bf19120ce2c24"}
Sep 09 21:06:04.121282 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:04.121247    2581 scope.go:117] "RemoveContainer" containerID="3a0b1605105c6a244e96a9f9fbdf8bd7391061c67f995a9404ec63e412c1eeda"
Sep 09 21:06:04.130593 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:04.130574    2581 scope.go:117] "RemoveContainer" containerID="e7d8e6ce9077e1334b6ef6604dc99665271aac4a4d5eba0f7381e83a6c8a96cc"
Sep 09 21:06:04.140571 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:04.140542    2581 scope.go:117] "RemoveContainer" containerID="83c895941236fd5a58490e0ee53ce744207be05b943e10203f916db92ac9a2d1"
Sep 09 21:06:04.145427 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:04.145406    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"]
Sep 09 21:06:04.151048 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:04.151020    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fz7m4t"]
Sep 09 21:06:04.636482 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:04.636434    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29750413-775b-4dbe-a8d0-00350a2878fc" path="/var/lib/kubelet/pods/29750413-775b-4dbe-a8d0-00350a2878fc/volumes"
Sep 09 21:06:51.753747 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.753668    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"]
Sep 09 21:06:51.754186 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.753987    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29750413-775b-4dbe-a8d0-00350a2878fc" containerName="main"
Sep 09 21:06:51.754186 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.753997    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="29750413-775b-4dbe-a8d0-00350a2878fc" containerName="main"
Sep 09 21:06:51.754186 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.754005    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29750413-775b-4dbe-a8d0-00350a2878fc" containerName="modelcar"
Sep 09 21:06:51.754186 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.754011    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="29750413-775b-4dbe-a8d0-00350a2878fc" containerName="modelcar"
Sep 09 21:06:51.754186 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.754022    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29750413-775b-4dbe-a8d0-00350a2878fc" containerName="modelcar-init"
Sep 09 21:06:51.754186 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.754028    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="29750413-775b-4dbe-a8d0-00350a2878fc" containerName="modelcar-init"
Sep 09 21:06:51.754186 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.754094    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="29750413-775b-4dbe-a8d0-00350a2878fc" containerName="main"
Sep 09 21:06:51.754186 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.754102    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="29750413-775b-4dbe-a8d0-00350a2878fc" containerName="modelcar"
Sep 09 21:06:51.757085 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.757068    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:51.759654 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.759568    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-c2fdl\""
Sep 09 21:06:51.759654 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.759605    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 09 21:06:51.759654 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.759622    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:06:51.759654 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.759642    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 09 21:06:51.759955 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.759604    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 09 21:06:51.759955 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.759612    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 09 21:06:51.769779 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.769759    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"]
Sep 09 21:06:51.850609 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.850571    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7455c584fb-8nmqc\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:51.850609 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.850607    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-home\") pod \"custom-route-timeout-test-kserve-7455c584fb-8nmqc\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:51.850827 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.850623    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-dshm\") pod \"custom-route-timeout-test-kserve-7455c584fb-8nmqc\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:51.850827 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.850641    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-model-cache\") pod \"custom-route-timeout-test-kserve-7455c584fb-8nmqc\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:51.850827 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.850662    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-tmp-dir\") pod \"custom-route-timeout-test-kserve-7455c584fb-8nmqc\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:51.850827 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.850679    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee22c9dc-b24d-4412-b4c4-547243142a04-tls-certs\") pod \"custom-route-timeout-test-kserve-7455c584fb-8nmqc\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:51.850827 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.850695    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee22c9dc-b24d-4412-b4c4-547243142a04-cabundle-cert\") pod \"custom-route-timeout-test-kserve-7455c584fb-8nmqc\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:51.850827 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.850720    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5wlv\" (UniqueName: \"kubernetes.io/projected/ee22c9dc-b24d-4412-b4c4-547243142a04-kube-api-access-p5wlv\") pod \"custom-route-timeout-test-kserve-7455c584fb-8nmqc\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:51.951758 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.951720    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5wlv\" (UniqueName: \"kubernetes.io/projected/ee22c9dc-b24d-4412-b4c4-547243142a04-kube-api-access-p5wlv\") pod \"custom-route-timeout-test-kserve-7455c584fb-8nmqc\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:51.951955 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.951866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7455c584fb-8nmqc\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:51.951955 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.951909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-home\") pod \"custom-route-timeout-test-kserve-7455c584fb-8nmqc\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:51.952063 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.951977    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-dshm\") pod \"custom-route-timeout-test-kserve-7455c584fb-8nmqc\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:51.952063 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.952002    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-model-cache\") pod \"custom-route-timeout-test-kserve-7455c584fb-8nmqc\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:51.952063 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.952044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-tmp-dir\") pod \"custom-route-timeout-test-kserve-7455c584fb-8nmqc\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:51.952218 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.952070    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee22c9dc-b24d-4412-b4c4-547243142a04-tls-certs\") pod \"custom-route-timeout-test-kserve-7455c584fb-8nmqc\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:51.952218 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.952102    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee22c9dc-b24d-4412-b4c4-547243142a04-cabundle-cert\") pod \"custom-route-timeout-test-kserve-7455c584fb-8nmqc\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:51.952382 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.952355    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7455c584fb-8nmqc\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:51.952476 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.952381    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-model-cache\") pod \"custom-route-timeout-test-kserve-7455c584fb-8nmqc\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:51.952653 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.952627    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-home\") pod \"custom-route-timeout-test-kserve-7455c584fb-8nmqc\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:51.952745 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.952682    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-tmp-dir\") pod \"custom-route-timeout-test-kserve-7455c584fb-8nmqc\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:51.952914 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.952892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee22c9dc-b24d-4412-b4c4-547243142a04-cabundle-cert\") pod \"custom-route-timeout-test-kserve-7455c584fb-8nmqc\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:51.954823 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.954801    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-dshm\") pod \"custom-route-timeout-test-kserve-7455c584fb-8nmqc\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:51.955121 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.955097    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee22c9dc-b24d-4412-b4c4-547243142a04-tls-certs\") pod \"custom-route-timeout-test-kserve-7455c584fb-8nmqc\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:51.961011 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:51.960988    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5wlv\" (UniqueName: \"kubernetes.io/projected/ee22c9dc-b24d-4412-b4c4-547243142a04-kube-api-access-p5wlv\") pod \"custom-route-timeout-test-kserve-7455c584fb-8nmqc\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:52.066823 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:52.066736    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:06:52.351926 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:52.351827    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"]
Sep 09 21:06:52.354276 ip-10-0-134-84 kubenswrapper[2581]: W0909 21:06:52.354249    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee22c9dc_b24d_4412_b4c4_547243142a04.slice/crio-441b21f817e310816cb6be419bf9ca0f6dc6a0f92e14cd6a2c449a6e7a3cb569 WatchSource:0}: Error finding container 441b21f817e310816cb6be419bf9ca0f6dc6a0f92e14cd6a2c449a6e7a3cb569: Status 404 returned error can't find the container with id 441b21f817e310816cb6be419bf9ca0f6dc6a0f92e14cd6a2c449a6e7a3cb569
Sep 09 21:06:53.358340 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:53.358299    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc" event={"ID":"ee22c9dc-b24d-4412-b4c4-547243142a04","Type":"ContainerStarted","Data":"990e6d75638145e13029a673c7b2e17fc2e8a0dd712594a79f2b88b87f538191"}
Sep 09 21:06:53.358740 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:53.358358    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc" event={"ID":"ee22c9dc-b24d-4412-b4c4-547243142a04","Type":"ContainerStarted","Data":"441b21f817e310816cb6be419bf9ca0f6dc6a0f92e14cd6a2c449a6e7a3cb569"}
Sep 09 21:06:55.371758 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:55.371722    2581 generic.go:356] "Generic (PLEG): container finished" podID="ee22c9dc-b24d-4412-b4c4-547243142a04" containerID="990e6d75638145e13029a673c7b2e17fc2e8a0dd712594a79f2b88b87f538191" exitCode=0
Sep 09 21:06:55.371758 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:55.371751    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc" event={"ID":"ee22c9dc-b24d-4412-b4c4-547243142a04","Type":"ContainerDied","Data":"990e6d75638145e13029a673c7b2e17fc2e8a0dd712594a79f2b88b87f538191"}
Sep 09 21:06:56.379526 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:56.379493    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc" event={"ID":"ee22c9dc-b24d-4412-b4c4-547243142a04","Type":"ContainerStarted","Data":"aeb8a9d33959958560141102704e695628884ef1af35e6e4263b62015d427b4b"}
Sep 09 21:06:56.407732 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:56.407655    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc" podStartSLOduration=3.024568353 podStartE2EDuration="5.407634752s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.383066399s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:06:56.404500502 +0000 UTC m=+3358.409677494" watchObservedRunningTime="2026-09-09 21:06:56.407634752 +0000 UTC m=+3358.412811749"
Sep 09 21:06:57.067810 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:06:57.067776    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:07:02.067009 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:07:02.066968    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:07:02.068722 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:07:02.068692    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc" podUID="ee22c9dc-b24d-4412-b4c4-547243142a04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 09 21:07:12.067115 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:07:12.067071    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc" podUID="ee22c9dc-b24d-4412-b4c4-547243142a04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 09 21:07:22.067200 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:07:22.067152    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc" podUID="ee22c9dc-b24d-4412-b4c4-547243142a04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 09 21:07:32.067479 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:07:32.067412    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc" podUID="ee22c9dc-b24d-4412-b4c4-547243142a04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 09 21:07:42.067159 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:07:42.067114    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc" podUID="ee22c9dc-b24d-4412-b4c4-547243142a04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 09 21:07:52.067440 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:07:52.067347    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc" podUID="ee22c9dc-b24d-4412-b4c4-547243142a04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 09 21:08:02.067413 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:02.067361    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc" podUID="ee22c9dc-b24d-4412-b4c4-547243142a04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 09 21:08:12.067519 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:12.067441    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc" podUID="ee22c9dc-b24d-4412-b4c4-547243142a04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 09 21:08:22.067646 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:22.067595    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc" podUID="ee22c9dc-b24d-4412-b4c4-547243142a04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 09 21:08:29.869240 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:29.869206    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z"]
Sep 09 21:08:29.872990 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:29.872958    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z"
Sep 09 21:08:29.877163 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:29.876914    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-ctg7s\""
Sep 09 21:08:29.877163 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:29.876992    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-epp-sa-dockercfg-xmqgv\""
Sep 09 21:08:29.877163 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:29.877030    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-kserve-self-signed-certs\""
Sep 09 21:08:29.877163 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:29.877062    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Sep 09 21:08:29.877163 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:29.877152    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Sep 09 21:08:29.883718 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:29.883656    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z"]
Sep 09 21:08:29.910869 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:29.910836    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d554e31-e24c-415d-aa0d-f76a411d87f8-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z\" (UID: \"4d554e31-e24c-415d-aa0d-f76a411d87f8\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z"
Sep 09 21:08:29.911050 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:29.910886    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ppfc\" (UniqueName: \"kubernetes.io/projected/4d554e31-e24c-415d-aa0d-f76a411d87f8-kube-api-access-5ppfc\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z\" (UID: \"4d554e31-e24c-415d-aa0d-f76a411d87f8\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z"
Sep 09 21:08:30.012355 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:30.012314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d554e31-e24c-415d-aa0d-f76a411d87f8-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z\" (UID: \"4d554e31-e24c-415d-aa0d-f76a411d87f8\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z"
Sep 09 21:08:30.012590 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:30.012377    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ppfc\" (UniqueName: \"kubernetes.io/projected/4d554e31-e24c-415d-aa0d-f76a411d87f8-kube-api-access-5ppfc\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z\" (UID: \"4d554e31-e24c-415d-aa0d-f76a411d87f8\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z"
Sep 09 21:08:30.014925 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:30.014901    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d554e31-e24c-415d-aa0d-f76a411d87f8-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z\" (UID: \"4d554e31-e24c-415d-aa0d-f76a411d87f8\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z"
Sep 09 21:08:30.027232 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:30.027193    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ppfc\" (UniqueName: \"kubernetes.io/projected/4d554e31-e24c-415d-aa0d-f76a411d87f8-kube-api-access-5ppfc\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z\" (UID: \"4d554e31-e24c-415d-aa0d-f76a411d87f8\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z"
Sep 09 21:08:30.186399 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:30.186299    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z"
Sep 09 21:08:30.469433 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:30.469407    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z"]
Sep 09 21:08:30.471145 ip-10-0-134-84 kubenswrapper[2581]: W0909 21:08:30.471115    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d554e31_e24c_415d_aa0d_f76a411d87f8.slice/crio-59ebcb779644a4a02c0c9c70044bc9b90c091f99e4b8af4a486f0b6b8ddb1fa9 WatchSource:0}: Error finding container 59ebcb779644a4a02c0c9c70044bc9b90c091f99e4b8af4a486f0b6b8ddb1fa9: Status 404 returned error can't find the container with id 59ebcb779644a4a02c0c9c70044bc9b90c091f99e4b8af4a486f0b6b8ddb1fa9
Sep 09 21:08:30.863842 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:30.863805    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z" event={"ID":"4d554e31-e24c-415d-aa0d-f76a411d87f8","Type":"ContainerStarted","Data":"b1afa331e0b955af36f186242926e4dd4e87348ac3d56883f27ae202fb93c024"}
Sep 09 21:08:30.864046 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:30.863856    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z" event={"ID":"4d554e31-e24c-415d-aa0d-f76a411d87f8","Type":"ContainerStarted","Data":"59ebcb779644a4a02c0c9c70044bc9b90c091f99e4b8af4a486f0b6b8ddb1fa9"}
Sep 09 21:08:30.864046 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:30.863887    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z"
Sep 09 21:08:30.892641 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:30.892592    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z" podStartSLOduration=1.892575919 podStartE2EDuration="1.892575919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:08:30.890000421 +0000 UTC m=+3452.895177418" watchObservedRunningTime="2026-09-09 21:08:30.892575919 +0000 UTC m=+3452.897752914"
Sep 09 21:08:32.067040 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:32.066993    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc" podUID="ee22c9dc-b24d-4412-b4c4-547243142a04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 09 21:08:32.619585 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:32.619545    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z"]
Sep 09 21:08:40.879085 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:40.879039    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k"]
Sep 09 21:08:40.885943 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:40.885916    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k"
Sep 09 21:08:40.893629 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:40.893597    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-ww8kj\""
Sep 09 21:08:40.894422 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:40.894393    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-n78zb\""
Sep 09 21:08:40.894811 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:40.894787    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 09 21:08:40.895009 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:40.894842    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 09 21:08:40.895139 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:40.894926    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 09 21:08:40.897269 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:40.897246    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k"]
Sep 09 21:08:41.006922 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:41.006886    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dab98b5-26fb-4b50-bf77-2d50dd6245d2-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k\" (UID: \"5dab98b5-26fb-4b50-bf77-2d50dd6245d2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k"
Sep 09 21:08:41.007122 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:41.006952    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wljn5\" (UniqueName: \"kubernetes.io/projected/5dab98b5-26fb-4b50-bf77-2d50dd6245d2-kube-api-access-wljn5\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k\" (UID: \"5dab98b5-26fb-4b50-bf77-2d50dd6245d2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k"
Sep 09 21:08:41.108363 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:41.108313    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dab98b5-26fb-4b50-bf77-2d50dd6245d2-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k\" (UID: \"5dab98b5-26fb-4b50-bf77-2d50dd6245d2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k"
Sep 09 21:08:41.108613 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:41.108431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wljn5\" (UniqueName: \"kubernetes.io/projected/5dab98b5-26fb-4b50-bf77-2d50dd6245d2-kube-api-access-wljn5\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k\" (UID: \"5dab98b5-26fb-4b50-bf77-2d50dd6245d2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k"
Sep 09 21:08:41.110951 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:41.110923    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dab98b5-26fb-4b50-bf77-2d50dd6245d2-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k\" (UID: \"5dab98b5-26fb-4b50-bf77-2d50dd6245d2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k"
Sep 09 21:08:41.120908 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:41.120870    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wljn5\" (UniqueName: \"kubernetes.io/projected/5dab98b5-26fb-4b50-bf77-2d50dd6245d2-kube-api-access-wljn5\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k\" (UID: \"5dab98b5-26fb-4b50-bf77-2d50dd6245d2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k"
Sep 09 21:08:41.198632 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:41.198527    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k"
Sep 09 21:08:41.496136 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:41.496098    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k"]
Sep 09 21:08:41.496356 ip-10-0-134-84 kubenswrapper[2581]: W0909 21:08:41.496326    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dab98b5_26fb_4b50_bf77_2d50dd6245d2.slice/crio-1dfe5b7154db0d03a3011921b0d5b9881ff78c4354f8e83e3550b1dd17d488dd WatchSource:0}: Error finding container 1dfe5b7154db0d03a3011921b0d5b9881ff78c4354f8e83e3550b1dd17d488dd: Status 404 returned error can't find the container with id 1dfe5b7154db0d03a3011921b0d5b9881ff78c4354f8e83e3550b1dd17d488dd
Sep 09 21:08:41.927735 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:41.927635    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k" event={"ID":"5dab98b5-26fb-4b50-bf77-2d50dd6245d2","Type":"ContainerStarted","Data":"39486848181c90f814e69df047653779efaa1c5d53488cd79022f9b2f93cde38"}
Sep 09 21:08:41.927735 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:41.927689    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k" event={"ID":"5dab98b5-26fb-4b50-bf77-2d50dd6245d2","Type":"ContainerStarted","Data":"1dfe5b7154db0d03a3011921b0d5b9881ff78c4354f8e83e3550b1dd17d488dd"}
Sep 09 21:08:41.928255 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:41.927733    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k"
Sep 09 21:08:41.960013 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:41.959928    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k" podStartSLOduration=1.95991063 podStartE2EDuration="1.95991063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:08:41.959045741 +0000 UTC m=+3463.964222736" watchObservedRunningTime="2026-09-09 21:08:41.95991063 +0000 UTC m=+3463.965087647"
Sep 09 21:08:42.067708 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:42.067662    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc" podUID="ee22c9dc-b24d-4412-b4c4-547243142a04" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 09 21:08:47.897403 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:47.897352    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z" podUID="4d554e31-e24c-415d-aa0d-f76a411d87f8" containerName="main" containerID="cri-o://b1afa331e0b955af36f186242926e4dd4e87348ac3d56883f27ae202fb93c024" gracePeriod=45
Sep 09 21:08:52.077656 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:52.077619    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:08:52.085262 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:52.085237    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:08:57.209385 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:08:57.209352    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"]
Sep 09 21:09:02.878364 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:02.878328    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z" podUID="4d554e31-e24c-415d-aa0d-f76a411d87f8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:09:12.232474 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:12.232424    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc" podUID="ee22c9dc-b24d-4412-b4c4-547243142a04" containerName="main" containerID="cri-o://aeb8a9d33959958560141102704e695628884ef1af35e6e4263b62015d427b4b" gracePeriod=45
Sep 09 21:09:12.877179 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:12.877139    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z" podUID="4d554e31-e24c-415d-aa0d-f76a411d87f8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:09:12.916189 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:12.916167    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:09:12.934633 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:12.934610    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k"
Sep 09 21:09:13.092065 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.091975    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5wlv\" (UniqueName: \"kubernetes.io/projected/ee22c9dc-b24d-4412-b4c4-547243142a04-kube-api-access-p5wlv\") pod \"ee22c9dc-b24d-4412-b4c4-547243142a04\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") "
Sep 09 21:09:13.092065 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.092019    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-model-cache\") pod \"ee22c9dc-b24d-4412-b4c4-547243142a04\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") "
Sep 09 21:09:13.092065 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.092040    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-home\") pod \"ee22c9dc-b24d-4412-b4c4-547243142a04\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") "
Sep 09 21:09:13.092339 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.092067    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-dshm\") pod \"ee22c9dc-b24d-4412-b4c4-547243142a04\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") "
Sep 09 21:09:13.092339 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.092103    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-kserve-provision-location\") pod \"ee22c9dc-b24d-4412-b4c4-547243142a04\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") "
Sep 09 21:09:13.092339 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.092140    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee22c9dc-b24d-4412-b4c4-547243142a04-cabundle-cert\") pod \"ee22c9dc-b24d-4412-b4c4-547243142a04\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") "
Sep 09 21:09:13.092339 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.092202    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-tmp-dir\") pod \"ee22c9dc-b24d-4412-b4c4-547243142a04\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") "
Sep 09 21:09:13.092339 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.092259    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee22c9dc-b24d-4412-b4c4-547243142a04-tls-certs\") pod \"ee22c9dc-b24d-4412-b4c4-547243142a04\" (UID: \"ee22c9dc-b24d-4412-b4c4-547243142a04\") "
Sep 09 21:09:13.092339 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.092289    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee22c9dc-b24d-4412-b4c4-547243142a04" (UID: "ee22c9dc-b24d-4412-b4c4-547243142a04"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:13.092701 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.092598    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:09:13.092977 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.092948    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-home" (OuterVolumeSpecName: "home") pod "ee22c9dc-b24d-4412-b4c4-547243142a04" (UID: "ee22c9dc-b24d-4412-b4c4-547243142a04"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:13.093121 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.093077    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee22c9dc-b24d-4412-b4c4-547243142a04-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ee22c9dc-b24d-4412-b4c4-547243142a04" (UID: "ee22c9dc-b24d-4412-b4c4-547243142a04"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:09:13.094822 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.094798    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee22c9dc-b24d-4412-b4c4-547243142a04-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee22c9dc-b24d-4412-b4c4-547243142a04" (UID: "ee22c9dc-b24d-4412-b4c4-547243142a04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:09:13.094945 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.094919    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-dshm" (OuterVolumeSpecName: "dshm") pod "ee22c9dc-b24d-4412-b4c4-547243142a04" (UID: "ee22c9dc-b24d-4412-b4c4-547243142a04"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:13.095621 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.095588    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee22c9dc-b24d-4412-b4c4-547243142a04-kube-api-access-p5wlv" (OuterVolumeSpecName: "kube-api-access-p5wlv") pod "ee22c9dc-b24d-4412-b4c4-547243142a04" (UID: "ee22c9dc-b24d-4412-b4c4-547243142a04"). InnerVolumeSpecName "kube-api-access-p5wlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:13.097936 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.097826    2581 generic.go:356] "Generic (PLEG): container finished" podID="ee22c9dc-b24d-4412-b4c4-547243142a04" containerID="aeb8a9d33959958560141102704e695628884ef1af35e6e4263b62015d427b4b" exitCode=0
Sep 09 21:09:13.097936 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.097903    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"
Sep 09 21:09:13.097936 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.097914    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc" event={"ID":"ee22c9dc-b24d-4412-b4c4-547243142a04","Type":"ContainerDied","Data":"aeb8a9d33959958560141102704e695628884ef1af35e6e4263b62015d427b4b"}
Sep 09 21:09:13.098141 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.098003    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc" event={"ID":"ee22c9dc-b24d-4412-b4c4-547243142a04","Type":"ContainerDied","Data":"441b21f817e310816cb6be419bf9ca0f6dc6a0f92e14cd6a2c449a6e7a3cb569"}
Sep 09 21:09:13.098141 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.098045    2581 scope.go:117] "RemoveContainer" containerID="aeb8a9d33959958560141102704e695628884ef1af35e6e4263b62015d427b4b"
Sep 09 21:09:13.107268 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.107236    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee22c9dc-b24d-4412-b4c4-547243142a04" (UID: "ee22c9dc-b24d-4412-b4c4-547243142a04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:13.113668 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.113645    2581 scope.go:117] "RemoveContainer" containerID="990e6d75638145e13029a673c7b2e17fc2e8a0dd712594a79f2b88b87f538191"
Sep 09 21:09:13.120742 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.120716    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ee22c9dc-b24d-4412-b4c4-547243142a04" (UID: "ee22c9dc-b24d-4412-b4c4-547243142a04"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:13.121892 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.121878    2581 scope.go:117] "RemoveContainer" containerID="aeb8a9d33959958560141102704e695628884ef1af35e6e4263b62015d427b4b"
Sep 09 21:09:13.122159 ip-10-0-134-84 kubenswrapper[2581]: E0909 21:09:13.122142    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeb8a9d33959958560141102704e695628884ef1af35e6e4263b62015d427b4b\": container with ID starting with aeb8a9d33959958560141102704e695628884ef1af35e6e4263b62015d427b4b not found: ID does not exist" containerID="aeb8a9d33959958560141102704e695628884ef1af35e6e4263b62015d427b4b"
Sep 09 21:09:13.122212 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.122166    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeb8a9d33959958560141102704e695628884ef1af35e6e4263b62015d427b4b"} err="failed to get container status \"aeb8a9d33959958560141102704e695628884ef1af35e6e4263b62015d427b4b\": rpc error: code = NotFound desc = could not find container \"aeb8a9d33959958560141102704e695628884ef1af35e6e4263b62015d427b4b\": container with ID starting with aeb8a9d33959958560141102704e695628884ef1af35e6e4263b62015d427b4b not found: ID does not exist"
Sep 09 21:09:13.122212 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.122183    2581 scope.go:117] "RemoveContainer" containerID="990e6d75638145e13029a673c7b2e17fc2e8a0dd712594a79f2b88b87f538191"
Sep 09 21:09:13.122400 ip-10-0-134-84 kubenswrapper[2581]: E0909 21:09:13.122380    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"990e6d75638145e13029a673c7b2e17fc2e8a0dd712594a79f2b88b87f538191\": container with ID starting with 990e6d75638145e13029a673c7b2e17fc2e8a0dd712594a79f2b88b87f538191 not found: ID does not exist" containerID="990e6d75638145e13029a673c7b2e17fc2e8a0dd712594a79f2b88b87f538191"
Sep 09 21:09:13.122471 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.122410    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"990e6d75638145e13029a673c7b2e17fc2e8a0dd712594a79f2b88b87f538191"} err="failed to get container status \"990e6d75638145e13029a673c7b2e17fc2e8a0dd712594a79f2b88b87f538191\": rpc error: code = NotFound desc = could not find container \"990e6d75638145e13029a673c7b2e17fc2e8a0dd712594a79f2b88b87f538191\": container with ID starting with 990e6d75638145e13029a673c7b2e17fc2e8a0dd712594a79f2b88b87f538191 not found: ID does not exist"
Sep 09 21:09:13.194002 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.193975    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5wlv\" (UniqueName: \"kubernetes.io/projected/ee22c9dc-b24d-4412-b4c4-547243142a04-kube-api-access-p5wlv\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:09:13.194002 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.194000    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:09:13.194146 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.194010    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:09:13.194146 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.194019    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-kserve-provision-location\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:09:13.194146 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.194028    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee22c9dc-b24d-4412-b4c4-547243142a04-cabundle-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:09:13.194146 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.194036    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee22c9dc-b24d-4412-b4c4-547243142a04-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:09:13.194146 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.194043    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee22c9dc-b24d-4412-b4c4-547243142a04-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:09:13.422611 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.422580    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"]
Sep 09 21:09:13.426863 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:13.426834    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7455c584fb-8nmqc"]
Sep 09 21:09:14.634761 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:14.634730    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee22c9dc-b24d-4412-b4c4-547243142a04" path="/var/lib/kubelet/pods/ee22c9dc-b24d-4412-b4c4-547243142a04/volumes"
Sep 09 21:09:17.933734 ip-10-0-134-84 kubenswrapper[2581]: E0909 21:09:17.933700    2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d554e31_e24c_415d_aa0d_f76a411d87f8.slice/crio-conmon-b1afa331e0b955af36f186242926e4dd4e87348ac3d56883f27ae202fb93c024.scope\": RecentStats: unable to find data in memory cache]"
Sep 09 21:09:18.126154 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:18.126122    2581 generic.go:356] "Generic (PLEG): container finished" podID="4d554e31-e24c-415d-aa0d-f76a411d87f8" containerID="b1afa331e0b955af36f186242926e4dd4e87348ac3d56883f27ae202fb93c024" exitCode=0
Sep 09 21:09:18.126331 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:18.126203    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z" event={"ID":"4d554e31-e24c-415d-aa0d-f76a411d87f8","Type":"ContainerDied","Data":"b1afa331e0b955af36f186242926e4dd4e87348ac3d56883f27ae202fb93c024"}
Sep 09 21:09:18.145346 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:18.145319    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z"
Sep 09 21:09:18.335569 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:18.335525    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ppfc\" (UniqueName: \"kubernetes.io/projected/4d554e31-e24c-415d-aa0d-f76a411d87f8-kube-api-access-5ppfc\") pod \"4d554e31-e24c-415d-aa0d-f76a411d87f8\" (UID: \"4d554e31-e24c-415d-aa0d-f76a411d87f8\") "
Sep 09 21:09:18.335757 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:18.335586    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d554e31-e24c-415d-aa0d-f76a411d87f8-tls-certs\") pod \"4d554e31-e24c-415d-aa0d-f76a411d87f8\" (UID: \"4d554e31-e24c-415d-aa0d-f76a411d87f8\") "
Sep 09 21:09:18.337789 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:18.337761    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d554e31-e24c-415d-aa0d-f76a411d87f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d554e31-e24c-415d-aa0d-f76a411d87f8" (UID: "4d554e31-e24c-415d-aa0d-f76a411d87f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:09:18.337894 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:18.337839    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d554e31-e24c-415d-aa0d-f76a411d87f8-kube-api-access-5ppfc" (OuterVolumeSpecName: "kube-api-access-5ppfc") pod "4d554e31-e24c-415d-aa0d-f76a411d87f8" (UID: "4d554e31-e24c-415d-aa0d-f76a411d87f8"). InnerVolumeSpecName "kube-api-access-5ppfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:18.437133 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:18.437090    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ppfc\" (UniqueName: \"kubernetes.io/projected/4d554e31-e24c-415d-aa0d-f76a411d87f8-kube-api-access-5ppfc\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:09:18.437133 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:18.437123    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d554e31-e24c-415d-aa0d-f76a411d87f8-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:09:19.132451 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:19.132417    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z" event={"ID":"4d554e31-e24c-415d-aa0d-f76a411d87f8","Type":"ContainerDied","Data":"59ebcb779644a4a02c0c9c70044bc9b90c091f99e4b8af4a486f0b6b8ddb1fa9"}
Sep 09 21:09:19.132946 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:19.132498    2581 scope.go:117] "RemoveContainer" containerID="b1afa331e0b955af36f186242926e4dd4e87348ac3d56883f27ae202fb93c024"
Sep 09 21:09:19.132946 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:19.132495    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z"
Sep 09 21:09:19.152540 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:19.152508    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z"]
Sep 09 21:09:19.156769 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:19.156739    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddmrb4z"]
Sep 09 21:09:20.635942 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:09:20.635913    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d554e31-e24c-415d-aa0d-f76a411d87f8" path="/var/lib/kubelet/pods/4d554e31-e24c-415d-aa0d-f76a411d87f8/volumes"
Sep 09 21:10:07.271107 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.271057    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"]
Sep 09 21:10:07.271790 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.271583    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d554e31-e24c-415d-aa0d-f76a411d87f8" containerName="main"
Sep 09 21:10:07.271790 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.271605    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d554e31-e24c-415d-aa0d-f76a411d87f8" containerName="main"
Sep 09 21:10:07.271790 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.271618    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee22c9dc-b24d-4412-b4c4-547243142a04" containerName="storage-initializer"
Sep 09 21:10:07.271790 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.271626    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee22c9dc-b24d-4412-b4c4-547243142a04" containerName="storage-initializer"
Sep 09 21:10:07.271790 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.271653    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee22c9dc-b24d-4412-b4c4-547243142a04" containerName="main"
Sep 09 21:10:07.271790 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.271662    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee22c9dc-b24d-4412-b4c4-547243142a04" containerName="main"
Sep 09 21:10:07.271790 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.271754    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d554e31-e24c-415d-aa0d-f76a411d87f8" containerName="main"
Sep 09 21:10:07.271790 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.271768    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee22c9dc-b24d-4412-b4c4-547243142a04" containerName="main"
Sep 09 21:10:07.276391 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.276368    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.278918 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.278896    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 09 21:10:07.279025 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.278951    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:10:07.279113 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.279038    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 09 21:10:07.279193 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.279171    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-7qcx4\""
Sep 09 21:10:07.279255 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.279176    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 09 21:10:07.279312 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.279266    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 09 21:10:07.290268 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.290244    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"]
Sep 09 21:10:07.351710 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.351671    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-home\") pod \"router-with-refs-test-kserve-7c677d668-dvzs5\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.351710 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.351713    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-model-cache\") pod \"router-with-refs-test-kserve-7c677d668-dvzs5\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.351907 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.351778    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-kserve-provision-location\") pod \"router-with-refs-test-kserve-7c677d668-dvzs5\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.351907 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.351840    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4308d306-098e-4d42-b26a-fba4fd5e90d4-cabundle-cert\") pod \"router-with-refs-test-kserve-7c677d668-dvzs5\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.351907 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.351868    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-dshm\") pod \"router-with-refs-test-kserve-7c677d668-dvzs5\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.351907 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.351887    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4308d306-098e-4d42-b26a-fba4fd5e90d4-tls-certs\") pod \"router-with-refs-test-kserve-7c677d668-dvzs5\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.351907 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.351905    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zwpp\" (UniqueName: \"kubernetes.io/projected/4308d306-098e-4d42-b26a-fba4fd5e90d4-kube-api-access-5zwpp\") pod \"router-with-refs-test-kserve-7c677d668-dvzs5\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.352080 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.351932    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-tmp-dir\") pod \"router-with-refs-test-kserve-7c677d668-dvzs5\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.452810 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.452770    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-home\") pod \"router-with-refs-test-kserve-7c677d668-dvzs5\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.452990 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.452834    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-model-cache\") pod \"router-with-refs-test-kserve-7c677d668-dvzs5\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.452990 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.452873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-kserve-provision-location\") pod \"router-with-refs-test-kserve-7c677d668-dvzs5\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.452990 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.452903    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4308d306-098e-4d42-b26a-fba4fd5e90d4-cabundle-cert\") pod \"router-with-refs-test-kserve-7c677d668-dvzs5\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.453154 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.453041    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-dshm\") pod \"router-with-refs-test-kserve-7c677d668-dvzs5\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.453154 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.453086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4308d306-098e-4d42-b26a-fba4fd5e90d4-tls-certs\") pod \"router-with-refs-test-kserve-7c677d668-dvzs5\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.453154 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.453121    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zwpp\" (UniqueName: \"kubernetes.io/projected/4308d306-098e-4d42-b26a-fba4fd5e90d4-kube-api-access-5zwpp\") pod \"router-with-refs-test-kserve-7c677d668-dvzs5\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.453313 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.453167    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-tmp-dir\") pod \"router-with-refs-test-kserve-7c677d668-dvzs5\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.453313 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.453246    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-home\") pod \"router-with-refs-test-kserve-7c677d668-dvzs5\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.453313 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.453283    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-kserve-provision-location\") pod \"router-with-refs-test-kserve-7c677d668-dvzs5\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.453608 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.453586    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-model-cache\") pod \"router-with-refs-test-kserve-7c677d668-dvzs5\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.453688 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.453607    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-tmp-dir\") pod \"router-with-refs-test-kserve-7c677d668-dvzs5\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.453688 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.453632    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4308d306-098e-4d42-b26a-fba4fd5e90d4-cabundle-cert\") pod \"router-with-refs-test-kserve-7c677d668-dvzs5\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.455380 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.455361    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-dshm\") pod \"router-with-refs-test-kserve-7c677d668-dvzs5\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.455741 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.455724    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4308d306-098e-4d42-b26a-fba4fd5e90d4-tls-certs\") pod \"router-with-refs-test-kserve-7c677d668-dvzs5\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.462853 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.462827    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zwpp\" (UniqueName: \"kubernetes.io/projected/4308d306-098e-4d42-b26a-fba4fd5e90d4-kube-api-access-5zwpp\") pod \"router-with-refs-test-kserve-7c677d668-dvzs5\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.586739 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.586637    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:07.872704 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:07.872675    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"]
Sep 09 21:10:07.875448 ip-10-0-134-84 kubenswrapper[2581]: W0909 21:10:07.875421    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4308d306_098e_4d42_b26a_fba4fd5e90d4.slice/crio-73bd3d23e69cc7df4d6d0d2a0df5c1d01426e73551fef0510b7ffc7a4bc8b017 WatchSource:0}: Error finding container 73bd3d23e69cc7df4d6d0d2a0df5c1d01426e73551fef0510b7ffc7a4bc8b017: Status 404 returned error can't find the container with id 73bd3d23e69cc7df4d6d0d2a0df5c1d01426e73551fef0510b7ffc7a4bc8b017
Sep 09 21:10:08.382323 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:08.382281    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5" event={"ID":"4308d306-098e-4d42-b26a-fba4fd5e90d4","Type":"ContainerStarted","Data":"737bea3e8959aa6496b4382f432162d95412a8a0e1a4ac99c7898311190e85a2"}
Sep 09 21:10:08.382840 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:08.382344    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5" event={"ID":"4308d306-098e-4d42-b26a-fba4fd5e90d4","Type":"ContainerStarted","Data":"73bd3d23e69cc7df4d6d0d2a0df5c1d01426e73551fef0510b7ffc7a4bc8b017"}
Sep 09 21:10:10.393599 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:10.393568    2581 generic.go:356] "Generic (PLEG): container finished" podID="4308d306-098e-4d42-b26a-fba4fd5e90d4" containerID="737bea3e8959aa6496b4382f432162d95412a8a0e1a4ac99c7898311190e85a2" exitCode=0
Sep 09 21:10:10.393977 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:10.393654    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5" event={"ID":"4308d306-098e-4d42-b26a-fba4fd5e90d4","Type":"ContainerDied","Data":"737bea3e8959aa6496b4382f432162d95412a8a0e1a4ac99c7898311190e85a2"}
Sep 09 21:10:11.400763 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:11.400725    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5" event={"ID":"4308d306-098e-4d42-b26a-fba4fd5e90d4","Type":"ContainerStarted","Data":"2cbb4379f5e433dfd2caf3a173a3675660a17b0b6c79abe03f2b90e5985a22fe"}
Sep 09 21:10:11.425073 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:11.425016    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5" podStartSLOduration=2.198847702 podStartE2EDuration="4.424997924s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.226150222s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:10:11.423019561 +0000 UTC m=+3553.428196566" watchObservedRunningTime="2026-09-09 21:10:11.424997924 +0000 UTC m=+3553.430174918"
Sep 09 21:10:11.587252 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:11.587212    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:17.586983 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:17.586945    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:10:17.588542 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:17.588512    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5" podUID="4308d306-098e-4d42-b26a-fba4fd5e90d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 21:10:27.587861 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:27.587820    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5" podUID="4308d306-098e-4d42-b26a-fba4fd5e90d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 21:10:37.587836 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:37.587789    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5" podUID="4308d306-098e-4d42-b26a-fba4fd5e90d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 21:10:47.587394 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:47.587345    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5" podUID="4308d306-098e-4d42-b26a-fba4fd5e90d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 21:10:53.164552 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:53.164436    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k"]
Sep 09 21:10:57.587789 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:57.587746    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5" podUID="4308d306-098e-4d42-b26a-fba4fd5e90d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 21:10:58.801490 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:58.801339    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 21:10:58.811790 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:58.804640    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 21:10:58.814443 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:58.814425    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 21:10:58.817627 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:10:58.817607    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 21:11:07.588078 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:07.588017    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5" podUID="4308d306-098e-4d42-b26a-fba4fd5e90d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 21:11:08.185067 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:08.185028    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k" podUID="5dab98b5-26fb-4b50-bf77-2d50dd6245d2" containerName="main" containerID="cri-o://39486848181c90f814e69df047653779efaa1c5d53488cd79022f9b2f93cde38" gracePeriod=45
Sep 09 21:11:12.935052 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:12.935014    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k" podUID="5dab98b5-26fb-4b50-bf77-2d50dd6245d2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:11:17.587839 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:17.587795    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5" podUID="4308d306-098e-4d42-b26a-fba4fd5e90d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 21:11:22.935505 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:22.935443    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k" podUID="5dab98b5-26fb-4b50-bf77-2d50dd6245d2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:11:27.587307 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:27.587263    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5" podUID="4308d306-098e-4d42-b26a-fba4fd5e90d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 21:11:32.935247 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:32.935205    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k" podUID="5dab98b5-26fb-4b50-bf77-2d50dd6245d2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:11:32.935742 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:32.935320    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k"
Sep 09 21:11:37.587821 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:37.587779    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5" podUID="4308d306-098e-4d42-b26a-fba4fd5e90d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 21:11:38.426053 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:38.426028    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k"
Sep 09 21:11:38.522905 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:38.522864    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wljn5\" (UniqueName: \"kubernetes.io/projected/5dab98b5-26fb-4b50-bf77-2d50dd6245d2-kube-api-access-wljn5\") pod \"5dab98b5-26fb-4b50-bf77-2d50dd6245d2\" (UID: \"5dab98b5-26fb-4b50-bf77-2d50dd6245d2\") "
Sep 09 21:11:38.523103 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:38.522917    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dab98b5-26fb-4b50-bf77-2d50dd6245d2-tls-certs\") pod \"5dab98b5-26fb-4b50-bf77-2d50dd6245d2\" (UID: \"5dab98b5-26fb-4b50-bf77-2d50dd6245d2\") "
Sep 09 21:11:38.525127 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:38.525089    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dab98b5-26fb-4b50-bf77-2d50dd6245d2-kube-api-access-wljn5" (OuterVolumeSpecName: "kube-api-access-wljn5") pod "5dab98b5-26fb-4b50-bf77-2d50dd6245d2" (UID: "5dab98b5-26fb-4b50-bf77-2d50dd6245d2"). InnerVolumeSpecName "kube-api-access-wljn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:11:38.525241 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:38.525134    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dab98b5-26fb-4b50-bf77-2d50dd6245d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5dab98b5-26fb-4b50-bf77-2d50dd6245d2" (UID: "5dab98b5-26fb-4b50-bf77-2d50dd6245d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:11:38.624429 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:38.624394    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wljn5\" (UniqueName: \"kubernetes.io/projected/5dab98b5-26fb-4b50-bf77-2d50dd6245d2-kube-api-access-wljn5\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:11:38.624429 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:38.624422    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dab98b5-26fb-4b50-bf77-2d50dd6245d2-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:11:38.872036 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:38.871947    2581 generic.go:356] "Generic (PLEG): container finished" podID="5dab98b5-26fb-4b50-bf77-2d50dd6245d2" containerID="39486848181c90f814e69df047653779efaa1c5d53488cd79022f9b2f93cde38" exitCode=0
Sep 09 21:11:38.872036 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:38.871984    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k"
Sep 09 21:11:38.872235 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:38.871984    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k" event={"ID":"5dab98b5-26fb-4b50-bf77-2d50dd6245d2","Type":"ContainerDied","Data":"39486848181c90f814e69df047653779efaa1c5d53488cd79022f9b2f93cde38"}
Sep 09 21:11:38.872235 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:38.872102    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k" event={"ID":"5dab98b5-26fb-4b50-bf77-2d50dd6245d2","Type":"ContainerDied","Data":"1dfe5b7154db0d03a3011921b0d5b9881ff78c4354f8e83e3550b1dd17d488dd"}
Sep 09 21:11:38.872235 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:38.872125    2581 scope.go:117] "RemoveContainer" containerID="39486848181c90f814e69df047653779efaa1c5d53488cd79022f9b2f93cde38"
Sep 09 21:11:38.881134 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:38.881117    2581 scope.go:117] "RemoveContainer" containerID="39486848181c90f814e69df047653779efaa1c5d53488cd79022f9b2f93cde38"
Sep 09 21:11:38.881481 ip-10-0-134-84 kubenswrapper[2581]: E0909 21:11:38.881448    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39486848181c90f814e69df047653779efaa1c5d53488cd79022f9b2f93cde38\": container with ID starting with 39486848181c90f814e69df047653779efaa1c5d53488cd79022f9b2f93cde38 not found: ID does not exist" containerID="39486848181c90f814e69df047653779efaa1c5d53488cd79022f9b2f93cde38"
Sep 09 21:11:38.881554 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:38.881493    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39486848181c90f814e69df047653779efaa1c5d53488cd79022f9b2f93cde38"} err="failed to get container status \"39486848181c90f814e69df047653779efaa1c5d53488cd79022f9b2f93cde38\": rpc error: code = NotFound desc = could not find container \"39486848181c90f814e69df047653779efaa1c5d53488cd79022f9b2f93cde38\": container with ID starting with 39486848181c90f814e69df047653779efaa1c5d53488cd79022f9b2f93cde38 not found: ID does not exist"
Sep 09 21:11:38.891548 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:38.891528    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k"]
Sep 09 21:11:38.898126 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:38.898105    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-wkh7k"]
Sep 09 21:11:40.635960 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:40.635925    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dab98b5-26fb-4b50-bf77-2d50dd6245d2" path="/var/lib/kubelet/pods/5dab98b5-26fb-4b50-bf77-2d50dd6245d2/volumes"
Sep 09 21:11:47.587037 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:47.586989    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5" podUID="4308d306-098e-4d42-b26a-fba4fd5e90d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 21:11:57.587385 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:11:57.587344    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5" podUID="4308d306-098e-4d42-b26a-fba4fd5e90d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 21:12:07.596735 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:07.596701    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:12:07.604406 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:07.604379    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:12:12.490780 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:12.490742    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"]
Sep 09 21:12:27.513202 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:27.513111    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5" podUID="4308d306-098e-4d42-b26a-fba4fd5e90d4" containerName="main" containerID="cri-o://2cbb4379f5e433dfd2caf3a173a3675660a17b0b6c79abe03f2b90e5985a22fe" gracePeriod=45
Sep 09 21:12:27.597850 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:27.597804    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5" podUID="4308d306-098e-4d42-b26a-fba4fd5e90d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 21:12:28.134490 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.134436    2581 generic.go:356] "Generic (PLEG): container finished" podID="4308d306-098e-4d42-b26a-fba4fd5e90d4" containerID="2cbb4379f5e433dfd2caf3a173a3675660a17b0b6c79abe03f2b90e5985a22fe" exitCode=0
Sep 09 21:12:28.134623 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.134500    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5" event={"ID":"4308d306-098e-4d42-b26a-fba4fd5e90d4","Type":"ContainerDied","Data":"2cbb4379f5e433dfd2caf3a173a3675660a17b0b6c79abe03f2b90e5985a22fe"}
Sep 09 21:12:28.270672 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.270643    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:12:28.462120 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.462023    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zwpp\" (UniqueName: \"kubernetes.io/projected/4308d306-098e-4d42-b26a-fba4fd5e90d4-kube-api-access-5zwpp\") pod \"4308d306-098e-4d42-b26a-fba4fd5e90d4\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") "
Sep 09 21:12:28.462120 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.462066    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-kserve-provision-location\") pod \"4308d306-098e-4d42-b26a-fba4fd5e90d4\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") "
Sep 09 21:12:28.462120 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.462103    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4308d306-098e-4d42-b26a-fba4fd5e90d4-cabundle-cert\") pod \"4308d306-098e-4d42-b26a-fba4fd5e90d4\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") "
Sep 09 21:12:28.462120 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.462125    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-home\") pod \"4308d306-098e-4d42-b26a-fba4fd5e90d4\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") "
Sep 09 21:12:28.462485 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.462231    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4308d306-098e-4d42-b26a-fba4fd5e90d4-tls-certs\") pod \"4308d306-098e-4d42-b26a-fba4fd5e90d4\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") "
Sep 09 21:12:28.462485 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.462272    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-dshm\") pod \"4308d306-098e-4d42-b26a-fba4fd5e90d4\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") "
Sep 09 21:12:28.462485 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.462295    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-model-cache\") pod \"4308d306-098e-4d42-b26a-fba4fd5e90d4\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") "
Sep 09 21:12:28.462485 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.462350    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-tmp-dir\") pod \"4308d306-098e-4d42-b26a-fba4fd5e90d4\" (UID: \"4308d306-098e-4d42-b26a-fba4fd5e90d4\") "
Sep 09 21:12:28.462703 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.462545    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4308d306-098e-4d42-b26a-fba4fd5e90d4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4308d306-098e-4d42-b26a-fba4fd5e90d4" (UID: "4308d306-098e-4d42-b26a-fba4fd5e90d4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:12:28.462703 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.462679    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4308d306-098e-4d42-b26a-fba4fd5e90d4-cabundle-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:12:28.463044 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.463022    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "4308d306-098e-4d42-b26a-fba4fd5e90d4" (UID: "4308d306-098e-4d42-b26a-fba4fd5e90d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:12:28.463154 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.463115    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-home" (OuterVolumeSpecName: "home") pod "4308d306-098e-4d42-b26a-fba4fd5e90d4" (UID: "4308d306-098e-4d42-b26a-fba4fd5e90d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:12:28.464562 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.464535    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4308d306-098e-4d42-b26a-fba4fd5e90d4-kube-api-access-5zwpp" (OuterVolumeSpecName: "kube-api-access-5zwpp") pod "4308d306-098e-4d42-b26a-fba4fd5e90d4" (UID: "4308d306-098e-4d42-b26a-fba4fd5e90d4"). InnerVolumeSpecName "kube-api-access-5zwpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:12:28.464658 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.464616    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4308d306-098e-4d42-b26a-fba4fd5e90d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4308d306-098e-4d42-b26a-fba4fd5e90d4" (UID: "4308d306-098e-4d42-b26a-fba4fd5e90d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:12:28.464801 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.464780    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-dshm" (OuterVolumeSpecName: "dshm") pod "4308d306-098e-4d42-b26a-fba4fd5e90d4" (UID: "4308d306-098e-4d42-b26a-fba4fd5e90d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:12:28.476924 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.476896    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4308d306-098e-4d42-b26a-fba4fd5e90d4" (UID: "4308d306-098e-4d42-b26a-fba4fd5e90d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:12:28.486107 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.486078    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4308d306-098e-4d42-b26a-fba4fd5e90d4" (UID: "4308d306-098e-4d42-b26a-fba4fd5e90d4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:12:28.563539 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.563497    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:12:28.563908 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.563559    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:12:28.563908 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.563573    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zwpp\" (UniqueName: \"kubernetes.io/projected/4308d306-098e-4d42-b26a-fba4fd5e90d4-kube-api-access-5zwpp\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:12:28.563908 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.563585    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-kserve-provision-location\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:12:28.563908 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.563599    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:12:28.563908 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.563611    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4308d306-098e-4d42-b26a-fba4fd5e90d4-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:12:28.563908 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:28.563623    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4308d306-098e-4d42-b26a-fba4fd5e90d4-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:12:29.141239 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:29.141209    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"
Sep 09 21:12:29.141443 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:29.141242    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5" event={"ID":"4308d306-098e-4d42-b26a-fba4fd5e90d4","Type":"ContainerDied","Data":"73bd3d23e69cc7df4d6d0d2a0df5c1d01426e73551fef0510b7ffc7a4bc8b017"}
Sep 09 21:12:29.141443 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:29.141305    2581 scope.go:117] "RemoveContainer" containerID="2cbb4379f5e433dfd2caf3a173a3675660a17b0b6c79abe03f2b90e5985a22fe"
Sep 09 21:12:29.150312 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:29.150290    2581 scope.go:117] "RemoveContainer" containerID="737bea3e8959aa6496b4382f432162d95412a8a0e1a4ac99c7898311190e85a2"
Sep 09 21:12:29.163088 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:29.163045    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"]
Sep 09 21:12:29.167193 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:29.167169    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7c677d668-dvzs5"]
Sep 09 21:12:30.635191 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:12:30.635148    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4308d306-098e-4d42-b26a-fba4fd5e90d4" path="/var/lib/kubelet/pods/4308d306-098e-4d42-b26a-fba4fd5e90d4/volumes"
Sep 09 21:13:13.965219 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:13.965182    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"]
Sep 09 21:13:13.965687 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:13.965619    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dab98b5-26fb-4b50-bf77-2d50dd6245d2" containerName="main"
Sep 09 21:13:13.965687 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:13.965635    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dab98b5-26fb-4b50-bf77-2d50dd6245d2" containerName="main"
Sep 09 21:13:13.965687 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:13.965654    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4308d306-098e-4d42-b26a-fba4fd5e90d4" containerName="main"
Sep 09 21:13:13.965687 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:13.965660    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4308d306-098e-4d42-b26a-fba4fd5e90d4" containerName="main"
Sep 09 21:13:13.965687 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:13.965669    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4308d306-098e-4d42-b26a-fba4fd5e90d4" containerName="storage-initializer"
Sep 09 21:13:13.965687 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:13.965675    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4308d306-098e-4d42-b26a-fba4fd5e90d4" containerName="storage-initializer"
Sep 09 21:13:13.965881 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:13.965733    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4308d306-098e-4d42-b26a-fba4fd5e90d4" containerName="main"
Sep 09 21:13:13.965881 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:13.965744    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dab98b5-26fb-4b50-bf77-2d50dd6245d2" containerName="main"
Sep 09 21:13:13.970170 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:13.970149    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:13.973491 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:13.973452    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-9xwtg\""
Sep 09 21:13:13.973593 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:13.973514    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 09 21:13:13.973707 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:13.973685    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 09 21:13:13.973951 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:13.973933    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-7pm8s\""
Sep 09 21:13:13.974223 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:13.974205    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:13:13.974684 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:13.974666    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 09 21:13:13.974990 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:13.974974    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 09 21:13:13.987288 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:13.987263    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"]
Sep 09 21:13:14.100600 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.100569    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.100760 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.100607    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.100760 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.100636    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6r5w\" (UniqueName: \"kubernetes.io/projected/857f9131-77cc-42f6-933f-57ab4ee26506-kube-api-access-q6r5w\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.100760 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.100673    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.100760 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.100712    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.100923 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.100780    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/857f9131-77cc-42f6-933f-57ab4ee26506-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.100923 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.100806    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/857f9131-77cc-42f6-933f-57ab4ee26506-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.100923 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.100832    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.202148 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.202115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/857f9131-77cc-42f6-933f-57ab4ee26506-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.202148 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.202152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.202366 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.202178    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.202366 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.202199    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.202366 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.202220    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6r5w\" (UniqueName: \"kubernetes.io/projected/857f9131-77cc-42f6-933f-57ab4ee26506-kube-api-access-q6r5w\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.202366 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.202243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.202366 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.202276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.202366 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.202301    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/857f9131-77cc-42f6-933f-57ab4ee26506-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.202724 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.202632    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.202881 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.202854    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.203143 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.203108    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.203274 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.203154    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.203406 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.203385    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/857f9131-77cc-42f6-933f-57ab4ee26506-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.204940 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.204921    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.205073 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.205055    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/857f9131-77cc-42f6-933f-57ab4ee26506-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.214945 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.214918    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6r5w\" (UniqueName: \"kubernetes.io/projected/857f9131-77cc-42f6-933f-57ab4ee26506-kube-api-access-q6r5w\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.258624 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.258595    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27"]
Sep 09 21:13:14.263143 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.263119    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27"
Sep 09 21:13:14.267063 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.267039    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-zw94x\""
Sep 09 21:13:14.276340 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.276316    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27"]
Sep 09 21:13:14.280190 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.280170    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:14.404332 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.404295    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba0475d-ac7b-4469-881d-15cf6151509d-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27\" (UID: \"eba0475d-ac7b-4469-881d-15cf6151509d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27"
Sep 09 21:13:14.404530 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.404337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx2pw\" (UniqueName: \"kubernetes.io/projected/eba0475d-ac7b-4469-881d-15cf6151509d-kube-api-access-lx2pw\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27\" (UID: \"eba0475d-ac7b-4469-881d-15cf6151509d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27"
Sep 09 21:13:14.505243 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.505206    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba0475d-ac7b-4469-881d-15cf6151509d-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27\" (UID: \"eba0475d-ac7b-4469-881d-15cf6151509d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27"
Sep 09 21:13:14.505377 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.505249    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx2pw\" (UniqueName: \"kubernetes.io/projected/eba0475d-ac7b-4469-881d-15cf6151509d-kube-api-access-lx2pw\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27\" (UID: \"eba0475d-ac7b-4469-881d-15cf6151509d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27"
Sep 09 21:13:14.507710 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.507689    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba0475d-ac7b-4469-881d-15cf6151509d-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27\" (UID: \"eba0475d-ac7b-4469-881d-15cf6151509d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27"
Sep 09 21:13:14.518716 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.518663    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx2pw\" (UniqueName: \"kubernetes.io/projected/eba0475d-ac7b-4469-881d-15cf6151509d-kube-api-access-lx2pw\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27\" (UID: \"eba0475d-ac7b-4469-881d-15cf6151509d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27"
Sep 09 21:13:14.573997 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.573962    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"]
Sep 09 21:13:14.574179 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.574164    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27"
Sep 09 21:13:14.583505 ip-10-0-134-84 kubenswrapper[2581]: W0909 21:13:14.583480    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857f9131_77cc_42f6_933f_57ab4ee26506.slice/crio-ba3ae0dc1eb9461fba5d40121468d87c0241c5947628804a17eb0238f12220f1 WatchSource:0}: Error finding container ba3ae0dc1eb9461fba5d40121468d87c0241c5947628804a17eb0238f12220f1: Status 404 returned error can't find the container with id ba3ae0dc1eb9461fba5d40121468d87c0241c5947628804a17eb0238f12220f1
Sep 09 21:13:14.854029 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:14.854003    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27"]
Sep 09 21:13:14.855681 ip-10-0-134-84 kubenswrapper[2581]: W0909 21:13:14.855651    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba0475d_ac7b_4469_881d_15cf6151509d.slice/crio-e5106973a1fe6f6cbccab99db211aabd2f11275383cbc55d8ae61e0a726f3d98 WatchSource:0}: Error finding container e5106973a1fe6f6cbccab99db211aabd2f11275383cbc55d8ae61e0a726f3d98: Status 404 returned error can't find the container with id e5106973a1fe6f6cbccab99db211aabd2f11275383cbc55d8ae61e0a726f3d98
Sep 09 21:13:15.368862 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:15.368816    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27" event={"ID":"eba0475d-ac7b-4469-881d-15cf6151509d","Type":"ContainerStarted","Data":"862a5be40b684ae0125b642eea8a357e5263b758bcd1ab6f14ca1498f9325060"}
Sep 09 21:13:15.369338 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:15.368977    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27"
Sep 09 21:13:15.369338 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:15.369007    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27" event={"ID":"eba0475d-ac7b-4469-881d-15cf6151509d","Type":"ContainerStarted","Data":"e5106973a1fe6f6cbccab99db211aabd2f11275383cbc55d8ae61e0a726f3d98"}
Sep 09 21:13:15.370628 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:15.370598    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl" event={"ID":"857f9131-77cc-42f6-933f-57ab4ee26506","Type":"ContainerStarted","Data":"ace0eca649d8b613e441111f31621c24eee235c7de6fe1efa355d4b2e0a89141"}
Sep 09 21:13:15.370763 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:15.370642    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl" event={"ID":"857f9131-77cc-42f6-933f-57ab4ee26506","Type":"ContainerStarted","Data":"ba3ae0dc1eb9461fba5d40121468d87c0241c5947628804a17eb0238f12220f1"}
Sep 09 21:13:15.370763 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:15.370701    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:15.391442 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:15.391397    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27" podStartSLOduration=1.391383543 podStartE2EDuration="1.391383543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:13:15.389119531 +0000 UTC m=+3737.394296527" watchObservedRunningTime="2026-09-09 21:13:15.391383543 +0000 UTC m=+3737.396560538"
Sep 09 21:13:16.376969 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:16.376934    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl" event={"ID":"857f9131-77cc-42f6-933f-57ab4ee26506","Type":"ContainerStarted","Data":"dec3279716f9a008f0fcbbb20034aca478ba1f68d44291f1ee7b8ee01a4f0c95"}
Sep 09 21:13:18.388944 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:18.388907    2581 generic.go:356] "Generic (PLEG): container finished" podID="857f9131-77cc-42f6-933f-57ab4ee26506" containerID="dec3279716f9a008f0fcbbb20034aca478ba1f68d44291f1ee7b8ee01a4f0c95" exitCode=0
Sep 09 21:13:18.389424 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:18.388982    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl" event={"ID":"857f9131-77cc-42f6-933f-57ab4ee26506","Type":"ContainerDied","Data":"dec3279716f9a008f0fcbbb20034aca478ba1f68d44291f1ee7b8ee01a4f0c95"}
Sep 09 21:13:19.396736 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:19.396702    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl" event={"ID":"857f9131-77cc-42f6-933f-57ab4ee26506","Type":"ContainerStarted","Data":"f3bbc602ed7ece1ff9bde46ec2e7a66fa7fe5c8df508c8e84a10f9dfd87cf289"}
Sep 09 21:13:19.427706 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:19.427653    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl" podStartSLOduration=4.088090128 podStartE2EDuration="6.427638401s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.339548273s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:13:19.424399525 +0000 UTC m=+3741.429576520" watchObservedRunningTime="2026-09-09 21:13:19.427638401 +0000 UTC m=+3741.432815419"
Sep 09 21:13:24.280757 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:24.280720    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:24.281322 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:24.280787    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:24.282331 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:24.282291    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl" podUID="857f9131-77cc-42f6-933f-57ab4ee26506" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8001/health\": dial tcp 10.132.0.77:8001: connect: connection refused"
Sep 09 21:13:24.301505 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:24.301481    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:13:34.281514 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:34.281436    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl" podUID="857f9131-77cc-42f6-933f-57ab4ee26506" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8001/health\": dial tcp 10.132.0.77:8001: connect: connection refused"
Sep 09 21:13:44.281225 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:44.281175    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl" podUID="857f9131-77cc-42f6-933f-57ab4ee26506" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8001/health\": dial tcp 10.132.0.77:8001: connect: connection refused"
Sep 09 21:13:46.379273 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:46.379242    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27"
Sep 09 21:13:54.281678 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:13:54.281567    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl" podUID="857f9131-77cc-42f6-933f-57ab4ee26506" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8001/health\": dial tcp 10.132.0.77:8001: connect: connection refused"
Sep 09 21:14:04.281161 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:14:04.281101    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl" podUID="857f9131-77cc-42f6-933f-57ab4ee26506" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8001/health\": dial tcp 10.132.0.77:8001: connect: connection refused"
Sep 09 21:14:14.280605 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:14:14.280555    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl" podUID="857f9131-77cc-42f6-933f-57ab4ee26506" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8001/health\": dial tcp 10.132.0.77:8001: connect: connection refused"
Sep 09 21:14:24.280918 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:14:24.280868    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl" podUID="857f9131-77cc-42f6-933f-57ab4ee26506" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8001/health\": dial tcp 10.132.0.77:8001: connect: connection refused"
Sep 09 21:14:34.280608 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:14:34.280554    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl" podUID="857f9131-77cc-42f6-933f-57ab4ee26506" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8001/health\": dial tcp 10.132.0.77:8001: connect: connection refused"
Sep 09 21:14:44.281409 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:14:44.281352    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl" podUID="857f9131-77cc-42f6-933f-57ab4ee26506" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8001/health\": dial tcp 10.132.0.77:8001: connect: connection refused"
Sep 09 21:14:54.280804 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:14:54.280754    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl" podUID="857f9131-77cc-42f6-933f-57ab4ee26506" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8001/health\": dial tcp 10.132.0.77:8001: connect: connection refused"
Sep 09 21:15:04.281706 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:04.281632    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl" podUID="857f9131-77cc-42f6-933f-57ab4ee26506" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8001/health\": dial tcp 10.132.0.77:8001: connect: connection refused"
Sep 09 21:15:14.290201 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:14.290168    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:15:14.301850 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:14.301823    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:15:34.475778 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:34.475701    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"]
Sep 09 21:15:34.488272 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:34.488241    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27"]
Sep 09 21:15:36.378813 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:36.378774    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27" podUID="eba0475d-ac7b-4469-881d-15cf6151509d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:15:46.379346 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:46.379308    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27" podUID="eba0475d-ac7b-4469-881d-15cf6151509d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:15:49.499386 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:49.499335    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl" podUID="857f9131-77cc-42f6-933f-57ab4ee26506" containerName="main" containerID="cri-o://f3bbc602ed7ece1ff9bde46ec2e7a66fa7fe5c8df508c8e84a10f9dfd87cf289" gracePeriod=45
Sep 09 21:15:49.508183 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:49.508150    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27" podUID="eba0475d-ac7b-4469-881d-15cf6151509d" containerName="main" containerID="cri-o://862a5be40b684ae0125b642eea8a357e5263b758bcd1ab6f14ca1498f9325060" gracePeriod=45
Sep 09 21:15:50.115955 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.115921    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl" podUID="857f9131-77cc-42f6-933f-57ab4ee26506" containerName="llm-d-routing-sidecar" containerID="cri-o://ace0eca649d8b613e441111f31621c24eee235c7de6fe1efa355d4b2e0a89141" gracePeriod=45
Sep 09 21:15:50.204432 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.204402    2581 generic.go:356] "Generic (PLEG): container finished" podID="857f9131-77cc-42f6-933f-57ab4ee26506" containerID="f3bbc602ed7ece1ff9bde46ec2e7a66fa7fe5c8df508c8e84a10f9dfd87cf289" exitCode=0
Sep 09 21:15:50.204594 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.204497    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl" event={"ID":"857f9131-77cc-42f6-933f-57ab4ee26506","Type":"ContainerDied","Data":"f3bbc602ed7ece1ff9bde46ec2e7a66fa7fe5c8df508c8e84a10f9dfd87cf289"}
Sep 09 21:15:50.356998 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.356973    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:15:50.443917 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.443882    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-model-cache\") pod \"857f9131-77cc-42f6-933f-57ab4ee26506\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") "
Sep 09 21:15:50.444089 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.443937    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-home\") pod \"857f9131-77cc-42f6-933f-57ab4ee26506\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") "
Sep 09 21:15:50.444089 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.443976    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/857f9131-77cc-42f6-933f-57ab4ee26506-cabundle-cert\") pod \"857f9131-77cc-42f6-933f-57ab4ee26506\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") "
Sep 09 21:15:50.444089 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.444035    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-dshm\") pod \"857f9131-77cc-42f6-933f-57ab4ee26506\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") "
Sep 09 21:15:50.444089 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.444062    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-kserve-provision-location\") pod \"857f9131-77cc-42f6-933f-57ab4ee26506\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") "
Sep 09 21:15:50.444089 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.444080    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6r5w\" (UniqueName: \"kubernetes.io/projected/857f9131-77cc-42f6-933f-57ab4ee26506-kube-api-access-q6r5w\") pod \"857f9131-77cc-42f6-933f-57ab4ee26506\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") "
Sep 09 21:15:50.444368 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.444096    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/857f9131-77cc-42f6-933f-57ab4ee26506-tls-certs\") pod \"857f9131-77cc-42f6-933f-57ab4ee26506\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") "
Sep 09 21:15:50.444368 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.444114    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-tmp-dir\") pod \"857f9131-77cc-42f6-933f-57ab4ee26506\" (UID: \"857f9131-77cc-42f6-933f-57ab4ee26506\") "
Sep 09 21:15:50.444368 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.444161    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-model-cache" (OuterVolumeSpecName: "model-cache") pod "857f9131-77cc-42f6-933f-57ab4ee26506" (UID: "857f9131-77cc-42f6-933f-57ab4ee26506"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:50.444368 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.444340    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:15:50.444568 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.444411    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/857f9131-77cc-42f6-933f-57ab4ee26506-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "857f9131-77cc-42f6-933f-57ab4ee26506" (UID: "857f9131-77cc-42f6-933f-57ab4ee26506"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:15:50.444692 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.444664    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-home" (OuterVolumeSpecName: "home") pod "857f9131-77cc-42f6-933f-57ab4ee26506" (UID: "857f9131-77cc-42f6-933f-57ab4ee26506"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:50.446666 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.446638    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/857f9131-77cc-42f6-933f-57ab4ee26506-kube-api-access-q6r5w" (OuterVolumeSpecName: "kube-api-access-q6r5w") pod "857f9131-77cc-42f6-933f-57ab4ee26506" (UID: "857f9131-77cc-42f6-933f-57ab4ee26506"). InnerVolumeSpecName "kube-api-access-q6r5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:15:50.446808 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.446783    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/857f9131-77cc-42f6-933f-57ab4ee26506-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "857f9131-77cc-42f6-933f-57ab4ee26506" (UID: "857f9131-77cc-42f6-933f-57ab4ee26506"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:15:50.446917 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.446893    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-dshm" (OuterVolumeSpecName: "dshm") pod "857f9131-77cc-42f6-933f-57ab4ee26506" (UID: "857f9131-77cc-42f6-933f-57ab4ee26506"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:50.456044 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.456023    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "857f9131-77cc-42f6-933f-57ab4ee26506" (UID: "857f9131-77cc-42f6-933f-57ab4ee26506"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:50.470081 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.470055    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "857f9131-77cc-42f6-933f-57ab4ee26506" (UID: "857f9131-77cc-42f6-933f-57ab4ee26506"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:50.545648 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.545567    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:15:50.545648 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.545595    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/857f9131-77cc-42f6-933f-57ab4ee26506-cabundle-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:15:50.545648 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.545605    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:15:50.545648 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.545613    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-kserve-provision-location\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:15:50.545648 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.545623    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6r5w\" (UniqueName: \"kubernetes.io/projected/857f9131-77cc-42f6-933f-57ab4ee26506-kube-api-access-q6r5w\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:15:50.545648 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.545631    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/857f9131-77cc-42f6-933f-57ab4ee26506-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:15:50.545648 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:50.545640    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/857f9131-77cc-42f6-933f-57ab4ee26506-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:15:51.212175 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:51.212139    2581 generic.go:356] "Generic (PLEG): container finished" podID="857f9131-77cc-42f6-933f-57ab4ee26506" containerID="ace0eca649d8b613e441111f31621c24eee235c7de6fe1efa355d4b2e0a89141" exitCode=0
Sep 09 21:15:51.212340 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:51.212195    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"
Sep 09 21:15:51.212340 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:51.212230    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl" event={"ID":"857f9131-77cc-42f6-933f-57ab4ee26506","Type":"ContainerDied","Data":"ace0eca649d8b613e441111f31621c24eee235c7de6fe1efa355d4b2e0a89141"}
Sep 09 21:15:51.212340 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:51.212288    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl" event={"ID":"857f9131-77cc-42f6-933f-57ab4ee26506","Type":"ContainerDied","Data":"ba3ae0dc1eb9461fba5d40121468d87c0241c5947628804a17eb0238f12220f1"}
Sep 09 21:15:51.212340 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:51.212314    2581 scope.go:117] "RemoveContainer" containerID="f3bbc602ed7ece1ff9bde46ec2e7a66fa7fe5c8df508c8e84a10f9dfd87cf289"
Sep 09 21:15:51.222562 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:51.222544    2581 scope.go:117] "RemoveContainer" containerID="dec3279716f9a008f0fcbbb20034aca478ba1f68d44291f1ee7b8ee01a4f0c95"
Sep 09 21:15:51.230211 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:51.230194    2581 scope.go:117] "RemoveContainer" containerID="ace0eca649d8b613e441111f31621c24eee235c7de6fe1efa355d4b2e0a89141"
Sep 09 21:15:51.237552 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:51.237533    2581 scope.go:117] "RemoveContainer" containerID="f3bbc602ed7ece1ff9bde46ec2e7a66fa7fe5c8df508c8e84a10f9dfd87cf289"
Sep 09 21:15:51.238030 ip-10-0-134-84 kubenswrapper[2581]: E0909 21:15:51.238000    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3bbc602ed7ece1ff9bde46ec2e7a66fa7fe5c8df508c8e84a10f9dfd87cf289\": container with ID starting with f3bbc602ed7ece1ff9bde46ec2e7a66fa7fe5c8df508c8e84a10f9dfd87cf289 not found: ID does not exist" containerID="f3bbc602ed7ece1ff9bde46ec2e7a66fa7fe5c8df508c8e84a10f9dfd87cf289"
Sep 09 21:15:51.238148 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:51.238042    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3bbc602ed7ece1ff9bde46ec2e7a66fa7fe5c8df508c8e84a10f9dfd87cf289"} err="failed to get container status \"f3bbc602ed7ece1ff9bde46ec2e7a66fa7fe5c8df508c8e84a10f9dfd87cf289\": rpc error: code = NotFound desc = could not find container \"f3bbc602ed7ece1ff9bde46ec2e7a66fa7fe5c8df508c8e84a10f9dfd87cf289\": container with ID starting with f3bbc602ed7ece1ff9bde46ec2e7a66fa7fe5c8df508c8e84a10f9dfd87cf289 not found: ID does not exist"
Sep 09 21:15:51.238148 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:51.238065    2581 scope.go:117] "RemoveContainer" containerID="dec3279716f9a008f0fcbbb20034aca478ba1f68d44291f1ee7b8ee01a4f0c95"
Sep 09 21:15:51.238401 ip-10-0-134-84 kubenswrapper[2581]: E0909 21:15:51.238382    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec3279716f9a008f0fcbbb20034aca478ba1f68d44291f1ee7b8ee01a4f0c95\": container with ID starting with dec3279716f9a008f0fcbbb20034aca478ba1f68d44291f1ee7b8ee01a4f0c95 not found: ID does not exist" containerID="dec3279716f9a008f0fcbbb20034aca478ba1f68d44291f1ee7b8ee01a4f0c95"
Sep 09 21:15:51.238502 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:51.238407    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec3279716f9a008f0fcbbb20034aca478ba1f68d44291f1ee7b8ee01a4f0c95"} err="failed to get container status \"dec3279716f9a008f0fcbbb20034aca478ba1f68d44291f1ee7b8ee01a4f0c95\": rpc error: code = NotFound desc = could not find container \"dec3279716f9a008f0fcbbb20034aca478ba1f68d44291f1ee7b8ee01a4f0c95\": container with ID starting with dec3279716f9a008f0fcbbb20034aca478ba1f68d44291f1ee7b8ee01a4f0c95 not found: ID does not exist"
Sep 09 21:15:51.238502 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:51.238423    2581 scope.go:117] "RemoveContainer" containerID="ace0eca649d8b613e441111f31621c24eee235c7de6fe1efa355d4b2e0a89141"
Sep 09 21:15:51.238719 ip-10-0-134-84 kubenswrapper[2581]: E0909 21:15:51.238695    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ace0eca649d8b613e441111f31621c24eee235c7de6fe1efa355d4b2e0a89141\": container with ID starting with ace0eca649d8b613e441111f31621c24eee235c7de6fe1efa355d4b2e0a89141 not found: ID does not exist" containerID="ace0eca649d8b613e441111f31621c24eee235c7de6fe1efa355d4b2e0a89141"
Sep 09 21:15:51.238776 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:51.238723    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ace0eca649d8b613e441111f31621c24eee235c7de6fe1efa355d4b2e0a89141"} err="failed to get container status \"ace0eca649d8b613e441111f31621c24eee235c7de6fe1efa355d4b2e0a89141\": rpc error: code = NotFound desc = could not find container \"ace0eca649d8b613e441111f31621c24eee235c7de6fe1efa355d4b2e0a89141\": container with ID starting with ace0eca649d8b613e441111f31621c24eee235c7de6fe1efa355d4b2e0a89141 not found: ID does not exist"
Sep 09 21:15:51.239064 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:51.239036    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"]
Sep 09 21:15:51.245453 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:51.245428    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7bb8ddd95-z79xl"]
Sep 09 21:15:52.634744 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:52.634710    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="857f9131-77cc-42f6-933f-57ab4ee26506" path="/var/lib/kubelet/pods/857f9131-77cc-42f6-933f-57ab4ee26506/volumes"
Sep 09 21:15:56.379365 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:56.379325    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27" podUID="eba0475d-ac7b-4469-881d-15cf6151509d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:15:56.379859 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:56.379442    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27"
Sep 09 21:15:58.838753 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:58.838646    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 21:15:58.856615 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:58.841815    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 21:15:58.856615 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:58.850017    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 21:15:58.856615 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:15:58.852717    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 21:16:06.379265 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:06.379227    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27" podUID="eba0475d-ac7b-4469-881d-15cf6151509d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:16:16.379067 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:16.379017    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27" podUID="eba0475d-ac7b-4469-881d-15cf6151509d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:16:19.752944 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:19.752917    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27"
Sep 09 21:16:19.895575 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:19.895480    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lx2pw\" (UniqueName: \"kubernetes.io/projected/eba0475d-ac7b-4469-881d-15cf6151509d-kube-api-access-lx2pw\") pod \"eba0475d-ac7b-4469-881d-15cf6151509d\" (UID: \"eba0475d-ac7b-4469-881d-15cf6151509d\") "
Sep 09 21:16:19.895720 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:19.895578    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba0475d-ac7b-4469-881d-15cf6151509d-tls-certs\") pod \"eba0475d-ac7b-4469-881d-15cf6151509d\" (UID: \"eba0475d-ac7b-4469-881d-15cf6151509d\") "
Sep 09 21:16:19.897594 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:19.897566    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba0475d-ac7b-4469-881d-15cf6151509d-kube-api-access-lx2pw" (OuterVolumeSpecName: "kube-api-access-lx2pw") pod "eba0475d-ac7b-4469-881d-15cf6151509d" (UID: "eba0475d-ac7b-4469-881d-15cf6151509d"). InnerVolumeSpecName "kube-api-access-lx2pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:19.897594 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:19.897574    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eba0475d-ac7b-4469-881d-15cf6151509d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eba0475d-ac7b-4469-881d-15cf6151509d" (UID: "eba0475d-ac7b-4469-881d-15cf6151509d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:16:19.997103 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:19.997072    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx2pw\" (UniqueName: \"kubernetes.io/projected/eba0475d-ac7b-4469-881d-15cf6151509d-kube-api-access-lx2pw\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:16:19.997103 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:19.997098    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba0475d-ac7b-4469-881d-15cf6151509d-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:16:20.358477 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:20.358419    2581 generic.go:356] "Generic (PLEG): container finished" podID="eba0475d-ac7b-4469-881d-15cf6151509d" containerID="862a5be40b684ae0125b642eea8a357e5263b758bcd1ab6f14ca1498f9325060" exitCode=0
Sep 09 21:16:20.358654 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:20.358484    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27"
Sep 09 21:16:20.358654 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:20.358506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27" event={"ID":"eba0475d-ac7b-4469-881d-15cf6151509d","Type":"ContainerDied","Data":"862a5be40b684ae0125b642eea8a357e5263b758bcd1ab6f14ca1498f9325060"}
Sep 09 21:16:20.358654 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:20.358559    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27" event={"ID":"eba0475d-ac7b-4469-881d-15cf6151509d","Type":"ContainerDied","Data":"e5106973a1fe6f6cbccab99db211aabd2f11275383cbc55d8ae61e0a726f3d98"}
Sep 09 21:16:20.358654 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:20.358581    2581 scope.go:117] "RemoveContainer" containerID="862a5be40b684ae0125b642eea8a357e5263b758bcd1ab6f14ca1498f9325060"
Sep 09 21:16:20.367371 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:20.367343    2581 scope.go:117] "RemoveContainer" containerID="862a5be40b684ae0125b642eea8a357e5263b758bcd1ab6f14ca1498f9325060"
Sep 09 21:16:20.367643 ip-10-0-134-84 kubenswrapper[2581]: E0909 21:16:20.367624    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"862a5be40b684ae0125b642eea8a357e5263b758bcd1ab6f14ca1498f9325060\": container with ID starting with 862a5be40b684ae0125b642eea8a357e5263b758bcd1ab6f14ca1498f9325060 not found: ID does not exist" containerID="862a5be40b684ae0125b642eea8a357e5263b758bcd1ab6f14ca1498f9325060"
Sep 09 21:16:20.367704 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:20.367651    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"862a5be40b684ae0125b642eea8a357e5263b758bcd1ab6f14ca1498f9325060"} err="failed to get container status \"862a5be40b684ae0125b642eea8a357e5263b758bcd1ab6f14ca1498f9325060\": rpc error: code = NotFound desc = could not find container \"862a5be40b684ae0125b642eea8a357e5263b758bcd1ab6f14ca1498f9325060\": container with ID starting with 862a5be40b684ae0125b642eea8a357e5263b758bcd1ab6f14ca1498f9325060 not found: ID does not exist"
Sep 09 21:16:20.381758 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:20.381735    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27"]
Sep 09 21:16:20.388574 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:20.388552    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schem7h27"]
Sep 09 21:16:20.635496 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:20.635404    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eba0475d-ac7b-4469-881d-15cf6151509d" path="/var/lib/kubelet/pods/eba0475d-ac7b-4469-881d-15cf6151509d/volumes"
Sep 09 21:16:35.451272 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.451233    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"]
Sep 09 21:16:35.451701 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.451574    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="857f9131-77cc-42f6-933f-57ab4ee26506" containerName="llm-d-routing-sidecar"
Sep 09 21:16:35.451701 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.451587    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="857f9131-77cc-42f6-933f-57ab4ee26506" containerName="llm-d-routing-sidecar"
Sep 09 21:16:35.451701 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.451598    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="857f9131-77cc-42f6-933f-57ab4ee26506" containerName="storage-initializer"
Sep 09 21:16:35.451701 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.451603    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="857f9131-77cc-42f6-933f-57ab4ee26506" containerName="storage-initializer"
Sep 09 21:16:35.451701 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.451614    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="857f9131-77cc-42f6-933f-57ab4ee26506" containerName="main"
Sep 09 21:16:35.451701 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.451620    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="857f9131-77cc-42f6-933f-57ab4ee26506" containerName="main"
Sep 09 21:16:35.451701 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.451628    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba0475d-ac7b-4469-881d-15cf6151509d" containerName="main"
Sep 09 21:16:35.451701 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.451634    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba0475d-ac7b-4469-881d-15cf6151509d" containerName="main"
Sep 09 21:16:35.451701 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.451685    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="857f9131-77cc-42f6-933f-57ab4ee26506" containerName="llm-d-routing-sidecar"
Sep 09 21:16:35.451701 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.451695    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eba0475d-ac7b-4469-881d-15cf6151509d" containerName="main"
Sep 09 21:16:35.451701 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.451701    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="857f9131-77cc-42f6-933f-57ab4ee26506" containerName="main"
Sep 09 21:16:35.456050 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.456027    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.458910 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.458881    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 09 21:16:35.459056 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.458931    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:16:35.459056 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.458957    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 09 21:16:35.459178 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.459062    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 09 21:16:35.459229 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.459180    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 09 21:16:35.459784 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.459766    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-pkls4\""
Sep 09 21:16:35.469021 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.469000    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"]
Sep 09 21:16:35.515498 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.515454    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6d7q\" (UniqueName: \"kubernetes.io/projected/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-kube-api-access-w6d7q\") pod \"custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.515633 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.515515    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.515633 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.515541    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.515633 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.515572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.515800 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.515681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.515800 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.515735    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.515800 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.515761    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.515800 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.515791    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.617058 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.617024    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.617232 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.617072    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.617232 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.617098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.617232 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.617124    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.617232 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.617178    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6d7q\" (UniqueName: \"kubernetes.io/projected/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-kube-api-access-w6d7q\") pod \"custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.617232 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.617209    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.617554 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.617241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.617554 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.617288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.617554 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.617511    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.617685 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.617615    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.617726 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.617680    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.617726 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.617698    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.618005 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.617988    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.619428 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.619407    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.619649 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.619631    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.632890 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.632865    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6d7q\" (UniqueName: \"kubernetes.io/projected/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-kube-api-access-w6d7q\") pod \"custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:35.766568 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:35.766533    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:36.046198 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:36.046120    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"]
Sep 09 21:16:36.049880 ip-10-0-134-84 kubenswrapper[2581]: W0909 21:16:36.049853    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69de8d5b_9d9e_47f8_a0c9_d5a53209526c.slice/crio-3c8302bb76fc23215251982eed297a5bf84b3fdf366e2bd18d4bb1be96af6069 WatchSource:0}: Error finding container 3c8302bb76fc23215251982eed297a5bf84b3fdf366e2bd18d4bb1be96af6069: Status 404 returned error can't find the container with id 3c8302bb76fc23215251982eed297a5bf84b3fdf366e2bd18d4bb1be96af6069
Sep 09 21:16:36.437945 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:36.437865    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg" event={"ID":"69de8d5b-9d9e-47f8-a0c9-d5a53209526c","Type":"ContainerStarted","Data":"7ba8216fc496f159918fb8a424b8c6f4f9ee5422087aefcce655b3ee997b2b24"}
Sep 09 21:16:36.437945 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:36.437922    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg" event={"ID":"69de8d5b-9d9e-47f8-a0c9-d5a53209526c","Type":"ContainerStarted","Data":"3c8302bb76fc23215251982eed297a5bf84b3fdf366e2bd18d4bb1be96af6069"}
Sep 09 21:16:38.459173 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:38.459138    2581 generic.go:356] "Generic (PLEG): container finished" podID="69de8d5b-9d9e-47f8-a0c9-d5a53209526c" containerID="7ba8216fc496f159918fb8a424b8c6f4f9ee5422087aefcce655b3ee997b2b24" exitCode=0
Sep 09 21:16:38.459595 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:38.459210    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg" event={"ID":"69de8d5b-9d9e-47f8-a0c9-d5a53209526c","Type":"ContainerDied","Data":"7ba8216fc496f159918fb8a424b8c6f4f9ee5422087aefcce655b3ee997b2b24"}
Sep 09 21:16:39.466954 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:39.466914    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg" event={"ID":"69de8d5b-9d9e-47f8-a0c9-d5a53209526c","Type":"ContainerStarted","Data":"6edf66d10e75ebd58158f9abe5f559f5639a0377141d06f18f3df7ab757b27c6"}
Sep 09 21:16:39.497230 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:39.497175    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg" podStartSLOduration=2.21507191 podStartE2EDuration="4.497154259s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.282082349s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:16:39.49481768 +0000 UTC m=+3941.499994675" watchObservedRunningTime="2026-09-09 21:16:39.497154259 +0000 UTC m=+3941.502331256"
Sep 09 21:16:45.767364 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:45.767317    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:45.767833 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:45.767400    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:16:45.769211 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:45.769182    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg" podUID="69de8d5b-9d9e-47f8-a0c9-d5a53209526c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 09 21:16:55.767058 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:16:55.766995    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg" podUID="69de8d5b-9d9e-47f8-a0c9-d5a53209526c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 09 21:17:05.767839 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:17:05.767796    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg" podUID="69de8d5b-9d9e-47f8-a0c9-d5a53209526c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 09 21:17:15.768065 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:17:15.768017    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg" podUID="69de8d5b-9d9e-47f8-a0c9-d5a53209526c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 09 21:17:25.767764 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:17:25.767722    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg" podUID="69de8d5b-9d9e-47f8-a0c9-d5a53209526c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 09 21:17:35.767170 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:17:35.767131    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg" podUID="69de8d5b-9d9e-47f8-a0c9-d5a53209526c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 09 21:17:45.767522 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:17:45.767479    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg" podUID="69de8d5b-9d9e-47f8-a0c9-d5a53209526c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 09 21:17:55.767363 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:17:55.767318    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg" podUID="69de8d5b-9d9e-47f8-a0c9-d5a53209526c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 09 21:18:05.767347 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:18:05.767298    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg" podUID="69de8d5b-9d9e-47f8-a0c9-d5a53209526c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 09 21:18:15.767379 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:18:15.767331    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg" podUID="69de8d5b-9d9e-47f8-a0c9-d5a53209526c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 09 21:18:25.767852 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:18:25.767611    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg" podUID="69de8d5b-9d9e-47f8-a0c9-d5a53209526c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 09 21:18:35.777713 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:18:35.777678    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:18:35.785615 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:18:35.785590    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:19:15.490760 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:15.490724    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"]
Sep 09 21:19:30.515724 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:30.515674    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg" podUID="69de8d5b-9d9e-47f8-a0c9-d5a53209526c" containerName="main" containerID="cri-o://6edf66d10e75ebd58158f9abe5f559f5639a0377141d06f18f3df7ab757b27c6" gracePeriod=45
Sep 09 21:19:31.263208 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.263180    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:19:31.347267 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.347170    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-cabundle-cert\") pod \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") "
Sep 09 21:19:31.347267 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.347227    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-home\") pod \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") "
Sep 09 21:19:31.347267 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.347264    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-tmp-dir\") pod \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") "
Sep 09 21:19:31.347534 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.347361    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6d7q\" (UniqueName: \"kubernetes.io/projected/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-kube-api-access-w6d7q\") pod \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") "
Sep 09 21:19:31.347534 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.347402    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-kserve-provision-location\") pod \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") "
Sep 09 21:19:31.347534 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.347439    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-dshm\") pod \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") "
Sep 09 21:19:31.347534 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.347481    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-tls-certs\") pod \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") "
Sep 09 21:19:31.347750 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.347542    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-model-cache\") pod \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\" (UID: \"69de8d5b-9d9e-47f8-a0c9-d5a53209526c\") "
Sep 09 21:19:31.347807 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.347671    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "69de8d5b-9d9e-47f8-a0c9-d5a53209526c" (UID: "69de8d5b-9d9e-47f8-a0c9-d5a53209526c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:19:31.347865 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.347854    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-cabundle-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:19:31.348053 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.348027    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-model-cache" (OuterVolumeSpecName: "model-cache") pod "69de8d5b-9d9e-47f8-a0c9-d5a53209526c" (UID: "69de8d5b-9d9e-47f8-a0c9-d5a53209526c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:31.348053 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.348036    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-home" (OuterVolumeSpecName: "home") pod "69de8d5b-9d9e-47f8-a0c9-d5a53209526c" (UID: "69de8d5b-9d9e-47f8-a0c9-d5a53209526c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:31.349827 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.349800    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-dshm" (OuterVolumeSpecName: "dshm") pod "69de8d5b-9d9e-47f8-a0c9-d5a53209526c" (UID: "69de8d5b-9d9e-47f8-a0c9-d5a53209526c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:31.350012 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.349990    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69de8d5b-9d9e-47f8-a0c9-d5a53209526c" (UID: "69de8d5b-9d9e-47f8-a0c9-d5a53209526c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:19:31.350084 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.350023    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-kube-api-access-w6d7q" (OuterVolumeSpecName: "kube-api-access-w6d7q") pod "69de8d5b-9d9e-47f8-a0c9-d5a53209526c" (UID: "69de8d5b-9d9e-47f8-a0c9-d5a53209526c"). InnerVolumeSpecName "kube-api-access-w6d7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:19:31.361688 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.361654    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "69de8d5b-9d9e-47f8-a0c9-d5a53209526c" (UID: "69de8d5b-9d9e-47f8-a0c9-d5a53209526c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:31.375229 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.375184    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "69de8d5b-9d9e-47f8-a0c9-d5a53209526c" (UID: "69de8d5b-9d9e-47f8-a0c9-d5a53209526c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:31.382432 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.382396    2581 generic.go:356] "Generic (PLEG): container finished" podID="69de8d5b-9d9e-47f8-a0c9-d5a53209526c" containerID="6edf66d10e75ebd58158f9abe5f559f5639a0377141d06f18f3df7ab757b27c6" exitCode=0
Sep 09 21:19:31.382606 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.382439    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg" event={"ID":"69de8d5b-9d9e-47f8-a0c9-d5a53209526c","Type":"ContainerDied","Data":"6edf66d10e75ebd58158f9abe5f559f5639a0377141d06f18f3df7ab757b27c6"}
Sep 09 21:19:31.382606 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.382491    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"
Sep 09 21:19:31.382606 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.382535    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg" event={"ID":"69de8d5b-9d9e-47f8-a0c9-d5a53209526c","Type":"ContainerDied","Data":"3c8302bb76fc23215251982eed297a5bf84b3fdf366e2bd18d4bb1be96af6069"}
Sep 09 21:19:31.382606 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.382559    2581 scope.go:117] "RemoveContainer" containerID="6edf66d10e75ebd58158f9abe5f559f5639a0377141d06f18f3df7ab757b27c6"
Sep 09 21:19:31.392001 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.391983    2581 scope.go:117] "RemoveContainer" containerID="7ba8216fc496f159918fb8a424b8c6f4f9ee5422087aefcce655b3ee997b2b24"
Sep 09 21:19:31.399940 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.399918    2581 scope.go:117] "RemoveContainer" containerID="6edf66d10e75ebd58158f9abe5f559f5639a0377141d06f18f3df7ab757b27c6"
Sep 09 21:19:31.400266 ip-10-0-134-84 kubenswrapper[2581]: E0909 21:19:31.400246    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6edf66d10e75ebd58158f9abe5f559f5639a0377141d06f18f3df7ab757b27c6\": container with ID starting with 6edf66d10e75ebd58158f9abe5f559f5639a0377141d06f18f3df7ab757b27c6 not found: ID does not exist" containerID="6edf66d10e75ebd58158f9abe5f559f5639a0377141d06f18f3df7ab757b27c6"
Sep 09 21:19:31.400317 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.400278    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6edf66d10e75ebd58158f9abe5f559f5639a0377141d06f18f3df7ab757b27c6"} err="failed to get container status \"6edf66d10e75ebd58158f9abe5f559f5639a0377141d06f18f3df7ab757b27c6\": rpc error: code = NotFound desc = could not find container \"6edf66d10e75ebd58158f9abe5f559f5639a0377141d06f18f3df7ab757b27c6\": container with ID starting with 6edf66d10e75ebd58158f9abe5f559f5639a0377141d06f18f3df7ab757b27c6 not found: ID does not exist"
Sep 09 21:19:31.400317 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.400295    2581 scope.go:117] "RemoveContainer" containerID="7ba8216fc496f159918fb8a424b8c6f4f9ee5422087aefcce655b3ee997b2b24"
Sep 09 21:19:31.400592 ip-10-0-134-84 kubenswrapper[2581]: E0909 21:19:31.400573    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba8216fc496f159918fb8a424b8c6f4f9ee5422087aefcce655b3ee997b2b24\": container with ID starting with 7ba8216fc496f159918fb8a424b8c6f4f9ee5422087aefcce655b3ee997b2b24 not found: ID does not exist" containerID="7ba8216fc496f159918fb8a424b8c6f4f9ee5422087aefcce655b3ee997b2b24"
Sep 09 21:19:31.400646 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.400598    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba8216fc496f159918fb8a424b8c6f4f9ee5422087aefcce655b3ee997b2b24"} err="failed to get container status \"7ba8216fc496f159918fb8a424b8c6f4f9ee5422087aefcce655b3ee997b2b24\": rpc error: code = NotFound desc = could not find container \"7ba8216fc496f159918fb8a424b8c6f4f9ee5422087aefcce655b3ee997b2b24\": container with ID starting with 7ba8216fc496f159918fb8a424b8c6f4f9ee5422087aefcce655b3ee997b2b24 not found: ID does not exist"
Sep 09 21:19:31.409195 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.409165    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"]
Sep 09 21:19:31.417467 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.417433    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7886bbd8d4-dg4dg"]
Sep 09 21:19:31.448821 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.448783    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:19:31.448821 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.448814    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6d7q\" (UniqueName: \"kubernetes.io/projected/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-kube-api-access-w6d7q\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:19:31.448821 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.448824    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-kserve-provision-location\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:19:31.449023 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.448834    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:19:31.449023 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.448843    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:19:31.449023 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.448851    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:19:31.449023 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:31.448858    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69de8d5b-9d9e-47f8-a0c9-d5a53209526c-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:19:32.635930 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:32.635897    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69de8d5b-9d9e-47f8-a0c9-d5a53209526c" path="/var/lib/kubelet/pods/69de8d5b-9d9e-47f8-a0c9-d5a53209526c/volumes"
Sep 09 21:19:35.214186 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.214150    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"]
Sep 09 21:19:35.214592 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.214471    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69de8d5b-9d9e-47f8-a0c9-d5a53209526c" containerName="storage-initializer"
Sep 09 21:19:35.214592 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.214487    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="69de8d5b-9d9e-47f8-a0c9-d5a53209526c" containerName="storage-initializer"
Sep 09 21:19:35.214592 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.214499    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69de8d5b-9d9e-47f8-a0c9-d5a53209526c" containerName="main"
Sep 09 21:19:35.214592 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.214507    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="69de8d5b-9d9e-47f8-a0c9-d5a53209526c" containerName="main"
Sep 09 21:19:35.214592 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.214577    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="69de8d5b-9d9e-47f8-a0c9-d5a53209526c" containerName="main"
Sep 09 21:19:35.219182 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.219164    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.221857 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.221831    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 09 21:19:35.221997 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.221844    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 09 21:19:35.221997 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.221907    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-4bpx4\""
Sep 09 21:19:35.221997 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.221857    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 09 21:19:35.222747 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.222728    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 09 21:19:35.222852 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.222732    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:19:35.234227 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.234198    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"]
Sep 09 21:19:35.382529 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.382498    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-dshm\") pod \"tls-verification-test-kserve-776dcb8b74-fc624\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.382529 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.382533    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1c12ba-7b33-4755-a05f-cf834e04eadf-tls-certs\") pod \"tls-verification-test-kserve-776dcb8b74-fc624\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.382732 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.382558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlh46\" (UniqueName: \"kubernetes.io/projected/cd1c12ba-7b33-4755-a05f-cf834e04eadf-kube-api-access-jlh46\") pod \"tls-verification-test-kserve-776dcb8b74-fc624\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.382732 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.382614    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-kserve-provision-location\") pod \"tls-verification-test-kserve-776dcb8b74-fc624\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.382732 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.382638    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd1c12ba-7b33-4755-a05f-cf834e04eadf-cabundle-cert\") pod \"tls-verification-test-kserve-776dcb8b74-fc624\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.382732 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.382707    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-model-cache\") pod \"tls-verification-test-kserve-776dcb8b74-fc624\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.382863 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.382738    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-home\") pod \"tls-verification-test-kserve-776dcb8b74-fc624\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.382863 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.382779    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-tmp-dir\") pod \"tls-verification-test-kserve-776dcb8b74-fc624\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.483634 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.483595    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-home\") pod \"tls-verification-test-kserve-776dcb8b74-fc624\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.483836 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.483662    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-tmp-dir\") pod \"tls-verification-test-kserve-776dcb8b74-fc624\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.483836 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.483703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-dshm\") pod \"tls-verification-test-kserve-776dcb8b74-fc624\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.483836 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.483719    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1c12ba-7b33-4755-a05f-cf834e04eadf-tls-certs\") pod \"tls-verification-test-kserve-776dcb8b74-fc624\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.483836 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.483736    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlh46\" (UniqueName: \"kubernetes.io/projected/cd1c12ba-7b33-4755-a05f-cf834e04eadf-kube-api-access-jlh46\") pod \"tls-verification-test-kserve-776dcb8b74-fc624\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.483836 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.483784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-kserve-provision-location\") pod \"tls-verification-test-kserve-776dcb8b74-fc624\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.483836 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.483824    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd1c12ba-7b33-4755-a05f-cf834e04eadf-cabundle-cert\") pod \"tls-verification-test-kserve-776dcb8b74-fc624\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.484235 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.483869    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-model-cache\") pod \"tls-verification-test-kserve-776dcb8b74-fc624\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.484235 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.484060    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-home\") pod \"tls-verification-test-kserve-776dcb8b74-fc624\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.484235 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.484133    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-tmp-dir\") pod \"tls-verification-test-kserve-776dcb8b74-fc624\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.484235 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.484151    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-kserve-provision-location\") pod \"tls-verification-test-kserve-776dcb8b74-fc624\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.484397 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.484245    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-model-cache\") pod \"tls-verification-test-kserve-776dcb8b74-fc624\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.484533 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.484514    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd1c12ba-7b33-4755-a05f-cf834e04eadf-cabundle-cert\") pod \"tls-verification-test-kserve-776dcb8b74-fc624\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.486116 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.486096    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-dshm\") pod \"tls-verification-test-kserve-776dcb8b74-fc624\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.486273 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.486255    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1c12ba-7b33-4755-a05f-cf834e04eadf-tls-certs\") pod \"tls-verification-test-kserve-776dcb8b74-fc624\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.492919 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.492897    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlh46\" (UniqueName: \"kubernetes.io/projected/cd1c12ba-7b33-4755-a05f-cf834e04eadf-kube-api-access-jlh46\") pod \"tls-verification-test-kserve-776dcb8b74-fc624\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.529856 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.529823    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:35.810627 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:35.810597    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"]
Sep 09 21:19:35.812622 ip-10-0-134-84 kubenswrapper[2581]: W0909 21:19:35.812584    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1c12ba_7b33_4755_a05f_cf834e04eadf.slice/crio-d06122ef5e269c568919a2ed2f70eab9a4bd7ca252c19f1868cd1cbaf2fcccbe WatchSource:0}: Error finding container d06122ef5e269c568919a2ed2f70eab9a4bd7ca252c19f1868cd1cbaf2fcccbe: Status 404 returned error can't find the container with id d06122ef5e269c568919a2ed2f70eab9a4bd7ca252c19f1868cd1cbaf2fcccbe
Sep 09 21:19:36.410484 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:36.410435    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624" event={"ID":"cd1c12ba-7b33-4755-a05f-cf834e04eadf","Type":"ContainerStarted","Data":"40f3e3b862d12d124133d5cce65dbd2b301c006d75d04958c1316790c9345c29"}
Sep 09 21:19:36.411042 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:36.410513    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624" event={"ID":"cd1c12ba-7b33-4755-a05f-cf834e04eadf","Type":"ContainerStarted","Data":"d06122ef5e269c568919a2ed2f70eab9a4bd7ca252c19f1868cd1cbaf2fcccbe"}
Sep 09 21:19:38.423042 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:38.423010    2581 generic.go:356] "Generic (PLEG): container finished" podID="cd1c12ba-7b33-4755-a05f-cf834e04eadf" containerID="40f3e3b862d12d124133d5cce65dbd2b301c006d75d04958c1316790c9345c29" exitCode=0
Sep 09 21:19:38.423550 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:38.423061    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624" event={"ID":"cd1c12ba-7b33-4755-a05f-cf834e04eadf","Type":"ContainerDied","Data":"40f3e3b862d12d124133d5cce65dbd2b301c006d75d04958c1316790c9345c29"}
Sep 09 21:19:39.430970 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:39.430933    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624" event={"ID":"cd1c12ba-7b33-4755-a05f-cf834e04eadf","Type":"ContainerStarted","Data":"94ae6b2a05b364c0dd44e05085a81e2fbd96022d95cdd9e4ad73aadb240ae58b"}
Sep 09 21:19:39.459977 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:39.459924    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624" podStartSLOduration=2.169726337 podStartE2EDuration="4.459908855s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.290182518s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:19:39.457042954 +0000 UTC m=+4121.462219960" watchObservedRunningTime="2026-09-09 21:19:39.459908855 +0000 UTC m=+4121.465085850"
Sep 09 21:19:39.530574 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:39.530539    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:45.530064 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:45.530022    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:19:45.531777 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:45.531742    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624" podUID="cd1c12ba-7b33-4755-a05f-cf834e04eadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:19:55.531268 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:19:55.531173    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624" podUID="cd1c12ba-7b33-4755-a05f-cf834e04eadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:20:05.530622 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:05.530576    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624" podUID="cd1c12ba-7b33-4755-a05f-cf834e04eadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:20:15.530635 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:15.530594    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624" podUID="cd1c12ba-7b33-4755-a05f-cf834e04eadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:20:20.040408 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.040372    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"]
Sep 09 21:20:20.042767 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.042746    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.045705 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.045679    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 09 21:20:20.046687 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.046661    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:20:20.046792 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.046704    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 09 21:20:20.046974 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.046953    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 09 21:20:20.047122 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.046983    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 09 21:20:20.047252 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.047233    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-886t2\""
Sep 09 21:20:20.058908 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.058884    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"]
Sep 09 21:20:20.172789 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.172745    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-home\") pod \"router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.172789 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.172789    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa267f05-3c76-4616-b33f-1b3ffd19eb78-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.173060 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.172822    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.173060 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.172874    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.173060 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.172900    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.173060 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.172929    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa267f05-3c76-4616-b33f-1b3ffd19eb78-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.173060 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.173028    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb2vh\" (UniqueName: \"kubernetes.io/projected/fa267f05-3c76-4616-b33f-1b3ffd19eb78-kube-api-access-vb2vh\") pod \"router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.173346 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.173158    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.274563 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.274522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.274563 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.274563    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.274825 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.274586    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa267f05-3c76-4616-b33f-1b3ffd19eb78-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.274825 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.274606    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb2vh\" (UniqueName: \"kubernetes.io/projected/fa267f05-3c76-4616-b33f-1b3ffd19eb78-kube-api-access-vb2vh\") pod \"router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.274825 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.274673    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.274825 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.274708    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-home\") pod \"router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.274825 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.274736    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa267f05-3c76-4616-b33f-1b3ffd19eb78-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.274825 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.274769    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.275173 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.275043    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.275279 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.275256    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.275369 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.275349    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.275435 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.275402    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-home\") pod \"router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.275565 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.275544    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa267f05-3c76-4616-b33f-1b3ffd19eb78-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.276840 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.276817    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.277171 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.277149    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa267f05-3c76-4616-b33f-1b3ffd19eb78-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.284174 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.284149    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb2vh\" (UniqueName: \"kubernetes.io/projected/fa267f05-3c76-4616-b33f-1b3ffd19eb78-kube-api-access-vb2vh\") pod \"router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.354507 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.354403    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:20.641633 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:20.641358    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"]
Sep 09 21:20:20.645199 ip-10-0-134-84 kubenswrapper[2581]: W0909 21:20:20.645167    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa267f05_3c76_4616_b33f_1b3ffd19eb78.slice/crio-872a3fda20c47309c16f621f901a626025eac3473c6efa8082e670f8c314f067 WatchSource:0}: Error finding container 872a3fda20c47309c16f621f901a626025eac3473c6efa8082e670f8c314f067: Status 404 returned error can't find the container with id 872a3fda20c47309c16f621f901a626025eac3473c6efa8082e670f8c314f067
Sep 09 21:20:21.654107 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:21.654065    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf" event={"ID":"fa267f05-3c76-4616-b33f-1b3ffd19eb78","Type":"ContainerStarted","Data":"c152c6c363c70b4667faac53a34cd8276c466124dfe22ec5ffcf1523b859ef5d"}
Sep 09 21:20:21.654529 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:21.654132    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf" event={"ID":"fa267f05-3c76-4616-b33f-1b3ffd19eb78","Type":"ContainerStarted","Data":"872a3fda20c47309c16f621f901a626025eac3473c6efa8082e670f8c314f067"}
Sep 09 21:20:23.668960 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:23.668928    2581 generic.go:356] "Generic (PLEG): container finished" podID="fa267f05-3c76-4616-b33f-1b3ffd19eb78" containerID="c152c6c363c70b4667faac53a34cd8276c466124dfe22ec5ffcf1523b859ef5d" exitCode=0
Sep 09 21:20:23.669367 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:23.669012    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf" event={"ID":"fa267f05-3c76-4616-b33f-1b3ffd19eb78","Type":"ContainerDied","Data":"c152c6c363c70b4667faac53a34cd8276c466124dfe22ec5ffcf1523b859ef5d"}
Sep 09 21:20:24.678597 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:24.678562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf" event={"ID":"fa267f05-3c76-4616-b33f-1b3ffd19eb78","Type":"ContainerStarted","Data":"2ae90b09d87dc5c5c8edda29cc59ba527d51e794b546cb03616af2757088ed4f"}
Sep 09 21:20:24.710941 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:24.710869    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf" podStartSLOduration=2.306268434 podStartE2EDuration="4.710845794s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.40457736s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:20:24.706765245 +0000 UTC m=+4166.711942240" watchObservedRunningTime="2026-09-09 21:20:24.710845794 +0000 UTC m=+4166.716022790"
Sep 09 21:20:25.530856 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:25.530805    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624" podUID="cd1c12ba-7b33-4755-a05f-cf834e04eadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:20:30.355200 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:30.355156    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:30.355924 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:30.355270    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:20:30.356765 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:30.356725    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf" podUID="fa267f05-3c76-4616-b33f-1b3ffd19eb78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 09 21:20:35.531262 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:35.531199    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624" podUID="cd1c12ba-7b33-4755-a05f-cf834e04eadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:20:40.355549 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:40.355503    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf" podUID="fa267f05-3c76-4616-b33f-1b3ffd19eb78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 09 21:20:45.530979 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:45.530931    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624" podUID="cd1c12ba-7b33-4755-a05f-cf834e04eadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:20:50.355248 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:50.355195    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf" podUID="fa267f05-3c76-4616-b33f-1b3ffd19eb78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 09 21:20:55.530864 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:55.530820    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624" podUID="cd1c12ba-7b33-4755-a05f-cf834e04eadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:20:58.888538 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:58.888392    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 21:20:58.891281 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:58.891005    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 21:20:58.892187 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:58.892161    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 21:20:58.894192 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:20:58.894171    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 21:21:00.355534 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:00.355488    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf" podUID="fa267f05-3c76-4616-b33f-1b3ffd19eb78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 09 21:21:05.530542 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:05.530489    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624" podUID="cd1c12ba-7b33-4755-a05f-cf834e04eadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:21:10.355025 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:10.354980    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf" podUID="fa267f05-3c76-4616-b33f-1b3ffd19eb78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 09 21:21:15.531147 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:15.531100    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624" podUID="cd1c12ba-7b33-4755-a05f-cf834e04eadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:21:20.355667 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:20.355621    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf" podUID="fa267f05-3c76-4616-b33f-1b3ffd19eb78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 09 21:21:25.530617 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:25.530515    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624" podUID="cd1c12ba-7b33-4755-a05f-cf834e04eadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:21:30.354960 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:30.354912    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf" podUID="fa267f05-3c76-4616-b33f-1b3ffd19eb78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 09 21:21:35.540611 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:35.540580    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:21:35.548985 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:35.548957    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:21:40.354984 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:40.354942    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf" podUID="fa267f05-3c76-4616-b33f-1b3ffd19eb78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 09 21:21:43.783935 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:43.783902    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"]
Sep 09 21:21:50.355417 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:50.355372    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf" podUID="fa267f05-3c76-4616-b33f-1b3ffd19eb78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 09 21:21:58.807400 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:58.807340    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624" podUID="cd1c12ba-7b33-4755-a05f-cf834e04eadf" containerName="main" containerID="cri-o://94ae6b2a05b364c0dd44e05085a81e2fbd96022d95cdd9e4ad73aadb240ae58b" gracePeriod=45
Sep 09 21:21:59.541637 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.541587    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624" podUID="cd1c12ba-7b33-4755-a05f-cf834e04eadf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:21:59.792608 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.792534    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:21:59.831247 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.831214    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlh46\" (UniqueName: \"kubernetes.io/projected/cd1c12ba-7b33-4755-a05f-cf834e04eadf-kube-api-access-jlh46\") pod \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") "
Sep 09 21:21:59.831735 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.831311    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-dshm\") pod \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") "
Sep 09 21:21:59.831735 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.831360    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1c12ba-7b33-4755-a05f-cf834e04eadf-tls-certs\") pod \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") "
Sep 09 21:21:59.831735 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.831402    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-kserve-provision-location\") pod \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") "
Sep 09 21:21:59.831735 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.831448    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-tmp-dir\") pod \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") "
Sep 09 21:21:59.831735 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.831499    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd1c12ba-7b33-4755-a05f-cf834e04eadf-cabundle-cert\") pod \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") "
Sep 09 21:21:59.831735 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.831527    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-home\") pod \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") "
Sep 09 21:21:59.831735 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.831553    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-model-cache\") pod \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\" (UID: \"cd1c12ba-7b33-4755-a05f-cf834e04eadf\") "
Sep 09 21:21:59.832131 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.832002    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd1c12ba-7b33-4755-a05f-cf834e04eadf" (UID: "cd1c12ba-7b33-4755-a05f-cf834e04eadf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:21:59.832492 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.832383    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd1c12ba-7b33-4755-a05f-cf834e04eadf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cd1c12ba-7b33-4755-a05f-cf834e04eadf" (UID: "cd1c12ba-7b33-4755-a05f-cf834e04eadf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:21:59.832875 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.832830    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-home" (OuterVolumeSpecName: "home") pod "cd1c12ba-7b33-4755-a05f-cf834e04eadf" (UID: "cd1c12ba-7b33-4755-a05f-cf834e04eadf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:21:59.834860 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.834794    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-dshm" (OuterVolumeSpecName: "dshm") pod "cd1c12ba-7b33-4755-a05f-cf834e04eadf" (UID: "cd1c12ba-7b33-4755-a05f-cf834e04eadf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:21:59.834860 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.834795    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd1c12ba-7b33-4755-a05f-cf834e04eadf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd1c12ba-7b33-4755-a05f-cf834e04eadf" (UID: "cd1c12ba-7b33-4755-a05f-cf834e04eadf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:21:59.835522 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.835494    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1c12ba-7b33-4755-a05f-cf834e04eadf-kube-api-access-jlh46" (OuterVolumeSpecName: "kube-api-access-jlh46") pod "cd1c12ba-7b33-4755-a05f-cf834e04eadf" (UID: "cd1c12ba-7b33-4755-a05f-cf834e04eadf"). InnerVolumeSpecName "kube-api-access-jlh46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:21:59.852293 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.852263    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd1c12ba-7b33-4755-a05f-cf834e04eadf" (UID: "cd1c12ba-7b33-4755-a05f-cf834e04eadf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:21:59.863304 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.863274    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd1c12ba-7b33-4755-a05f-cf834e04eadf" (UID: "cd1c12ba-7b33-4755-a05f-cf834e04eadf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:21:59.932898 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.932867    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:21:59.932898 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.932895    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1c12ba-7b33-4755-a05f-cf834e04eadf-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:21:59.933077 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.932914    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-kserve-provision-location\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:21:59.933077 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.932923    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:21:59.933077 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.932932    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd1c12ba-7b33-4755-a05f-cf834e04eadf-cabundle-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:21:59.933077 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.932939    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:21:59.933077 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.932947    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1c12ba-7b33-4755-a05f-cf834e04eadf-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:21:59.933077 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:21:59.932954    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlh46\" (UniqueName: \"kubernetes.io/projected/cd1c12ba-7b33-4755-a05f-cf834e04eadf-kube-api-access-jlh46\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:22:00.236637 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:00.236594    2581 generic.go:356] "Generic (PLEG): container finished" podID="cd1c12ba-7b33-4755-a05f-cf834e04eadf" containerID="94ae6b2a05b364c0dd44e05085a81e2fbd96022d95cdd9e4ad73aadb240ae58b" exitCode=0
Sep 09 21:22:00.236828 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:00.236650    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"
Sep 09 21:22:00.236828 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:00.236671    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624" event={"ID":"cd1c12ba-7b33-4755-a05f-cf834e04eadf","Type":"ContainerDied","Data":"94ae6b2a05b364c0dd44e05085a81e2fbd96022d95cdd9e4ad73aadb240ae58b"}
Sep 09 21:22:00.236828 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:00.236722    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624" event={"ID":"cd1c12ba-7b33-4755-a05f-cf834e04eadf","Type":"ContainerDied","Data":"d06122ef5e269c568919a2ed2f70eab9a4bd7ca252c19f1868cd1cbaf2fcccbe"}
Sep 09 21:22:00.236828 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:00.236744    2581 scope.go:117] "RemoveContainer" containerID="94ae6b2a05b364c0dd44e05085a81e2fbd96022d95cdd9e4ad73aadb240ae58b"
Sep 09 21:22:00.245857 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:00.245839    2581 scope.go:117] "RemoveContainer" containerID="40f3e3b862d12d124133d5cce65dbd2b301c006d75d04958c1316790c9345c29"
Sep 09 21:22:00.253761 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:00.253739    2581 scope.go:117] "RemoveContainer" containerID="94ae6b2a05b364c0dd44e05085a81e2fbd96022d95cdd9e4ad73aadb240ae58b"
Sep 09 21:22:00.254038 ip-10-0-134-84 kubenswrapper[2581]: E0909 21:22:00.254016    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94ae6b2a05b364c0dd44e05085a81e2fbd96022d95cdd9e4ad73aadb240ae58b\": container with ID starting with 94ae6b2a05b364c0dd44e05085a81e2fbd96022d95cdd9e4ad73aadb240ae58b not found: ID does not exist" containerID="94ae6b2a05b364c0dd44e05085a81e2fbd96022d95cdd9e4ad73aadb240ae58b"
Sep 09 21:22:00.254133 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:00.254045    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94ae6b2a05b364c0dd44e05085a81e2fbd96022d95cdd9e4ad73aadb240ae58b"} err="failed to get container status \"94ae6b2a05b364c0dd44e05085a81e2fbd96022d95cdd9e4ad73aadb240ae58b\": rpc error: code = NotFound desc = could not find container \"94ae6b2a05b364c0dd44e05085a81e2fbd96022d95cdd9e4ad73aadb240ae58b\": container with ID starting with 94ae6b2a05b364c0dd44e05085a81e2fbd96022d95cdd9e4ad73aadb240ae58b not found: ID does not exist"
Sep 09 21:22:00.254133 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:00.254062    2581 scope.go:117] "RemoveContainer" containerID="40f3e3b862d12d124133d5cce65dbd2b301c006d75d04958c1316790c9345c29"
Sep 09 21:22:00.254312 ip-10-0-134-84 kubenswrapper[2581]: E0909 21:22:00.254293    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f3e3b862d12d124133d5cce65dbd2b301c006d75d04958c1316790c9345c29\": container with ID starting with 40f3e3b862d12d124133d5cce65dbd2b301c006d75d04958c1316790c9345c29 not found: ID does not exist" containerID="40f3e3b862d12d124133d5cce65dbd2b301c006d75d04958c1316790c9345c29"
Sep 09 21:22:00.254357 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:00.254316    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f3e3b862d12d124133d5cce65dbd2b301c006d75d04958c1316790c9345c29"} err="failed to get container status \"40f3e3b862d12d124133d5cce65dbd2b301c006d75d04958c1316790c9345c29\": rpc error: code = NotFound desc = could not find container \"40f3e3b862d12d124133d5cce65dbd2b301c006d75d04958c1316790c9345c29\": container with ID starting with 40f3e3b862d12d124133d5cce65dbd2b301c006d75d04958c1316790c9345c29 not found: ID does not exist"
Sep 09 21:22:00.264335 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:00.264311    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"]
Sep 09 21:22:00.270264 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:00.270235    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-776dcb8b74-fc624"]
Sep 09 21:22:00.355617 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:00.355575    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf" podUID="fa267f05-3c76-4616-b33f-1b3ffd19eb78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 09 21:22:00.636037 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:00.635959    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd1c12ba-7b33-4755-a05f-cf834e04eadf" path="/var/lib/kubelet/pods/cd1c12ba-7b33-4755-a05f-cf834e04eadf/volumes"
Sep 09 21:22:10.354872 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:10.354831    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf" podUID="fa267f05-3c76-4616-b33f-1b3ffd19eb78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 09 21:22:20.364598 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:20.364569    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:22:20.372218 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:20.372190    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:22:29.706295 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:29.706262    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"]
Sep 09 21:22:37.996015 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:37.995980    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"]
Sep 09 21:22:37.996504 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:37.996483    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd1c12ba-7b33-4755-a05f-cf834e04eadf" containerName="storage-initializer"
Sep 09 21:22:37.996587 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:37.996507    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd1c12ba-7b33-4755-a05f-cf834e04eadf" containerName="storage-initializer"
Sep 09 21:22:37.996587 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:37.996521    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd1c12ba-7b33-4755-a05f-cf834e04eadf" containerName="main"
Sep 09 21:22:37.996587 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:37.996529    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd1c12ba-7b33-4755-a05f-cf834e04eadf" containerName="main"
Sep 09 21:22:37.996746 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:37.996640    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd1c12ba-7b33-4755-a05f-cf834e04eadf" containerName="main"
Sep 09 21:22:37.999961 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:37.999939    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.002652 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.002628    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 09 21:22:38.002745 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.002677    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 09 21:22:38.003206 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.003189    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 09 21:22:38.003269 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.003219    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 09 21:22:38.003269 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.003261    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:22:38.003716 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.003693    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-wcs7q\""
Sep 09 21:22:38.012571 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.012550    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"]
Sep 09 21:22:38.075124 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.075083    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-home\") pod \"prestop-hook-test-kserve-78bfbf4ccd-dfqmz\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.075297 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.075133    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-model-cache\") pod \"prestop-hook-test-kserve-78bfbf4ccd-dfqmz\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.075297 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.075152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-tmp-dir\") pod \"prestop-hook-test-kserve-78bfbf4ccd-dfqmz\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.075297 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.075171    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-tls-certs\") pod \"prestop-hook-test-kserve-78bfbf4ccd-dfqmz\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.075297 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.075195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-cabundle-cert\") pod \"prestop-hook-test-kserve-78bfbf4ccd-dfqmz\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.075297 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.075227    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-kserve-provision-location\") pod \"prestop-hook-test-kserve-78bfbf4ccd-dfqmz\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.075297 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.075260    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnkws\" (UniqueName: \"kubernetes.io/projected/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-kube-api-access-gnkws\") pod \"prestop-hook-test-kserve-78bfbf4ccd-dfqmz\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.075544 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.075333    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-dshm\") pod \"prestop-hook-test-kserve-78bfbf4ccd-dfqmz\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.175906 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.175866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-home\") pod \"prestop-hook-test-kserve-78bfbf4ccd-dfqmz\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.175906 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.175909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-model-cache\") pod \"prestop-hook-test-kserve-78bfbf4ccd-dfqmz\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.176151 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.175925    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-tmp-dir\") pod \"prestop-hook-test-kserve-78bfbf4ccd-dfqmz\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.176151 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.175943    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-tls-certs\") pod \"prestop-hook-test-kserve-78bfbf4ccd-dfqmz\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.176151 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.175958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-cabundle-cert\") pod \"prestop-hook-test-kserve-78bfbf4ccd-dfqmz\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.176151 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.175983    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-kserve-provision-location\") pod \"prestop-hook-test-kserve-78bfbf4ccd-dfqmz\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.176151 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.176009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnkws\" (UniqueName: \"kubernetes.io/projected/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-kube-api-access-gnkws\") pod \"prestop-hook-test-kserve-78bfbf4ccd-dfqmz\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.176151 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.176038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-dshm\") pod \"prestop-hook-test-kserve-78bfbf4ccd-dfqmz\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.176451 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.176335    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-home\") pod \"prestop-hook-test-kserve-78bfbf4ccd-dfqmz\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.176451 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.176427    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-tmp-dir\") pod \"prestop-hook-test-kserve-78bfbf4ccd-dfqmz\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.176592 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.176510    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-kserve-provision-location\") pod \"prestop-hook-test-kserve-78bfbf4ccd-dfqmz\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.176592 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.176577    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-model-cache\") pod \"prestop-hook-test-kserve-78bfbf4ccd-dfqmz\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.176935 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.176912    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-cabundle-cert\") pod \"prestop-hook-test-kserve-78bfbf4ccd-dfqmz\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.178401 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.178384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-dshm\") pod \"prestop-hook-test-kserve-78bfbf4ccd-dfqmz\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.178544 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.178526    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-tls-certs\") pod \"prestop-hook-test-kserve-78bfbf4ccd-dfqmz\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.187607 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.187584    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnkws\" (UniqueName: \"kubernetes.io/projected/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-kube-api-access-gnkws\") pod \"prestop-hook-test-kserve-78bfbf4ccd-dfqmz\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.310589 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.310505    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:38.590191 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:38.590160    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"]
Sep 09 21:22:38.591807 ip-10-0-134-84 kubenswrapper[2581]: W0909 21:22:38.591774    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aed7d4e_5ef2_43ac_a0fd_3569e87f5675.slice/crio-ad06392f039d1baec853eea131e7ec88b87a50ab84e630d320c7fa7dc4ca441b WatchSource:0}: Error finding container ad06392f039d1baec853eea131e7ec88b87a50ab84e630d320c7fa7dc4ca441b: Status 404 returned error can't find the container with id ad06392f039d1baec853eea131e7ec88b87a50ab84e630d320c7fa7dc4ca441b
Sep 09 21:22:39.446510 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:39.446445    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz" event={"ID":"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675","Type":"ContainerStarted","Data":"d5aad78020f713a4ebf234b5316a59fd0d110e0251f2c30ecf8077126fe6ddd3"}
Sep 09 21:22:39.447004 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:39.446555    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz" event={"ID":"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675","Type":"ContainerStarted","Data":"ad06392f039d1baec853eea131e7ec88b87a50ab84e630d320c7fa7dc4ca441b"}
Sep 09 21:22:41.463430 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:41.463398    2581 generic.go:356] "Generic (PLEG): container finished" podID="4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" containerID="d5aad78020f713a4ebf234b5316a59fd0d110e0251f2c30ecf8077126fe6ddd3" exitCode=0
Sep 09 21:22:41.463990 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:41.463493    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz" event={"ID":"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675","Type":"ContainerDied","Data":"d5aad78020f713a4ebf234b5316a59fd0d110e0251f2c30ecf8077126fe6ddd3"}
Sep 09 21:22:42.470775 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:42.470738    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz" event={"ID":"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675","Type":"ContainerStarted","Data":"3d7444941194f247e2def7548ab1882752cdc6c59873690c78317a8850142977"}
Sep 09 21:22:42.496219 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:42.496170    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz" podStartSLOduration=2.848549694 podStartE2EDuration="5.496155036s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.647605342s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:22:42.496006889 +0000 UTC m=+4304.501183884" watchObservedRunningTime="2026-09-09 21:22:42.496155036 +0000 UTC m=+4304.501332060"
Sep 09 21:22:43.311362 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:43.311313    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:44.728932 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:44.728855    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf" podUID="fa267f05-3c76-4616-b33f-1b3ffd19eb78" containerName="main" containerID="cri-o://2ae90b09d87dc5c5c8edda29cc59ba527d51e794b546cb03616af2757088ed4f" gracePeriod=45
Sep 09 21:22:45.478285 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.478259    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:22:45.487622 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.487592    2581 generic.go:356] "Generic (PLEG): container finished" podID="fa267f05-3c76-4616-b33f-1b3ffd19eb78" containerID="2ae90b09d87dc5c5c8edda29cc59ba527d51e794b546cb03616af2757088ed4f" exitCode=0
Sep 09 21:22:45.487753 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.487650    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"
Sep 09 21:22:45.487753 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.487662    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf" event={"ID":"fa267f05-3c76-4616-b33f-1b3ffd19eb78","Type":"ContainerDied","Data":"2ae90b09d87dc5c5c8edda29cc59ba527d51e794b546cb03616af2757088ed4f"}
Sep 09 21:22:45.487753 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.487708    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf" event={"ID":"fa267f05-3c76-4616-b33f-1b3ffd19eb78","Type":"ContainerDied","Data":"872a3fda20c47309c16f621f901a626025eac3473c6efa8082e670f8c314f067"}
Sep 09 21:22:45.487753 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.487734    2581 scope.go:117] "RemoveContainer" containerID="2ae90b09d87dc5c5c8edda29cc59ba527d51e794b546cb03616af2757088ed4f"
Sep 09 21:22:45.497637 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.497615    2581 scope.go:117] "RemoveContainer" containerID="c152c6c363c70b4667faac53a34cd8276c466124dfe22ec5ffcf1523b859ef5d"
Sep 09 21:22:45.506733 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.506699    2581 scope.go:117] "RemoveContainer" containerID="2ae90b09d87dc5c5c8edda29cc59ba527d51e794b546cb03616af2757088ed4f"
Sep 09 21:22:45.507082 ip-10-0-134-84 kubenswrapper[2581]: E0909 21:22:45.507052    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ae90b09d87dc5c5c8edda29cc59ba527d51e794b546cb03616af2757088ed4f\": container with ID starting with 2ae90b09d87dc5c5c8edda29cc59ba527d51e794b546cb03616af2757088ed4f not found: ID does not exist" containerID="2ae90b09d87dc5c5c8edda29cc59ba527d51e794b546cb03616af2757088ed4f"
Sep 09 21:22:45.507168 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.507093    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ae90b09d87dc5c5c8edda29cc59ba527d51e794b546cb03616af2757088ed4f"} err="failed to get container status \"2ae90b09d87dc5c5c8edda29cc59ba527d51e794b546cb03616af2757088ed4f\": rpc error: code = NotFound desc = could not find container \"2ae90b09d87dc5c5c8edda29cc59ba527d51e794b546cb03616af2757088ed4f\": container with ID starting with 2ae90b09d87dc5c5c8edda29cc59ba527d51e794b546cb03616af2757088ed4f not found: ID does not exist"
Sep 09 21:22:45.507168 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.507112    2581 scope.go:117] "RemoveContainer" containerID="c152c6c363c70b4667faac53a34cd8276c466124dfe22ec5ffcf1523b859ef5d"
Sep 09 21:22:45.507430 ip-10-0-134-84 kubenswrapper[2581]: E0909 21:22:45.507401    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c152c6c363c70b4667faac53a34cd8276c466124dfe22ec5ffcf1523b859ef5d\": container with ID starting with c152c6c363c70b4667faac53a34cd8276c466124dfe22ec5ffcf1523b859ef5d not found: ID does not exist" containerID="c152c6c363c70b4667faac53a34cd8276c466124dfe22ec5ffcf1523b859ef5d"
Sep 09 21:22:45.507514 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.507435    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c152c6c363c70b4667faac53a34cd8276c466124dfe22ec5ffcf1523b859ef5d"} err="failed to get container status \"c152c6c363c70b4667faac53a34cd8276c466124dfe22ec5ffcf1523b859ef5d\": rpc error: code = NotFound desc = could not find container \"c152c6c363c70b4667faac53a34cd8276c466124dfe22ec5ffcf1523b859ef5d\": container with ID starting with c152c6c363c70b4667faac53a34cd8276c466124dfe22ec5ffcf1523b859ef5d not found: ID does not exist"
Sep 09 21:22:45.540920 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.540842    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-dshm\") pod \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") "
Sep 09 21:22:45.540920 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.540909    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa267f05-3c76-4616-b33f-1b3ffd19eb78-tls-certs\") pod \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") "
Sep 09 21:22:45.541130 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.540939    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-tmp-dir\") pod \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") "
Sep 09 21:22:45.541130 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.541115    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa267f05-3c76-4616-b33f-1b3ffd19eb78-cabundle-cert\") pod \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") "
Sep 09 21:22:45.541250 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.541193    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb2vh\" (UniqueName: \"kubernetes.io/projected/fa267f05-3c76-4616-b33f-1b3ffd19eb78-kube-api-access-vb2vh\") pod \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") "
Sep 09 21:22:45.541250 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.541226    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-home\") pod \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") "
Sep 09 21:22:45.541348 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.541250    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-model-cache\") pod \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") "
Sep 09 21:22:45.541348 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.541277    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-kserve-provision-location\") pod \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\" (UID: \"fa267f05-3c76-4616-b33f-1b3ffd19eb78\") "
Sep 09 21:22:45.542116 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.542069    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa267f05-3c76-4616-b33f-1b3ffd19eb78-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fa267f05-3c76-4616-b33f-1b3ffd19eb78" (UID: "fa267f05-3c76-4616-b33f-1b3ffd19eb78"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:22:45.542253 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.542145    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-home" (OuterVolumeSpecName: "home") pod "fa267f05-3c76-4616-b33f-1b3ffd19eb78" (UID: "fa267f05-3c76-4616-b33f-1b3ffd19eb78"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:45.542253 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.542217    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa267f05-3c76-4616-b33f-1b3ffd19eb78" (UID: "fa267f05-3c76-4616-b33f-1b3ffd19eb78"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:45.543683 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.543650    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa267f05-3c76-4616-b33f-1b3ffd19eb78-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa267f05-3c76-4616-b33f-1b3ffd19eb78" (UID: "fa267f05-3c76-4616-b33f-1b3ffd19eb78"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:22:45.543800 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.543782    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-dshm" (OuterVolumeSpecName: "dshm") pod "fa267f05-3c76-4616-b33f-1b3ffd19eb78" (UID: "fa267f05-3c76-4616-b33f-1b3ffd19eb78"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:45.543908 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.543880    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa267f05-3c76-4616-b33f-1b3ffd19eb78-kube-api-access-vb2vh" (OuterVolumeSpecName: "kube-api-access-vb2vh") pod "fa267f05-3c76-4616-b33f-1b3ffd19eb78" (UID: "fa267f05-3c76-4616-b33f-1b3ffd19eb78"). InnerVolumeSpecName "kube-api-access-vb2vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:22:45.556770 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.556738    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa267f05-3c76-4616-b33f-1b3ffd19eb78" (UID: "fa267f05-3c76-4616-b33f-1b3ffd19eb78"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:45.564892 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.564855    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa267f05-3c76-4616-b33f-1b3ffd19eb78" (UID: "fa267f05-3c76-4616-b33f-1b3ffd19eb78"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:45.641946 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.641909    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa267f05-3c76-4616-b33f-1b3ffd19eb78-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:22:45.641946 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.641938    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:22:45.641946 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.641950    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa267f05-3c76-4616-b33f-1b3ffd19eb78-cabundle-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:22:45.642172 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.641963    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb2vh\" (UniqueName: \"kubernetes.io/projected/fa267f05-3c76-4616-b33f-1b3ffd19eb78-kube-api-access-vb2vh\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:22:45.642172 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.641972    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:22:45.642172 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.641982    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:22:45.642172 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.641989    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-kserve-provision-location\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:22:45.642172 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.641998    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa267f05-3c76-4616-b33f-1b3ffd19eb78-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:22:45.823586 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.823547    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"]
Sep 09 21:22:45.834435 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:45.834401    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6bc486dfcb-v49wf"]
Sep 09 21:22:46.635983 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:46.635948    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa267f05-3c76-4616-b33f-1b3ffd19eb78" path="/var/lib/kubelet/pods/fa267f05-3c76-4616-b33f-1b3ffd19eb78/volumes"
Sep 09 21:22:48.310882 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:48.310840    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:22:48.312328 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:48.312300    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz" podUID="4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 09 21:22:58.311537 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:22:58.311492    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz" podUID="4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 09 21:23:08.311478 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:08.311415    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz" podUID="4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 09 21:23:18.311778 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:18.311733    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz" podUID="4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 09 21:23:28.311532 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:28.311477    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz" podUID="4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 09 21:23:29.850944 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:29.850896    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks"]
Sep 09 21:23:29.852164 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:29.852135    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa267f05-3c76-4616-b33f-1b3ffd19eb78" containerName="storage-initializer"
Sep 09 21:23:29.852307 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:29.852293    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa267f05-3c76-4616-b33f-1b3ffd19eb78" containerName="storage-initializer"
Sep 09 21:23:29.852429 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:29.852417    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa267f05-3c76-4616-b33f-1b3ffd19eb78" containerName="main"
Sep 09 21:23:29.852537 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:29.852525    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa267f05-3c76-4616-b33f-1b3ffd19eb78" containerName="main"
Sep 09 21:23:29.852752 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:29.852739    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa267f05-3c76-4616-b33f-1b3ffd19eb78" containerName="main"
Sep 09 21:23:29.856574 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:29.856550    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks"
Sep 09 21:23:29.860420 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:29.860391    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks"]
Sep 09 21:23:29.861017 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:29.860995    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 09 21:23:29.861217 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:29.861119    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-nk55k\""
Sep 09 21:23:29.861327 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:29.861228    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-jxfn7\""
Sep 09 21:23:29.861393 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:29.861329    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 09 21:23:29.861393 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:29.860995    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 09 21:23:29.926920 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:29.926881    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b870322e-28ee-4467-881d-4c1b4619fe56-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks\" (UID: \"b870322e-28ee-4467-881d-4c1b4619fe56\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks"
Sep 09 21:23:29.927111 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:29.926990    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfv8q\" (UniqueName: \"kubernetes.io/projected/b870322e-28ee-4467-881d-4c1b4619fe56-kube-api-access-cfv8q\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks\" (UID: \"b870322e-28ee-4467-881d-4c1b4619fe56\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks"
Sep 09 21:23:30.028433 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:30.028400    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfv8q\" (UniqueName: \"kubernetes.io/projected/b870322e-28ee-4467-881d-4c1b4619fe56-kube-api-access-cfv8q\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks\" (UID: \"b870322e-28ee-4467-881d-4c1b4619fe56\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks"
Sep 09 21:23:30.028640 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:30.028501    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b870322e-28ee-4467-881d-4c1b4619fe56-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks\" (UID: \"b870322e-28ee-4467-881d-4c1b4619fe56\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks"
Sep 09 21:23:30.032047 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:30.031990    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b870322e-28ee-4467-881d-4c1b4619fe56-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks\" (UID: \"b870322e-28ee-4467-881d-4c1b4619fe56\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks"
Sep 09 21:23:30.040324 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:30.040291    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfv8q\" (UniqueName: \"kubernetes.io/projected/b870322e-28ee-4467-881d-4c1b4619fe56-kube-api-access-cfv8q\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks\" (UID: \"b870322e-28ee-4467-881d-4c1b4619fe56\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks"
Sep 09 21:23:30.171443 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:30.171354    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks"
Sep 09 21:23:30.448260 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:30.448227    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks"]
Sep 09 21:23:30.449685 ip-10-0-134-84 kubenswrapper[2581]: W0909 21:23:30.449647    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb870322e_28ee_4467_881d_4c1b4619fe56.slice/crio-7fb7163132207ba0ff657cf663caa6144922626c966c4e8942a9dbbaf44c8031 WatchSource:0}: Error finding container 7fb7163132207ba0ff657cf663caa6144922626c966c4e8942a9dbbaf44c8031: Status 404 returned error can't find the container with id 7fb7163132207ba0ff657cf663caa6144922626c966c4e8942a9dbbaf44c8031
Sep 09 21:23:30.736585 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:30.736552    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks" event={"ID":"b870322e-28ee-4467-881d-4c1b4619fe56","Type":"ContainerStarted","Data":"7235a95d27c63dced4d5e5f2bbe6c2119fc58707d0ff1f8d0d69a5224212431c"}
Sep 09 21:23:30.736775 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:30.736616    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks" event={"ID":"b870322e-28ee-4467-881d-4c1b4619fe56","Type":"ContainerStarted","Data":"7fb7163132207ba0ff657cf663caa6144922626c966c4e8942a9dbbaf44c8031"}
Sep 09 21:23:30.736775 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:30.736716    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks"
Sep 09 21:23:30.756706 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:30.756656    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks" podStartSLOduration=1.7566403720000001 podStartE2EDuration="1.756640372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:23:30.754569862 +0000 UTC m=+4352.759746860" watchObservedRunningTime="2026-09-09 21:23:30.756640372 +0000 UTC m=+4352.761817435"
Sep 09 21:23:38.311397 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:38.311352    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz" podUID="4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 09 21:23:48.311117 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:48.311057    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz" podUID="4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 09 21:23:58.311168 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:23:58.311123    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz" podUID="4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 09 21:24:01.744826 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:01.744784    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks"
Sep 09 21:24:08.311367 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:08.311323    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz" podUID="4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 09 21:24:18.311121 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:18.311079    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz" podUID="4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 09 21:24:21.609580 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:21.609481    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks"]
Sep 09 21:24:21.744298 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:21.744249    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks" podUID="b870322e-28ee-4467-881d-4c1b4619fe56" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:24:28.311722 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:28.311678    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz" podUID="4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 09 21:24:31.744869 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:31.744822    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks" podUID="b870322e-28ee-4467-881d-4c1b4619fe56" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:24:36.630574 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:36.630524    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks" podUID="b870322e-28ee-4467-881d-4c1b4619fe56" containerName="main" containerID="cri-o://7235a95d27c63dced4d5e5f2bbe6c2119fc58707d0ff1f8d0d69a5224212431c" gracePeriod=45
Sep 09 21:24:38.321349 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:38.321315    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:24:38.328940 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:38.328918    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:24:39.708011 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:39.707973    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"]
Sep 09 21:24:41.744963 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:41.744925    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks" podUID="b870322e-28ee-4467-881d-4c1b4619fe56" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:24:41.745354 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:41.745030    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks"
Sep 09 21:24:51.744511 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:51.744447    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks" podUID="b870322e-28ee-4467-881d-4c1b4619fe56" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:24:55.135939 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:55.135899    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz" podUID="4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" containerName="main" containerID="cri-o://3d7444941194f247e2def7548ab1882752cdc6c59873690c78317a8850142977" gracePeriod=45
Sep 09 21:24:55.322891 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:55.322841    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz" podUID="4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 09 21:24:55.796223 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:55.796198    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:24:55.969393 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:55.969303    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-tmp-dir\") pod \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") "
Sep 09 21:24:55.969393 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:55.969340    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-tls-certs\") pod \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") "
Sep 09 21:24:55.969657 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:55.969400    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnkws\" (UniqueName: \"kubernetes.io/projected/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-kube-api-access-gnkws\") pod \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") "
Sep 09 21:24:55.969657 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:55.969482    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-model-cache\") pod \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") "
Sep 09 21:24:55.969657 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:55.969517    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-cabundle-cert\") pod \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") "
Sep 09 21:24:55.969657 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:55.969541    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-dshm\") pod \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") "
Sep 09 21:24:55.969657 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:55.969578    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-home\") pod \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") "
Sep 09 21:24:55.969657 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:55.969603    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-kserve-provision-location\") pod \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\" (UID: \"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675\") "
Sep 09 21:24:55.969989 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:55.969763    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-model-cache" (OuterVolumeSpecName: "model-cache") pod "4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" (UID: "4aed7d4e-5ef2-43ac-a0fd-3569e87f5675"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:55.969989 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:55.969915    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" (UID: "4aed7d4e-5ef2-43ac-a0fd-3569e87f5675"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:24:55.970102 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:55.969947    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-model-cache\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:24:55.970408 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:55.970383    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-home" (OuterVolumeSpecName: "home") pod "4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" (UID: "4aed7d4e-5ef2-43ac-a0fd-3569e87f5675"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:55.971862 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:55.971835    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-dshm" (OuterVolumeSpecName: "dshm") pod "4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" (UID: "4aed7d4e-5ef2-43ac-a0fd-3569e87f5675"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:55.972093 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:55.972060    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-kube-api-access-gnkws" (OuterVolumeSpecName: "kube-api-access-gnkws") pod "4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" (UID: "4aed7d4e-5ef2-43ac-a0fd-3569e87f5675"). InnerVolumeSpecName "kube-api-access-gnkws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:24:55.972218 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:55.972188    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" (UID: "4aed7d4e-5ef2-43ac-a0fd-3569e87f5675"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:24:55.989453 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:55.989420    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" (UID: "4aed7d4e-5ef2-43ac-a0fd-3569e87f5675"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:56.001794 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:56.001766    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" (UID: "4aed7d4e-5ef2-43ac-a0fd-3569e87f5675"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:56.070597 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:56.070564    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnkws\" (UniqueName: \"kubernetes.io/projected/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-kube-api-access-gnkws\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:24:56.070597 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:56.070593    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-cabundle-cert\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:24:56.070597 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:56.070603    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-dshm\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:24:56.070814 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:56.070610    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-home\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:24:56.070814 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:56.070618    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-kserve-provision-location\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:24:56.070814 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:56.070626    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-tmp-dir\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:24:56.070814 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:56.070634    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:24:56.196179 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:56.196136    2581 generic.go:356] "Generic (PLEG): container finished" podID="4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" containerID="3d7444941194f247e2def7548ab1882752cdc6c59873690c78317a8850142977" exitCode=0
Sep 09 21:24:56.196594 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:56.196208    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"
Sep 09 21:24:56.196594 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:56.196218    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz" event={"ID":"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675","Type":"ContainerDied","Data":"3d7444941194f247e2def7548ab1882752cdc6c59873690c78317a8850142977"}
Sep 09 21:24:56.196594 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:56.196303    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz" event={"ID":"4aed7d4e-5ef2-43ac-a0fd-3569e87f5675","Type":"ContainerDied","Data":"ad06392f039d1baec853eea131e7ec88b87a50ab84e630d320c7fa7dc4ca441b"}
Sep 09 21:24:56.196594 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:56.196327    2581 scope.go:117] "RemoveContainer" containerID="3d7444941194f247e2def7548ab1882752cdc6c59873690c78317a8850142977"
Sep 09 21:24:56.206253 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:56.206235    2581 scope.go:117] "RemoveContainer" containerID="d5aad78020f713a4ebf234b5316a59fd0d110e0251f2c30ecf8077126fe6ddd3"
Sep 09 21:24:56.213955 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:56.213938    2581 scope.go:117] "RemoveContainer" containerID="3d7444941194f247e2def7548ab1882752cdc6c59873690c78317a8850142977"
Sep 09 21:24:56.214231 ip-10-0-134-84 kubenswrapper[2581]: E0909 21:24:56.214211    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d7444941194f247e2def7548ab1882752cdc6c59873690c78317a8850142977\": container with ID starting with 3d7444941194f247e2def7548ab1882752cdc6c59873690c78317a8850142977 not found: ID does not exist" containerID="3d7444941194f247e2def7548ab1882752cdc6c59873690c78317a8850142977"
Sep 09 21:24:56.214292 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:56.214238    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d7444941194f247e2def7548ab1882752cdc6c59873690c78317a8850142977"} err="failed to get container status \"3d7444941194f247e2def7548ab1882752cdc6c59873690c78317a8850142977\": rpc error: code = NotFound desc = could not find container \"3d7444941194f247e2def7548ab1882752cdc6c59873690c78317a8850142977\": container with ID starting with 3d7444941194f247e2def7548ab1882752cdc6c59873690c78317a8850142977 not found: ID does not exist"
Sep 09 21:24:56.214292 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:56.214256    2581 scope.go:117] "RemoveContainer" containerID="d5aad78020f713a4ebf234b5316a59fd0d110e0251f2c30ecf8077126fe6ddd3"
Sep 09 21:24:56.214487 ip-10-0-134-84 kubenswrapper[2581]: E0909 21:24:56.214453    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5aad78020f713a4ebf234b5316a59fd0d110e0251f2c30ecf8077126fe6ddd3\": container with ID starting with d5aad78020f713a4ebf234b5316a59fd0d110e0251f2c30ecf8077126fe6ddd3 not found: ID does not exist" containerID="d5aad78020f713a4ebf234b5316a59fd0d110e0251f2c30ecf8077126fe6ddd3"
Sep 09 21:24:56.214534 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:56.214491    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5aad78020f713a4ebf234b5316a59fd0d110e0251f2c30ecf8077126fe6ddd3"} err="failed to get container status \"d5aad78020f713a4ebf234b5316a59fd0d110e0251f2c30ecf8077126fe6ddd3\": rpc error: code = NotFound desc = could not find container \"d5aad78020f713a4ebf234b5316a59fd0d110e0251f2c30ecf8077126fe6ddd3\": container with ID starting with d5aad78020f713a4ebf234b5316a59fd0d110e0251f2c30ecf8077126fe6ddd3 not found: ID does not exist"
Sep 09 21:24:56.222257 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:56.222208    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"]
Sep 09 21:24:56.227022 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:56.227001    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-dfqmz"]
Sep 09 21:24:56.635036 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:24:56.635004    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" path="/var/lib/kubelet/pods/4aed7d4e-5ef2-43ac-a0fd-3569e87f5675/volumes"
Sep 09 21:25:01.744832 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:25:01.744793    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks" podUID="b870322e-28ee-4467-881d-4c1b4619fe56" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:25:06.882289 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:25:06.882256    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks"
Sep 09 21:25:06.958599 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:25:06.958501    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfv8q\" (UniqueName: \"kubernetes.io/projected/b870322e-28ee-4467-881d-4c1b4619fe56-kube-api-access-cfv8q\") pod \"b870322e-28ee-4467-881d-4c1b4619fe56\" (UID: \"b870322e-28ee-4467-881d-4c1b4619fe56\") "
Sep 09 21:25:06.958599 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:25:06.958600    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b870322e-28ee-4467-881d-4c1b4619fe56-tls-certs\") pod \"b870322e-28ee-4467-881d-4c1b4619fe56\" (UID: \"b870322e-28ee-4467-881d-4c1b4619fe56\") "
Sep 09 21:25:06.960847 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:25:06.960810    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b870322e-28ee-4467-881d-4c1b4619fe56-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b870322e-28ee-4467-881d-4c1b4619fe56" (UID: "b870322e-28ee-4467-881d-4c1b4619fe56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:25:06.961008 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:25:06.960888    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b870322e-28ee-4467-881d-4c1b4619fe56-kube-api-access-cfv8q" (OuterVolumeSpecName: "kube-api-access-cfv8q") pod "b870322e-28ee-4467-881d-4c1b4619fe56" (UID: "b870322e-28ee-4467-881d-4c1b4619fe56"). InnerVolumeSpecName "kube-api-access-cfv8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:25:07.059497 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:25:07.059439    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b870322e-28ee-4467-881d-4c1b4619fe56-tls-certs\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:25:07.059497 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:25:07.059492    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfv8q\" (UniqueName: \"kubernetes.io/projected/b870322e-28ee-4467-881d-4c1b4619fe56-kube-api-access-cfv8q\") on node \"ip-10-0-134-84.ec2.internal\" DevicePath \"\""
Sep 09 21:25:07.255114 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:25:07.255068    2581 generic.go:356] "Generic (PLEG): container finished" podID="b870322e-28ee-4467-881d-4c1b4619fe56" containerID="7235a95d27c63dced4d5e5f2bbe6c2119fc58707d0ff1f8d0d69a5224212431c" exitCode=0
Sep 09 21:25:07.255290 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:25:07.255120    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks"
Sep 09 21:25:07.255290 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:25:07.255139    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks" event={"ID":"b870322e-28ee-4467-881d-4c1b4619fe56","Type":"ContainerDied","Data":"7235a95d27c63dced4d5e5f2bbe6c2119fc58707d0ff1f8d0d69a5224212431c"}
Sep 09 21:25:07.255290 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:25:07.255183    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks" event={"ID":"b870322e-28ee-4467-881d-4c1b4619fe56","Type":"ContainerDied","Data":"7fb7163132207ba0ff657cf663caa6144922626c966c4e8942a9dbbaf44c8031"}
Sep 09 21:25:07.255290 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:25:07.255205    2581 scope.go:117] "RemoveContainer" containerID="7235a95d27c63dced4d5e5f2bbe6c2119fc58707d0ff1f8d0d69a5224212431c"
Sep 09 21:25:07.264326 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:25:07.264308    2581 scope.go:117] "RemoveContainer" containerID="7235a95d27c63dced4d5e5f2bbe6c2119fc58707d0ff1f8d0d69a5224212431c"
Sep 09 21:25:07.264633 ip-10-0-134-84 kubenswrapper[2581]: E0909 21:25:07.264615    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7235a95d27c63dced4d5e5f2bbe6c2119fc58707d0ff1f8d0d69a5224212431c\": container with ID starting with 7235a95d27c63dced4d5e5f2bbe6c2119fc58707d0ff1f8d0d69a5224212431c not found: ID does not exist" containerID="7235a95d27c63dced4d5e5f2bbe6c2119fc58707d0ff1f8d0d69a5224212431c"
Sep 09 21:25:07.264685 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:25:07.264642    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7235a95d27c63dced4d5e5f2bbe6c2119fc58707d0ff1f8d0d69a5224212431c"} err="failed to get container status \"7235a95d27c63dced4d5e5f2bbe6c2119fc58707d0ff1f8d0d69a5224212431c\": rpc error: code = NotFound desc = could not find container \"7235a95d27c63dced4d5e5f2bbe6c2119fc58707d0ff1f8d0d69a5224212431c\": container with ID starting with 7235a95d27c63dced4d5e5f2bbe6c2119fc58707d0ff1f8d0d69a5224212431c not found: ID does not exist"
Sep 09 21:25:07.278071 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:25:07.278043    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks"]
Sep 09 21:25:07.283060 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:25:07.283036    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zspks"]
Sep 09 21:25:08.635522 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:25:08.635489    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b870322e-28ee-4467-881d-4c1b4619fe56" path="/var/lib/kubelet/pods/b870322e-28ee-4467-881d-4c1b4619fe56/volumes"
Sep 09 21:25:58.922819 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:25:58.922707    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 21:25:58.926894 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:25:58.925427    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 21:25:58.926894 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:25:58.926647    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 21:25:58.929578 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:25:58.929560    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jqqq8_e1d7bb0f-c4ae-41cc-bc1d-a12071b148b8/ovn-acl-logging/0.log"
Sep 09 21:26:19.827319 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:19.827286    2581 ???:1] "http2: server: error reading preface from client 10.0.134.84:42904: read tcp 10.0.134.84:10250->10.0.134.84:42904: read: connection reset by peer"
Sep 09 21:26:19.931224 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:19.931190    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-46gml_e75de6e0-0c7d-4e09-8408-5378cb8294b1/discovery/0.log"
Sep 09 21:26:19.945491 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:19.945450    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4_4e089356-1131-499c-a8e5-afe9c2c9b0c6/istio-proxy/0.log"
Sep 09 21:26:19.960936 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:19.960907    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4_4e089356-1131-499c-a8e5-afe9c2c9b0c6/istio-proxy/1.log"
Sep 09 21:26:20.759875 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:20.759842    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-46gml_e75de6e0-0c7d-4e09-8408-5378cb8294b1/discovery/0.log"
Sep 09 21:26:20.778993 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:20.778960    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4_4e089356-1131-499c-a8e5-afe9c2c9b0c6/istio-proxy/0.log"
Sep 09 21:26:20.795551 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:20.795520    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4_4e089356-1131-499c-a8e5-afe9c2c9b0c6/istio-proxy/1.log"
Sep 09 21:26:22.034797 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:22.034762    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-dxwm2_7b459bd6-a3b5-45af-b1df-cc0517f020ff/manager/0.log"
Sep 09 21:26:22.175026 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:22.174991    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-ns7nk_9fd7138a-f0ac-4adf-8206-c2a1e273cc00/manager/0.log"
Sep 09 21:26:22.217127 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:22.217096    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-j6nph_d21fa37d-4a14-42e2-b957-2d8a816294a1/manager/0.log"
Sep 09 21:26:28.271952 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:28.271911    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sghw5/must-gather-6vhkg"]
Sep 09 21:26:28.272440 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:28.272232    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" containerName="main"
Sep 09 21:26:28.272440 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:28.272242    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" containerName="main"
Sep 09 21:26:28.272440 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:28.272254    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" containerName="storage-initializer"
Sep 09 21:26:28.272440 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:28.272260    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" containerName="storage-initializer"
Sep 09 21:26:28.272440 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:28.272268    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b870322e-28ee-4467-881d-4c1b4619fe56" containerName="main"
Sep 09 21:26:28.272440 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:28.272274    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b870322e-28ee-4467-881d-4c1b4619fe56" containerName="main"
Sep 09 21:26:28.272440 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:28.272325    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b870322e-28ee-4467-881d-4c1b4619fe56" containerName="main"
Sep 09 21:26:28.272440 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:28.272333    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aed7d4e-5ef2-43ac-a0fd-3569e87f5675" containerName="main"
Sep 09 21:26:28.275470 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:28.275441    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sghw5/must-gather-6vhkg"
Sep 09 21:26:28.278582 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:28.278558    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sghw5\"/\"kube-root-ca.crt\""
Sep 09 21:26:28.278700 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:28.278610    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sghw5\"/\"openshift-service-ca.crt\""
Sep 09 21:26:28.279429 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:28.279408    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sghw5\"/\"default-dockercfg-mcj2p\""
Sep 09 21:26:28.285485 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:28.285449    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sghw5/must-gather-6vhkg"]
Sep 09 21:26:28.319252 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:28.319221    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1d314002-4c74-4a38-a5ac-a424183f15b0-must-gather-output\") pod \"must-gather-6vhkg\" (UID: \"1d314002-4c74-4a38-a5ac-a424183f15b0\") " pod="openshift-must-gather-sghw5/must-gather-6vhkg"
Sep 09 21:26:28.319422 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:28.319273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7x42\" (UniqueName: \"kubernetes.io/projected/1d314002-4c74-4a38-a5ac-a424183f15b0-kube-api-access-f7x42\") pod \"must-gather-6vhkg\" (UID: \"1d314002-4c74-4a38-a5ac-a424183f15b0\") " pod="openshift-must-gather-sghw5/must-gather-6vhkg"
Sep 09 21:26:28.420588 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:28.420550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7x42\" (UniqueName: \"kubernetes.io/projected/1d314002-4c74-4a38-a5ac-a424183f15b0-kube-api-access-f7x42\") pod \"must-gather-6vhkg\" (UID: \"1d314002-4c74-4a38-a5ac-a424183f15b0\") " pod="openshift-must-gather-sghw5/must-gather-6vhkg"
Sep 09 21:26:28.420777 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:28.420626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1d314002-4c74-4a38-a5ac-a424183f15b0-must-gather-output\") pod \"must-gather-6vhkg\" (UID: \"1d314002-4c74-4a38-a5ac-a424183f15b0\") " pod="openshift-must-gather-sghw5/must-gather-6vhkg"
Sep 09 21:26:28.420914 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:28.420896    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1d314002-4c74-4a38-a5ac-a424183f15b0-must-gather-output\") pod \"must-gather-6vhkg\" (UID: \"1d314002-4c74-4a38-a5ac-a424183f15b0\") " pod="openshift-must-gather-sghw5/must-gather-6vhkg"
Sep 09 21:26:28.429837 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:28.429807    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7x42\" (UniqueName: \"kubernetes.io/projected/1d314002-4c74-4a38-a5ac-a424183f15b0-kube-api-access-f7x42\") pod \"must-gather-6vhkg\" (UID: \"1d314002-4c74-4a38-a5ac-a424183f15b0\") " pod="openshift-must-gather-sghw5/must-gather-6vhkg"
Sep 09 21:26:28.584962 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:28.584860    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sghw5/must-gather-6vhkg"
Sep 09 21:26:28.857596 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:28.857546    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sghw5/must-gather-6vhkg"]
Sep 09 21:26:28.859826 ip-10-0-134-84 kubenswrapper[2581]: W0909 21:26:28.859798    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d314002_4c74_4a38_a5ac_a424183f15b0.slice/crio-7286570a9c3b63bd5ff0a9d5d060ca216127acc98bb97f39b60bb734cb2e4001 WatchSource:0}: Error finding container 7286570a9c3b63bd5ff0a9d5d060ca216127acc98bb97f39b60bb734cb2e4001: Status 404 returned error can't find the container with id 7286570a9c3b63bd5ff0a9d5d060ca216127acc98bb97f39b60bb734cb2e4001
Sep 09 21:26:28.861812 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:28.861792    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:26:29.667963 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:29.667922    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sghw5/must-gather-6vhkg" event={"ID":"1d314002-4c74-4a38-a5ac-a424183f15b0","Type":"ContainerStarted","Data":"7286570a9c3b63bd5ff0a9d5d060ca216127acc98bb97f39b60bb734cb2e4001"}
Sep 09 21:26:30.674611 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:30.674562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sghw5/must-gather-6vhkg" event={"ID":"1d314002-4c74-4a38-a5ac-a424183f15b0","Type":"ContainerStarted","Data":"3e45ea40abeb8507b2f93938b8c7d36a28e96773c58061793210b98e2e125b60"}
Sep 09 21:26:30.674611 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:30.674616    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sghw5/must-gather-6vhkg" event={"ID":"1d314002-4c74-4a38-a5ac-a424183f15b0","Type":"ContainerStarted","Data":"95cdf769f6f60df17bfe4745eed1985f757953edcf88d2d06fc8acfd50484ca8"}
Sep 09 21:26:30.693163 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:30.693107    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sghw5/must-gather-6vhkg" podStartSLOduration=1.8627609189999998 podStartE2EDuration="2.693092679s" totalImagesPullingTime="830.33176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:26:30.690802752 +0000 UTC m=+4532.695979745" watchObservedRunningTime="2026-09-09 21:26:30.693092679 +0000 UTC m=+4532.698269674"
Sep 09 21:26:31.376976 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:31.376942    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-z8q9m_5d3756a7-d67d-48ef-aa34-a792650e747d/global-pull-secret-syncer/0.log"
Sep 09 21:26:31.469221 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:31.469167    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-f28rc_bc4b3154-62b5-4d79-bfe4-673807839bf7/konnectivity-agent/0.log"
Sep 09 21:26:31.588790 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:31.588756    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-84.ec2.internal_8226d39d61364b61ba5e1ad9f7514144/haproxy/0.log"
Sep 09 21:26:36.714211 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:36.714145    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-dxwm2_7b459bd6-a3b5-45af-b1df-cc0517f020ff/manager/0.log"
Sep 09 21:26:36.944515 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:36.944406    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-ns7nk_9fd7138a-f0ac-4adf-8206-c2a1e273cc00/manager/0.log"
Sep 09 21:26:37.001520 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:37.001420    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-j6nph_d21fa37d-4a14-42e2-b957-2d8a816294a1/manager/0.log"
Sep 09 21:26:38.582118 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:38.582024    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4znjx_2e553183-00ef-479b-a711-f43feb043bc6/node-exporter/0.log"
Sep 09 21:26:38.607778 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:38.607748    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4znjx_2e553183-00ef-479b-a711-f43feb043bc6/kube-rbac-proxy/0.log"
Sep 09 21:26:38.640916 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:38.640891    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4znjx_2e553183-00ef-479b-a711-f43feb043bc6/init-textfile/0.log"
Sep 09 21:26:41.178128 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:41.178088    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq"]
Sep 09 21:26:41.183520 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:41.183488    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq"
Sep 09 21:26:41.193291 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:41.193220    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq"]
Sep 09 21:26:41.261008 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:41.260965    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/45009a7a-167b-438f-ac9d-68e575293bcb-proc\") pod \"perf-node-gather-daemonset-l4nxq\" (UID: \"45009a7a-167b-438f-ac9d-68e575293bcb\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq"
Sep 09 21:26:41.261259 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:41.261240    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45009a7a-167b-438f-ac9d-68e575293bcb-sys\") pod \"perf-node-gather-daemonset-l4nxq\" (UID: \"45009a7a-167b-438f-ac9d-68e575293bcb\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq"
Sep 09 21:26:41.261382 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:41.261367    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/45009a7a-167b-438f-ac9d-68e575293bcb-lib-modules\") pod \"perf-node-gather-daemonset-l4nxq\" (UID: \"45009a7a-167b-438f-ac9d-68e575293bcb\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq"
Sep 09 21:26:41.261495 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:41.261482    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/45009a7a-167b-438f-ac9d-68e575293bcb-podres\") pod \"perf-node-gather-daemonset-l4nxq\" (UID: \"45009a7a-167b-438f-ac9d-68e575293bcb\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq"
Sep 09 21:26:41.261751 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:41.261722    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk8k2\" (UniqueName: \"kubernetes.io/projected/45009a7a-167b-438f-ac9d-68e575293bcb-kube-api-access-vk8k2\") pod \"perf-node-gather-daemonset-l4nxq\" (UID: \"45009a7a-167b-438f-ac9d-68e575293bcb\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq"
Sep 09 21:26:41.282604 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:41.282495    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/2.log"
Sep 09 21:26:41.287904 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:41.287868    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-ldv8m_b60c4c0d-ce10-44ef-a2e7-a7562b103e46/console-operator/3.log"
Sep 09 21:26:41.362676 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:41.362638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk8k2\" (UniqueName: \"kubernetes.io/projected/45009a7a-167b-438f-ac9d-68e575293bcb-kube-api-access-vk8k2\") pod \"perf-node-gather-daemonset-l4nxq\" (UID: \"45009a7a-167b-438f-ac9d-68e575293bcb\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq"
Sep 09 21:26:41.362977 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:41.362954    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/45009a7a-167b-438f-ac9d-68e575293bcb-proc\") pod \"perf-node-gather-daemonset-l4nxq\" (UID: \"45009a7a-167b-438f-ac9d-68e575293bcb\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq"
Sep 09 21:26:41.363301 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:41.363056    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/45009a7a-167b-438f-ac9d-68e575293bcb-proc\") pod \"perf-node-gather-daemonset-l4nxq\" (UID: \"45009a7a-167b-438f-ac9d-68e575293bcb\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq"
Sep 09 21:26:41.363301 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:41.363093    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45009a7a-167b-438f-ac9d-68e575293bcb-sys\") pod \"perf-node-gather-daemonset-l4nxq\" (UID: \"45009a7a-167b-438f-ac9d-68e575293bcb\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq"
Sep 09 21:26:41.363301 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:41.363133    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/45009a7a-167b-438f-ac9d-68e575293bcb-lib-modules\") pod \"perf-node-gather-daemonset-l4nxq\" (UID: \"45009a7a-167b-438f-ac9d-68e575293bcb\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq"
Sep 09 21:26:41.363301 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:41.363144    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45009a7a-167b-438f-ac9d-68e575293bcb-sys\") pod \"perf-node-gather-daemonset-l4nxq\" (UID: \"45009a7a-167b-438f-ac9d-68e575293bcb\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq"
Sep 09 21:26:41.363301 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:41.363155    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/45009a7a-167b-438f-ac9d-68e575293bcb-podres\") pod \"perf-node-gather-daemonset-l4nxq\" (UID: \"45009a7a-167b-438f-ac9d-68e575293bcb\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq"
Sep 09 21:26:41.363301 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:41.363272    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/45009a7a-167b-438f-ac9d-68e575293bcb-lib-modules\") pod \"perf-node-gather-daemonset-l4nxq\" (UID: \"45009a7a-167b-438f-ac9d-68e575293bcb\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq"
Sep 09 21:26:41.363742 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:41.363324    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/45009a7a-167b-438f-ac9d-68e575293bcb-podres\") pod \"perf-node-gather-daemonset-l4nxq\" (UID: \"45009a7a-167b-438f-ac9d-68e575293bcb\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq"
Sep 09 21:26:41.376509 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:41.376480    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk8k2\" (UniqueName: \"kubernetes.io/projected/45009a7a-167b-438f-ac9d-68e575293bcb-kube-api-access-vk8k2\") pod \"perf-node-gather-daemonset-l4nxq\" (UID: \"45009a7a-167b-438f-ac9d-68e575293bcb\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq"
Sep 09 21:26:41.500188 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:41.500148    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq"
Sep 09 21:26:41.761383 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:41.761297    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-56fd7586dc-nmwz6_b6494ca8-56c5-478b-9871-edddb8a79b97/console/0.log"
Sep 09 21:26:41.788363 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:41.788340    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq"]
Sep 09 21:26:41.790507 ip-10-0-134-84 kubenswrapper[2581]: W0909 21:26:41.790477    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod45009a7a_167b_438f_ac9d_68e575293bcb.slice/crio-24dfcb107d9c8715c55edbcae058ce4f823c315ec26734580eef76108cf445d1 WatchSource:0}: Error finding container 24dfcb107d9c8715c55edbcae058ce4f823c315ec26734580eef76108cf445d1: Status 404 returned error can't find the container with id 24dfcb107d9c8715c55edbcae058ce4f823c315ec26734580eef76108cf445d1
Sep 09 21:26:42.437593 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:42.437561    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-r2fqt_ae4c0694-26b1-4a2a-a5fa-b134191927fc/volume-data-source-validator/0.log"
Sep 09 21:26:42.763141 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:42.763109    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq" event={"ID":"45009a7a-167b-438f-ac9d-68e575293bcb","Type":"ContainerStarted","Data":"8bd4aa63bc127dc3e6e403c64c3dc21bfbed661535f1febc09583e29ba6b1116"}
Sep 09 21:26:42.763349 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:42.763163    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq" event={"ID":"45009a7a-167b-438f-ac9d-68e575293bcb","Type":"ContainerStarted","Data":"24dfcb107d9c8715c55edbcae058ce4f823c315ec26734580eef76108cf445d1"}
Sep 09 21:26:42.763349 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:42.763193    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq"
Sep 09 21:26:42.782034 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:42.781964    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq" podStartSLOduration=1.781942397 podStartE2EDuration="1.781942397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:26:42.779894709 +0000 UTC m=+4544.785071705" watchObservedRunningTime="2026-09-09 21:26:42.781942397 +0000 UTC m=+4544.787119395"
Sep 09 21:26:43.570895 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:43.570863    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qd6j5_2bff29a8-0416-4090-af52-b62924509cd0/dns/0.log"
Sep 09 21:26:43.594306 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:43.594278    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qd6j5_2bff29a8-0416-4090-af52-b62924509cd0/kube-rbac-proxy/0.log"
Sep 09 21:26:43.646415 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:43.646383    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l7rwd_297425e2-5552-4654-b41b-336d88d5a56c/dns-node-resolver/0.log"
Sep 09 21:26:44.256656 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:44.256623    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qpmd2_610a91de-a1e2-445d-8f72-48aba0d8a2bf/node-ca/0.log"
Sep 09 21:26:45.507819 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:45.507782    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-46gml_e75de6e0-0c7d-4e09-8408-5378cb8294b1/discovery/0.log"
Sep 09 21:26:45.541440 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:45.541410    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4_4e089356-1131-499c-a8e5-afe9c2c9b0c6/istio-proxy/0.log"
Sep 09 21:26:45.558724 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:45.558693    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4_4e089356-1131-499c-a8e5-afe9c2c9b0c6/istio-proxy/1.log"
Sep 09 21:26:46.189978 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:46.189951    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4prkf_6d63ce9d-307d-4a8f-bcf1-9f53a3d3c956/serve-healthcheck-canary/0.log"
Sep 09 21:26:46.732976 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:46.732945    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-999gg_f211a633-063b-4450-bdb9-26288976f42f/kube-rbac-proxy/0.log"
Sep 09 21:26:46.760470 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:46.760430    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-999gg_f211a633-063b-4450-bdb9-26288976f42f/exporter/0.log"
Sep 09 21:26:46.790331 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:46.790295    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-999gg_f211a633-063b-4450-bdb9-26288976f42f/extractor/0.log"
Sep 09 21:26:48.780836 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:48.780808    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-l4nxq"
Sep 09 21:26:50.724914 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:50.724786    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-46gml_e75de6e0-0c7d-4e09-8408-5378cb8294b1/discovery/0.log"
Sep 09 21:26:50.753989 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:50.753904    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4_4e089356-1131-499c-a8e5-afe9c2c9b0c6/istio-proxy/0.log"
Sep 09 21:26:50.776474 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:50.776425    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4_4e089356-1131-499c-a8e5-afe9c2c9b0c6/istio-proxy/1.log"
Sep 09 21:26:51.068016 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:51.067983    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-55b9566966-ch2xc_ceccaadd-9d57-4932-947c-534e33017cad/manager/0.log"
Sep 09 21:26:52.044056 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:52.044015    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-46gml_e75de6e0-0c7d-4e09-8408-5378cb8294b1/discovery/0.log"
Sep 09 21:26:52.069090 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:52.069056    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4_4e089356-1131-499c-a8e5-afe9c2c9b0c6/istio-proxy/0.log"
Sep 09 21:26:52.098002 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:52.097966    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-hvkp4_4e089356-1131-499c-a8e5-afe9c2c9b0c6/istio-proxy/1.log"
Sep 09 21:26:52.671788 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:52.671648    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-d6sfs_d84788dd-b7ed-43bc-affd-9b982be11af5/manager/0.log"
Sep 09 21:26:52.759115 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:52.759077    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-jlltk_a2ce1df6-13fd-4804-b7a3-16c520bb7898/seaweedfs/0.log"
Sep 09 21:26:57.908663 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:57.908635    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-8vcxj_71bd9823-f654-4f3d-97b4-9c186b4e4028/kube-storage-version-migrator-operator/1.log"
Sep 09 21:26:57.909559 ip-10-0-134-84 kubenswrapper[2581]: I0909 21:26:57.909532    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-8vcxj_71bd9823-f654-4f3d-97b4-9c186b4e4028/kube-storage-version-migrator-operator/0.log"